Implement approved-file import and integrity verification #55

Merged

JackFrostbyte merged 2 commits from feature/issue-11-file-integrity-import into develop

2026-08-03 02:57:53 -04:00

JackFrostbyte commented

2026-08-03 02:49:17 -04:00

Owner

Summary

Add a deterministic library operation for locating and verifying approved Arkive source originals by document_id.
Derive the original-file path from the authoritative source.json and documents.json records.
Enforce repository and source-directory path boundaries before reading an original.
Verify the recorded byte size and SHA-256 checksum without modifying or transforming the file.
Return immutable success results and stable, machine-readable failure codes.
Add focused regression coverage for the five approved pilot originals and required negative cases.

Implementation

The new verify_approved_document(repository_root, document_id) operation:

scans the documented data/sources/<category>/<source-key>/ layout in deterministic sorted order;
locates a unique matching document record;
requires the co-located source to have review_status: accepted;
rejects metadata-only sources;
requires the document source_id to match the authoritative source record;
verifies only records with role: original;
requires a recorded byte size and SHA-256 checksum;
rejects unsupported checksum algorithms;
derives the input path exclusively from approved metadata;
requires the target to remain directly inside its approved source directory;
rejects absolute paths, parent traversal, malformed paths, and paths into other repository areas;
rejects missing files, broken links, all symbolic links, and non-regular files;
enforces the existing original-file naming convention;
checks the recorded media type against the permitted lowercase extension;
opens originals read-only and in binary mode;
calculates byte size and SHA-256 deterministically;
detects a file that changes while it is being read;
distinguishes byte-size mismatches from SHA-256 mismatches;
returns an immutable VerifiedDocument result on success;
raises a structured VerificationError with a stable failure code and JSON-serializable details on failure.

The approved boundary is the exact source directory containing the authoritative metadata, rather than the wider repository. Even an otherwise in-bound symbolic link is rejected so that committed originals remain direct, immutable files.

Processing approval remains separate from bundle eligibility. The verifier therefore does not require license.bundleable: true.

Verification coverage

The new test suite covers:

successful verification of all five approved pilot originals;
repeated verification with identical results;
proof that successful verification preserves file bytes, size, mode, and modification time;
a standard SHA-256 known-answer test;
missing files;
missing byte-size metadata;
byte-size mismatch;
same-size SHA-256 mismatch;
absolute paths;
parent-directory traversal;
paths into another approved source;
paths into an unapproved repository area;
external symbolic links;
broken symbolic links;
in-bound symbolic links;
directories and other non-regular targets;
unexpected original filenames;
media-type and extension mismatches;
unsupported media types;
unsupported checksum algorithms;
non-accepted and metadata-only sources;
non-original document records;
source identifier mismatch;
duplicate and missing document identifiers;
files that change during verification;
machine-readable success and failure results.

Validation

git diff --check
python pipeline/validate/validate_records.py
python -m unittest discover -s tests -v
python -m ruff check src pipeline tests
python -m ruff format --check src pipeline tests
python -m pip check

Results:

Metadata validation passed for 6 sources, 6 documents, and 2 chunks across 13 record files.
All 55 unit tests passed.
Ruff linting passed.
Ruff formatting validation passed for all 6 Python files.
Dependency validation reported no broken requirements.
All five approved pilot originals matched their recorded byte sizes and SHA-256 checksums.

Scope confirmation

Added only src/arkive/file_integrity.py and tests/test_file_integrity.py.
No schemas, metadata records, policies, committed originals, or dependencies changed.
No extraction, normalization, chunk generation, indexing, download, embedding, or RAG behavior was introduced.
No PDF-processing dependency was introduced.
qpdf and Poppler remain deferred to Issue #13.
No input file is written or transformed.

Linked issue

Closes #11

## Summary * Add a deterministic library operation for locating and verifying approved Arkive source originals by `document_id`. * Derive the original-file path from the authoritative `source.json` and `documents.json` records. * Enforce repository and source-directory path boundaries before reading an original. * Verify the recorded byte size and SHA-256 checksum without modifying or transforming the file. * Return immutable success results and stable, machine-readable failure codes. * Add focused regression coverage for the five approved pilot originals and required negative cases. ## Implementation The new `verify_approved_document(repository_root, document_id)` operation: * scans the documented `data/sources/<category>/<source-key>/` layout in deterministic sorted order; * locates a unique matching document record; * requires the co-located source to have `review_status: accepted`; * rejects metadata-only sources; * requires the document `source_id` to match the authoritative source record; * verifies only records with `role: original`; * requires a recorded byte size and SHA-256 checksum; * rejects unsupported checksum algorithms; * derives the input path exclusively from approved metadata; * requires the target to remain directly inside its approved source directory; * rejects absolute paths, parent traversal, malformed paths, and paths into other repository areas; * rejects missing files, broken links, all symbolic links, and non-regular files; * enforces the existing original-file naming convention; * checks the recorded media type against the permitted lowercase extension; * opens originals read-only and in binary mode; * calculates byte size and SHA-256 deterministically; * detects a file that changes while it is being read; * distinguishes byte-size mismatches from SHA-256 mismatches; * returns an immutable `VerifiedDocument` result on success; * raises a structured `VerificationError` with a stable failure code and JSON-serializable details on failure. The approved boundary is the exact source directory containing the authoritative metadata, rather than the wider repository. Even an otherwise in-bound symbolic link is rejected so that committed originals remain direct, immutable files. Processing approval remains separate from bundle eligibility. The verifier therefore does not require `license.bundleable: true`. ## Verification coverage The new test suite covers: * successful verification of all five approved pilot originals; * repeated verification with identical results; * proof that successful verification preserves file bytes, size, mode, and modification time; * a standard SHA-256 known-answer test; * missing files; * missing byte-size metadata; * byte-size mismatch; * same-size SHA-256 mismatch; * absolute paths; * parent-directory traversal; * paths into another approved source; * paths into an unapproved repository area; * external symbolic links; * broken symbolic links; * in-bound symbolic links; * directories and other non-regular targets; * unexpected original filenames; * media-type and extension mismatches; * unsupported media types; * unsupported checksum algorithms; * non-accepted and metadata-only sources; * non-original document records; * source identifier mismatch; * duplicate and missing document identifiers; * files that change during verification; * machine-readable success and failure results. ## Validation * [x] `git diff --check` * [x] `python pipeline/validate/validate_records.py` * [x] `python -m unittest discover -s tests -v` * [x] `python -m ruff check src pipeline tests` * [x] `python -m ruff format --check src pipeline tests` * [x] `python -m pip check` Results: * Metadata validation passed for 6 sources, 6 documents, and 2 chunks across 13 record files. * All 55 unit tests passed. * Ruff linting passed. * Ruff formatting validation passed for all 6 Python files. * Dependency validation reported no broken requirements. * All five approved pilot originals matched their recorded byte sizes and SHA-256 checksums. ## Scope confirmation * Added only `src/arkive/file_integrity.py` and `tests/test_file_integrity.py`. * No schemas, metadata records, policies, committed originals, or dependencies changed. * No extraction, normalization, chunk generation, indexing, download, embedding, or RAG behavior was introduced. * No PDF-processing dependency was introduced. * qpdf and Poppler remain deferred to Issue #13. * No input file is written or transformed. ## Linked issue Closes #11

JackFrostbyte added 1 commit

2026-08-03 02:49:17 -04:00