Implement approved-file import and integrity verification #11

New issue

Closed

opened 2026-07-24 19:38:14 -04:00 by JackFrostbyte · 1 comment

JackFrostbyte commented

2026-07-24 19:38:14 -04:00

Owner

Goal

    Implement a deterministic command or library operation that locates an approved input file and verifies its repository boundary, byte size, and SHA-256 integrity.

    ## Why

    Arkive must prove that a metadata record refers to the exact approved file before any extraction or transformation begins.

    ## Scope

    - Locate an approved source file through metadata and the documented repository layout.

Resolve and validate paths without following inputs outside approved repository boundaries.
Reject absolute paths, traversal attempts, broken links, missing files, and unexpected file types where appropriate.
Calculate file byte size and SHA-256 deterministically.
Verify existing document integrity metadata or produce a deterministic proposed metadata result as defined by the processing contract.
Return clear machine-readable or structured errors for missing and mismatched files.
Avoid modifying committed source files.
Add tests using the approved pilot corpus and focused negative fixtures.
```
  ## Out of scope

  - Text or PDF extraction.
```
Text normalization or chunking.
Automatic network downloads.
License review automation.

Index generation.

  ## Dependencies

  - #7 — Establish the Python project and development-tooling baseline

#9 — Align metadata schemas and validation with the intake and processing contracts

#10 — Select and approve the five-document pilot corpus

  ## Acceptance criteria

  - Approved pilot files are located and verified deterministically.

A byte-size mismatch and SHA-256 mismatch are both detected.
Missing files and unsafe paths fail clearly.
Paths outside the approved repository boundary cannot be imported.
Repeated verification of unchanged files produces identical results.
No input file is modified.

Positive and negative tests pass.

  ## Validation

  - Run the complete unit-test suite.

Exercise all five approved files.
Exercise fixtures for missing files, checksum mismatches, size mismatches, absolute paths, traversal, and out-of-bound paths.

Repeat a successful verification and compare results.

  ## Suggested branch

  `feature/issue-11-file-integrity-import`

  <!-- arkive-planning-automation:2026-07-24-v3:M1-06 -->

## Goal Implement a deterministic command or library operation that locates an approved input file and verifies its repository boundary, byte size, and SHA-256 integrity. ## Why Arkive must prove that a metadata record refers to the exact approved file before any extraction or transformation begins. ## Scope - Locate an approved source file through metadata and the documented repository layout. - Resolve and validate paths without following inputs outside approved repository boundaries. - Reject absolute paths, traversal attempts, broken links, missing files, and unexpected file types where appropriate. - Calculate file byte size and SHA-256 deterministically. - Verify existing document integrity metadata or produce a deterministic proposed metadata result as defined by the processing contract. - Return clear machine-readable or structured errors for missing and mismatched files. - Avoid modifying committed source files. - Add tests using the approved pilot corpus and focused negative fixtures. ## Out of scope - Text or PDF extraction. - Text normalization or chunking. - Automatic network downloads. - License review automation. - Index generation. ## Dependencies - #7 — Establish the Python project and development-tooling baseline - #9 — Align metadata schemas and validation with the intake and processing contracts - #10 — Select and approve the five-document pilot corpus ## Acceptance criteria - Approved pilot files are located and verified deterministically. - A byte-size mismatch and SHA-256 mismatch are both detected. - Missing files and unsafe paths fail clearly. - Paths outside the approved repository boundary cannot be imported. - Repeated verification of unchanged files produces identical results. - No input file is modified. - Positive and negative tests pass. ## Validation - Run the complete unit-test suite. - Exercise all five approved files. - Exercise fixtures for missing files, checksum mismatches, size mismatches, absolute paths, traversal, and out-of-bound paths. - Repeat a successful verification and compare results. ## Suggested branch `feature/issue-11-file-integrity-import`

JackFrostbyte added this to the Reproducible Acquisition & Processing v0.1 milestone

2026-07-24 19:38:14 -04:00

JackFrostbyte added the

area/pipeline

type/implementation

labels

2026-07-24 19:38:14 -04:00

JackFrostbyte referenced this issue

2026-07-24 19:38:15 -04:00

Define the processing-adapter contract and implement UTF-8 plain-text ingestion #12