Implement end-to-end pilot processing command #64

Merged

JackFrostbyte merged 1 commit from feature/issue-16-processing-command into develop

2026-08-17 02:59:07 -04:00

JackFrostbyte commented

2026-08-17 02:57:50 -04:00

Owner

Closes #16

Summary

Implement the deterministic end-to-end processing command for the approved
five-document pilot corpus.

The new arkive-process-pilot entry point can process either the complete pilot
corpus or one selected approved source through the existing processing stages:

integrity verification
processing-adapter selection
plaintext or born-digital PDF extraction
conservative text normalization
deterministic chunk generation
derived document and chunk record generation
schema and cross-record validation
transactional replacement and final validation

Implementation

Add arkive-process-pilot through [project.scripts].
Add source/stage-aware structured processing errors.
Preserve the existing #11–#15 stage contracts rather than introducing a new
workflow abstraction.
Persist source-intake lineage:
- plaintext: original -> cleaned_text -> chunks
- PDF: original -> extracted_text -> cleaned_text -> chunks
Generate stable derived-document IDs using the documented source-intake
identity and collision rules.
Regenerate chunk IDs against the persisted cleaned-text document identity.
Validate candidate output against the complete current bundled core before
modifying final records.
Replace generated files atomically.
Roll back the complete logical transaction if a replacement or final
validation fails.
Reject symlinked transaction paths.
Ignore transient staging and atomic-write files.
Document command usage, outputs, failure behavior, clean-checkout operation,
and offline execution.

Generated pilot outputs

The command generates and commits the deterministic derived text and chunk
records for all five approved pilot sources.

Final canonical corpus state:

6 sources
14 documents
38 chunks
18 record files

Validation

Full test suite: 289 tests passed
Ruff lint: passed
Ruff formatting: passed
Overall coverage: 92%
src/arkive/pilot_processing.py: 84%
Canonical metadata validation: passed
pip check: passed
git diff --check: passed
Full-corpus processing: passed
Each of the five pilot sources processed individually: passed
Repeated processing is byte-for-byte deterministic
Individual-source reruns leave generated output byte-for-byte unchanged
Injected failures at integrity, extraction, normalization, chunk generation,
and validation stages are non-destructive
Write failures and failed final validation restore the previous state
Processing with Python network access disabled: passed
Offline rerun remains byte-for-byte deterministic
Post-commit processing rerun leaves the working tree clean

The branch was synchronized with the current origin/develop before the final
validation pass.

Closes #16 ## Summary Implement the deterministic end-to-end processing command for the approved five-document pilot corpus. The new `arkive-process-pilot` entry point can process either the complete pilot corpus or one selected approved source through the existing processing stages: 1. integrity verification 2. processing-adapter selection 3. plaintext or born-digital PDF extraction 4. conservative text normalization 5. deterministic chunk generation 6. derived document and chunk record generation 7. schema and cross-record validation 8. transactional replacement and final validation ## Implementation - Add `arkive-process-pilot` through `[project.scripts]`. - Add source/stage-aware structured processing errors. - Preserve the existing #11–#15 stage contracts rather than introducing a new workflow abstraction. - Persist source-intake lineage: - plaintext: `original -> cleaned_text -> chunks` - PDF: `original -> extracted_text -> cleaned_text -> chunks` - Generate stable derived-document IDs using the documented source-intake identity and collision rules. - Regenerate chunk IDs against the persisted cleaned-text document identity. - Validate candidate output against the complete current bundled core before modifying final records. - Replace generated files atomically. - Roll back the complete logical transaction if a replacement or final validation fails. - Reject symlinked transaction paths. - Ignore transient staging and atomic-write files. - Document command usage, outputs, failure behavior, clean-checkout operation, and offline execution. ## Generated pilot outputs The command generates and commits the deterministic derived text and chunk records for all five approved pilot sources. Final canonical corpus state: - 6 sources - 14 documents - 38 chunks - 18 record files ## Validation - Full test suite: 289 tests passed - Ruff lint: passed - Ruff formatting: passed - Overall coverage: 92% - `src/arkive/pilot_processing.py`: 84% - Canonical metadata validation: passed - `pip check`: passed - `git diff --check`: passed - Full-corpus processing: passed - Each of the five pilot sources processed individually: passed - Repeated processing is byte-for-byte deterministic - Individual-source reruns leave generated output byte-for-byte unchanged - Injected failures at integrity, extraction, normalization, chunk generation, and validation stages are non-destructive - Write failures and failed final validation restore the previous state - Processing with Python network access disabled: passed - Offline rerun remains byte-for-byte deterministic - Post-commit processing rerun leaves the working tree clean The branch was synchronized with the current `origin/develop` before the final validation pass.

JackFrostbyte added 1 commit

2026-08-17 02:57:50 -04:00