Implement the end-to-end pilot processing command #16

New issue

Closed

opened 2026-07-24 19:38:17 -04:00 by JackFrostbyte · 1 comment

JackFrostbyte commented

2026-07-24 19:38:17 -04:00

Owner

Goal

    Provide one documented command that processes the approved pilot corpus through integrity verification, extraction, normalization, chunking, schema validation, and cross-record validation.

    ## Why

    The individual processing components need one reproducible orchestration path before clean-checkout regeneration and corpus review can be tested.

    ## Scope

    - Add one command-line entry point following the project convention.

Process either the complete approved pilot corpus or a selected approved source through the same deterministic stages.
Verify file integrity before extraction.
Select the appropriate plain-text or born-digital PDF adapter.
Run normalization and chunk generation.
Produce source, document, and chunk records in the approved repository layout.
Run schema and cross-record validation before treating output as successful.
Use a staging or atomic-write approach so failures do not leave partially updated committed outputs.
Provide clear stage-specific errors and a non-zero exit status on failure.
Document the command, inputs, outputs, and expected clean-checkout workflow.
```
  ## Out of scope

  - Downloading source files.
```
Source acceptance decisions.
OCR.
Index generation or search.
Embeddings, LLMs, translation, or speech.

A general workflow engine or distributed task system.

  ## Dependencies

  - #11 — Implement approved-file import and integrity verification

#12 — Define the processing-adapter contract and implement UTF-8 plain-text ingestion
#13 — Implement born-digital PDF text extraction
#14 — Implement deterministic conservative text normalization

#15 — Implement deterministic chunk generation

  ## Acceptance criteria

  - One documented command processes all approved pilot formats.

Every stage runs in the required order and validates its inputs and outputs.
A failed stage does not leave partially updated final records.
Successful output validates against schemas and cross-record rules.
The command operates without network access.

Errors identify the failed source and processing stage.

  ## Validation

  - Run the complete unit-test suite.

Run the command against each approved pilot source and against the full corpus.
Inject representative failures at integrity, extraction, normalization, chunking, and validation stages.
Confirm failed runs are non-destructive.

Run the documented workflow with networking disabled.

  ## Suggested branch

  `feature/issue-16-processing-command`

  <!-- arkive-planning-automation:2026-07-24-v3:M1-11 -->

## Goal Provide one documented command that processes the approved pilot corpus through integrity verification, extraction, normalization, chunking, schema validation, and cross-record validation. ## Why The individual processing components need one reproducible orchestration path before clean-checkout regeneration and corpus review can be tested. ## Scope - Add one command-line entry point following the project convention. - Process either the complete approved pilot corpus or a selected approved source through the same deterministic stages. - Verify file integrity before extraction. - Select the appropriate plain-text or born-digital PDF adapter. - Run normalization and chunk generation. - Produce source, document, and chunk records in the approved repository layout. - Run schema and cross-record validation before treating output as successful. - Use a staging or atomic-write approach so failures do not leave partially updated committed outputs. - Provide clear stage-specific errors and a non-zero exit status on failure. - Document the command, inputs, outputs, and expected clean-checkout workflow. ## Out of scope - Downloading source files. - Source acceptance decisions. - OCR. - Index generation or search. - Embeddings, LLMs, translation, or speech. - A general workflow engine or distributed task system. ## Dependencies - #11 — Implement approved-file import and integrity verification - #12 — Define the processing-adapter contract and implement UTF-8 plain-text ingestion - #13 — Implement born-digital PDF text extraction - #14 — Implement deterministic conservative text normalization - #15 — Implement deterministic chunk generation ## Acceptance criteria - One documented command processes all approved pilot formats. - Every stage runs in the required order and validates its inputs and outputs. - A failed stage does not leave partially updated final records. - Successful output validates against schemas and cross-record rules. - The command operates without network access. - Errors identify the failed source and processing stage. ## Validation - Run the complete unit-test suite. - Run the command against each approved pilot source and against the full corpus. - Inject representative failures at integrity, extraction, normalization, chunking, and validation stages. - Confirm failed runs are non-destructive. - Run the documented workflow with networking disabled. ## Suggested branch `feature/issue-16-processing-command`

JackFrostbyte added this to the Reproducible Acquisition & Processing v0.1 milestone

2026-07-24 19:38:17 -04:00

JackFrostbyte added the

area/pipeline

type/implementation

labels

2026-07-24 19:38:17 -04:00