Reproducible Acquisition & Processing v0.1
Goal
Take a small, legally redistributable, non-safety-critical pilot corpus from reviewed source files to validated and reproducible source, document, and chunk records.
The milestone covers five pilot documents, with plain text and born-digital PDF as the first supported formats.
Completion criteria
- Five approved, legally redistributable, non-safety-critical documents are committed.
- Every source, document, and chunk record validates.
- File size, SHA-256 checksums, lineage, and chunk offsets are verifiable.
- The processing pipeline is deterministic.
- A clean checkout can regenerate the committed processed output using documented commands.
- Failures are clear and non-destructive.
- The existing Arkive-created metadata example remains a regression fixture and does not count as one of the five pilot documents.
Explicitly out of scope
- OCR or scanned-PDF processing
- Safety-critical source material
- Embeddings or vector indexes
- LLM integration or generated answers
- Translation or speech systems
- Large-scale scraping
- Production release bundles
No due date is assigned.
No due date
84% Completed
