Define processing-adapter contract and add UTF-8 plain-text ingestion #57

Merged

JackFrostbyte merged 1 commit from feature/issue-12-plaintext-ingestion into develop

2026-08-08 02:25:23 -04:00

JackFrostbyte commented

2026-08-08 02:22:48 -04:00

Owner

Summary

Implements Issue #12 by defining Arkive's minimal processing-adapter contract and adding deterministic UTF-8 plain-text ingestion for approved originals.

The implementation builds directly on the approved-file integrity boundary introduced by Issue #11. Format processing receives the exact bytes from the same read that successfully passed integrity verification rather than reopening or independently resolving arbitrary filesystem paths.

Closes #12

Changes

add an immutable verified-document byte result to the approved-file integrity layer
add load_approved_document() while preserving the existing verify_approved_document() API and behavior
define a deliberately small shared processing-adapter contract
add immutable processing result and transformation metadata types
add stable machine-readable processing failures
implement the arkive.plain_text adapter at version 0.1.0
restrict the adapter to authoritative text/plain originals
decode UTF-8 strictly without encoding repair or replacement
preserve decoded text exactly, including whitespace and line endings
preserve a valid UTF-8 BOM as U+FEFF rather than stripping it
reject UTF-16 and UTF-32 BOM-marked input as unsupported encodings
reject malformed UTF-8 deterministically with the offending byte offset
preserve source ID, document ID, original path, byte size, SHA-256, media type, input encoding, and adapter metadata in the processing result
propagate Issue #11 verification failures unchanged
document the processing-adapter boundary and UTF-8 policy in the architecture documentation
add focused unit and integration coverage using both approved UTF-8 pilot originals

Processing boundary

The implemented flow is:

document_id
→ approved-file lookup and integrity verification
→ exact verified bytes
→ format-specific processing adapter
→ immutable processing result

Adapters do not accept arbitrary caller-supplied file paths and do not duplicate Issue #11 path-safety or integrity-verification logic.

The contract is intentionally small so a future born-digital PDF adapter can use the same boundary without introducing dynamic plugin discovery.

UTF-8 policy

Plain-text ingestion:

accepts authoritative text/plain originals
uses strict UTF-8 decoding
performs no newline normalization
performs no Unicode normalization
performs no trimming
performs no encoding repair
performs no replacement-character fallback
preserves a UTF-8 BOM as U+FEFF
rejects UTF-16 and UTF-32 BOM-marked inputs as unsupported encodings
reports malformed UTF-8 with a stable machine-readable failure

Runtime transformation metadata describes the decoding step and does not alter the persisted document transformation schema used for derived artifacts.

Scope

This PR deliberately does not implement:

text normalization
chunk generation
PDF extraction
OCR
automatic format conversion
encoding repair
dynamic plugin discovery
indexing
embeddings or RAG
local-content overlay behavior

No metadata schemas, source policies, or committed pilot originals are changed.

Validation

Complete pre-staging validation passed:

git diff --check
metadata validator: 6 sources, 6 documents, 2 chunks across 13 record files — all valid
full unit test suite: 73 tests passed
python -m ruff check src pipeline tests
python -m ruff format --check src pipeline tests
python -m pip check — no broken requirements

The tests include both approved UTF-8 plain-text pilot originals plus focused coverage for exact character/byte preservation, CRLF and whitespace preservation, UTF-8 BOM handling, unsupported BOM-marked encodings, malformed UTF-8, unsupported PDF media type, deterministic serialization, immutable results, lineage metadata, repeated ingestion, and Issue #11 verification-error propagation.

## Summary Implements Issue #12 by defining Arkive's minimal processing-adapter contract and adding deterministic UTF-8 plain-text ingestion for approved originals. The implementation builds directly on the approved-file integrity boundary introduced by Issue #11. Format processing receives the exact bytes from the same read that successfully passed integrity verification rather than reopening or independently resolving arbitrary filesystem paths. Closes #12 ## Changes * add an immutable verified-document byte result to the approved-file integrity layer * add `load_approved_document()` while preserving the existing `verify_approved_document()` API and behavior * define a deliberately small shared processing-adapter contract * add immutable processing result and transformation metadata types * add stable machine-readable processing failures * implement the `arkive.plain_text` adapter at version `0.1.0` * restrict the adapter to authoritative `text/plain` originals * decode UTF-8 strictly without encoding repair or replacement * preserve decoded text exactly, including whitespace and line endings * preserve a valid UTF-8 BOM as U+FEFF rather than stripping it * reject UTF-16 and UTF-32 BOM-marked input as unsupported encodings * reject malformed UTF-8 deterministically with the offending byte offset * preserve source ID, document ID, original path, byte size, SHA-256, media type, input encoding, and adapter metadata in the processing result * propagate Issue #11 verification failures unchanged * document the processing-adapter boundary and UTF-8 policy in the architecture documentation * add focused unit and integration coverage using both approved UTF-8 pilot originals ## Processing boundary The implemented flow is: `document_id` → approved-file lookup and integrity verification → exact verified bytes → format-specific processing adapter → immutable processing result Adapters do not accept arbitrary caller-supplied file paths and do not duplicate Issue #11 path-safety or integrity-verification logic. The contract is intentionally small so a future born-digital PDF adapter can use the same boundary without introducing dynamic plugin discovery. ## UTF-8 policy Plain-text ingestion: * accepts authoritative `text/plain` originals * uses strict UTF-8 decoding * performs no newline normalization * performs no Unicode normalization * performs no trimming * performs no encoding repair * performs no replacement-character fallback * preserves a UTF-8 BOM as U+FEFF * rejects UTF-16 and UTF-32 BOM-marked inputs as unsupported encodings * reports malformed UTF-8 with a stable machine-readable failure Runtime transformation metadata describes the decoding step and does not alter the persisted document transformation schema used for derived artifacts. ## Scope This PR deliberately does **not** implement: * text normalization * chunk generation * PDF extraction * OCR * automatic format conversion * encoding repair * dynamic plugin discovery * indexing * embeddings or RAG * local-content overlay behavior No metadata schemas, source policies, or committed pilot originals are changed. ## Validation Complete pre-staging validation passed: * `git diff --check` * metadata validator: 6 sources, 6 documents, 2 chunks across 13 record files — all valid * full unit test suite: 73 tests passed * `python -m ruff check src pipeline tests` * `python -m ruff format --check src pipeline tests` * `python -m pip check` — no broken requirements The tests include both approved UTF-8 plain-text pilot originals plus focused coverage for exact character/byte preservation, CRLF and whitespace preservation, UTF-8 BOM handling, unsupported BOM-marked encodings, malformed UTF-8, unsupported PDF media type, deterministic serialization, immutable results, lineage metadata, repeated ingestion, and Issue #11 verification-error propagation.

JackFrostbyte added 1 commit

2026-08-08 02:22:48 -04:00