Implement deterministic conservative text normalization

## Summary * Add a dedicated immutable text-normalization stage that consumes existing `ProcessingResult` output without rerunning source verification, plaintext decoding, or PDF extraction. * Normalize line endings deterministically to LF and apply Unicode NFC in a fixed documented order. * Preserve all other whitespace and extraction artifacts unless a future evidence-backed rule explicitly justifies changing them. * Normalize PDF pages independently while preserving 1-based physical page identity and reconstruct aggregate text using the existing `U+000C` page separator. * Preserve plaintext results as page-less (`pages=None`). * Retain upstream adapter transformation lineage separately from normalization transformation metadata. * Record deterministic normalizer metadata including normalizer/version, ordered rule effects, Unicode normalization form, Unicode database version, line-ending policy, and explicit no-op whitespace/artifact policies. * Add deterministic before/after review support without introducing an end-to-end CLI. * Document the normalization contract and developer review procedure. ## Normalization contract Normalizer: * ID: `arkive.text_normalization` * version: `0.1.0` Rules, in order: 1. Canonicalize `CRLF` and bare `CR` line endings to `LF`. 2. Apply Unicode NFC normalization. The v0.1 contract deliberately does **not**: * strip leading or trailing whitespace; * collapse repeated spaces; * convert tabs; * collapse blank lines; * convert non-breaking spaces or other Unicode whitespace; * repair PDF line-break hyphenation; * repair split URLs; * remove page numbers, headers, footers, table material, or other extraction artifacts; * perform semantic rewriting, OCR cleanup, chunking, or layout reconstruction. ## Page and lineage behavior For page-aware PDF processing results: * each physical page is normalized independently; * page numbers and ordering are preserved; * empty pages remain represented; * normalized aggregate text is deterministically reconstructed from normalized pages with `U+000C`. For plaintext processing results: * `pages=None` remains unchanged; * no artificial page/location data is introduced. Existing adapter transformation metadata is retained separately from the normalization transformation. No source originals, metadata records, schemas, policies, adapters, extraction behavior, or dependencies are changed. ## Pilot-corpus review All five approved pilot documents were processed using the existing Issue #11/#12/#13 pipeline and manually reviewed before and after normalization. Results: * all five current pilot adapter outputs are already LF-only; * all five current pilot outputs are already NFC; * all five are therefore true normalization no-op cases under v0.1; * original file SHA-256 values remained unchanged; * before/after normalized text hashes and lengths are identical; * both plaintext pilots remain page-less; * all PDF page counts, 1-based physical page numbers, page ordering, page text, and aggregate page boundaries remain unchanged; * repeated normalization and serialized review output are deterministic; * normalization is idempotent for focused fixtures and all five pilots. The pilot corpus also demonstrates why broader cleanup is intentionally excluded: current PDF extraction contains potentially meaningful spacing, line-break hyphenation, split URLs, page material, and layout artifacts that cannot safely be removed generically. ## Verification * [x] Issue #14 focused normalization tests — 22 passed * [x] Issue #11 integrity tests — 37 passed * [x] Issue #12 processing/plaintext tests — 17 passed * [x] Issue #13 PDF tests — 12 passed * [x] Complete `python -m unittest discover -s tests -v` — 164 passed * [x] `python pipeline/validate/validate_records.py` * [x] `python -m ruff check src pipeline tests` * [x] `python -m ruff format --check src pipeline tests` * [x] `git diff --check` * [x] `python -m pip check` * [x] Five-pilot before/after manual review * [x] Repeated normalization comparison * [x] Idempotence validation * [x] PDF physical-page association review * [x] Original pilot hashes verified unchanged * [x] Complete staged diff reviewed ## Scope confirmation * [x] No source originals or metadata records changed * [x] No schema or policy changes * [x] No new dependency * [x] No plaintext ingestion changes * [x] No PDF extraction changes * [x] No chunk generation * [x] No end-to-end orchestration * [x] No OCR, semantic rewriting, translation, indexing, embeddings, or RAG work * [x] No unrelated cleanup or refactoring ## Linked issue Closes #14

JackFrostbyte commented

2026-08-10 02:46:17 -04:00

Owner

Summary

Add a dedicated immutable text-normalization stage that consumes existing ProcessingResult output without rerunning source verification, plaintext decoding, or PDF extraction.
Normalize line endings deterministically to LF and apply Unicode NFC in a fixed documented order.
Preserve all other whitespace and extraction artifacts unless a future evidence-backed rule explicitly justifies changing them.
Normalize PDF pages independently while preserving 1-based physical page identity and reconstruct aggregate text using the existing U+000C page separator.
Preserve plaintext results as page-less (pages=None).
Retain upstream adapter transformation lineage separately from normalization transformation metadata.
Record deterministic normalizer metadata including normalizer/version, ordered rule effects, Unicode normalization form, Unicode database version, line-ending policy, and explicit no-op whitespace/artifact policies.
Add deterministic before/after review support without introducing an end-to-end CLI.
Document the normalization contract and developer review procedure.

Normalization contract

Normalizer:

ID: arkive.text_normalization
version: 0.1.0

Rules, in order:

Canonicalize CRLF and bare CR line endings to LF.
Apply Unicode NFC normalization.

The v0.1 contract deliberately does not:

strip leading or trailing whitespace;
collapse repeated spaces;
convert tabs;
collapse blank lines;
convert non-breaking spaces or other Unicode whitespace;
repair PDF line-break hyphenation;
repair split URLs;
remove page numbers, headers, footers, table material, or other extraction artifacts;
perform semantic rewriting, OCR cleanup, chunking, or layout reconstruction.

Page and lineage behavior

For page-aware PDF processing results:

each physical page is normalized independently;
page numbers and ordering are preserved;
empty pages remain represented;
normalized aggregate text is deterministically reconstructed from normalized pages with U+000C.

For plaintext processing results:

pages=None remains unchanged;
no artificial page/location data is introduced.

Existing adapter transformation metadata is retained separately from the normalization transformation.

No source originals, metadata records, schemas, policies, adapters, extraction behavior, or dependencies are changed.

Pilot-corpus review

All five approved pilot documents were processed using the existing Issue #11/#12/#13 pipeline and manually reviewed before and after normalization.

Results:

all five current pilot adapter outputs are already LF-only;
all five current pilot outputs are already NFC;
all five are therefore true normalization no-op cases under v0.1;
original file SHA-256 values remained unchanged;
before/after normalized text hashes and lengths are identical;
both plaintext pilots remain page-less;
all PDF page counts, 1-based physical page numbers, page ordering, page text, and aggregate page boundaries remain unchanged;
repeated normalization and serialized review output are deterministic;
normalization is idempotent for focused fixtures and all five pilots.

The pilot corpus also demonstrates why broader cleanup is intentionally excluded: current PDF extraction contains potentially meaningful spacing, line-break hyphenation, split URLs, page material, and layout artifacts that cannot safely be removed generically.