Implement deterministic born-digital PDF text extraction #58

Merged

JackFrostbyte merged 1 commit from feature/issue-13-pdf-extraction into develop

2026-08-09 02:39:27 -04:00

JackFrostbyte commented

2026-08-09 02:37:40 -04:00

Owner

Summary

Implements Issue #13 by adding deterministic, page-aware text extraction for approved born-digital PDF originals.

The implementation builds directly on the Issue #11 verified-byte boundary and Issue #12 processing-adapter contract. PDF processing consumes the exact bytes retained by successful integrity verification and never reopens or modifies the original source file.

Closes #13

What changed

Added arkive.pdf_text born-digital PDF processing adapter.
Added exact runtime dependency pin:
- pypdf==6.15.0
Added immutable page-aware processing results using 1-based physical PDF page numbers.
Extended ProcessingResult with optional ordered page information while preserving the existing serialized shape for non-paginated/plain-text results.
Added deterministic aggregate text serialization using a single form-feed (U+000C) between physical pages.
Added stable Arkive processing failures for:
- encrypted PDFs;
- malformed/unreadable PDFs;
- PDFs with no usable extractable text;
- otherwise unsupported PDF features.
Preserved Issue #11 verification failures unchanged.
Added documentation for:
- PDF extraction behavior and known limitations;
- the exact pypdf reproducibility pin;
- the controlled future pypdf upgrade process;
- qpdf and Poppler as developer/reference inspection tools rather than production runtime dependencies.
Updated the changelog.

Extraction behavior

The adapter:

accepts only approved application/pdf originals;
consumes VerifiedDocumentBytes;
uses PdfReader(..., strict=True);
extracts pages in physical PDF order;
records 1-based page numbers;
preserves extractor-returned page text without trimming or normalization;
joins aggregate page text with one form-feed character;
records adapter ID/version and pypdf library/version in transformation metadata;
performs no OCR, layout reconstruction, table understanding, image extraction, figure interpretation, or normalization.

A PDF containing some empty/image-only pages is accepted when at least one page contains usable extractable text. Empty pages retain their physical positions for later citation resolution.

A PDF with no usable non-whitespace extractable text fails clearly rather than silently producing an empty successful result.

Dependency choice

pypdf==6.15.0 is pinned exactly because PDF extraction output is reproducibility-sensitive.

The pin is a tested baseline, not a permanent restriction. Future pypdf upgrades should be deliberate compatibility changes validated against the pilot corpus and failure fixtures. Material extraction-behavior changes should also update the Arkive PDF adapter version so transformation lineage remains meaningful.

qpdf and Poppler remain independent development/reference tools and are not production runtime dependencies.

Pilot corpus validation

All three approved born-digital pilot PDFs were processed successfully:

doc_attributions-use-tasl-original
- 1 page
doc_summary-licensing-best-practices-for-sharing-s-original-be2e5116
- 4 pages
doc_ten-simple-rules-for-taking-advantage-of-git-and-github-original
- 11 pages

Tests verify:

source/document/original lineage;
original byte size and SHA-256;
physical page ordering;
representative passages on known pages;
representative page boundaries;
explicit adapter/library metadata;
deterministic repeated extraction;
deterministic serialized results;
preservation of original PDF bytes.

Failure coverage

Purpose-built tests cover:

encrypted PDFs;
malformed/truncated PDFs;
blank PDFs;
image-only PDFs;
whitespace-only extracted text;
mixed empty and usable pages;
unsupported media types;
unsupported extraction features;
unchanged propagation of Issue #11 verification failures.

Negative PDFs are generated deterministically in tests rather than committing opaque third-party binary fixtures.

Manual PDF validation

The three original pilot PDFs were independently checked using qpdf and Poppler.

qpdf --check confirmed for all three:

not encrypted;
no syntax or stream-encoding errors detected.

pdfinfo independently confirmed the expected page counts:

1 page;
4 pages;
11 pages.

Representative text and page boundaries were independently compared with pdftotext.

The comparison also confirmed a documented PDF-extraction limitation: text ordering within complex layouts such as the Canva metadata/table page can differ between extractors while remaining associated with the correct physical page. Issue #13 deliberately preserves extractor output rather than attempting layout reconstruction or normalization.

Validation

Final pre-staging validation:

python -m unittest discover -s tests -v
- 142 tests passed
metadata validator
- PASS: 6 sources, 6 documents, 2 chunks across 13 record files
Ruff lint
- PASS
Ruff formatting check
- PASS
git diff --check
- PASS
python -m pip check
- no broken requirements
pinned extractor sanity check
- pypdf 6.15.0
repeated extraction of all three pilot PDFs
- deterministic
qpdf structural validation
- PASS
Poppler manual passage/page-boundary comparison
- PASS

Scope

No changes were made to:

pilot PDF originals;
source/document metadata records;
metadata schemas;
acquisition policy;
metadata validator;
Issue #11 integrity implementation;
UTF-8 plain-text adapter.

No OCR, normalization, chunk generation, indexing, embeddings, layout reconstruction, or plugin discovery is introduced.

## Summary Implements Issue #13 by adding deterministic, page-aware text extraction for approved born-digital PDF originals. The implementation builds directly on the Issue #11 verified-byte boundary and Issue #12 processing-adapter contract. PDF processing consumes the exact bytes retained by successful integrity verification and never reopens or modifies the original source file. Closes #13 ## What changed * Added `arkive.pdf_text` born-digital PDF processing adapter. * Added exact runtime dependency pin: * `pypdf==6.15.0` * Added immutable page-aware processing results using 1-based physical PDF page numbers. * Extended `ProcessingResult` with optional ordered page information while preserving the existing serialized shape for non-paginated/plain-text results. * Added deterministic aggregate text serialization using a single form-feed (`U+000C`) between physical pages. * Added stable Arkive processing failures for: * encrypted PDFs; * malformed/unreadable PDFs; * PDFs with no usable extractable text; * otherwise unsupported PDF features. * Preserved Issue #11 verification failures unchanged. * Added documentation for: * PDF extraction behavior and known limitations; * the exact pypdf reproducibility pin; * the controlled future pypdf upgrade process; * qpdf and Poppler as developer/reference inspection tools rather than production runtime dependencies. * Updated the changelog. ## Extraction behavior The adapter: * accepts only approved `application/pdf` originals; * consumes `VerifiedDocumentBytes`; * uses `PdfReader(..., strict=True)`; * extracts pages in physical PDF order; * records 1-based page numbers; * preserves extractor-returned page text without trimming or normalization; * joins aggregate page text with one form-feed character; * records adapter ID/version and pypdf library/version in transformation metadata; * performs no OCR, layout reconstruction, table understanding, image extraction, figure interpretation, or normalization. A PDF containing some empty/image-only pages is accepted when at least one page contains usable extractable text. Empty pages retain their physical positions for later citation resolution. A PDF with no usable non-whitespace extractable text fails clearly rather than silently producing an empty successful result. ## Dependency choice `pypdf==6.15.0` is pinned exactly because PDF extraction output is reproducibility-sensitive. The pin is a tested baseline, not a permanent restriction. Future pypdf upgrades should be deliberate compatibility changes validated against the pilot corpus and failure fixtures. Material extraction-behavior changes should also update the Arkive PDF adapter version so transformation lineage remains meaningful. qpdf and Poppler remain independent development/reference tools and are not production runtime dependencies. ## Pilot corpus validation All three approved born-digital pilot PDFs were processed successfully: * `doc_attributions-use-tasl-original` * 1 page * `doc_summary-licensing-best-practices-for-sharing-s-original-be2e5116` * 4 pages * `doc_ten-simple-rules-for-taking-advantage-of-git-and-github-original` * 11 pages Tests verify: * source/document/original lineage; * original byte size and SHA-256; * physical page ordering; * representative passages on known pages; * representative page boundaries; * explicit adapter/library metadata; * deterministic repeated extraction; * deterministic serialized results; * preservation of original PDF bytes. ## Failure coverage Purpose-built tests cover: * encrypted PDFs; * malformed/truncated PDFs; * blank PDFs; * image-only PDFs; * whitespace-only extracted text; * mixed empty and usable pages; * unsupported media types; * unsupported extraction features; * unchanged propagation of Issue #11 verification failures. Negative PDFs are generated deterministically in tests rather than committing opaque third-party binary fixtures. ## Manual PDF validation The three original pilot PDFs were independently checked using qpdf and Poppler. `qpdf --check` confirmed for all three: * not encrypted; * no syntax or stream-encoding errors detected. `pdfinfo` independently confirmed the expected page counts: * 1 page; * 4 pages; * 11 pages. Representative text and page boundaries were independently compared with `pdftotext`. The comparison also confirmed a documented PDF-extraction limitation: text ordering within complex layouts such as the Canva metadata/table page can differ between extractors while remaining associated with the correct physical page. Issue #13 deliberately preserves extractor output rather than attempting layout reconstruction or normalization. ## Validation Final pre-staging validation: * `python -m unittest discover -s tests -v` * 142 tests passed * metadata validator * PASS: 6 sources, 6 documents, 2 chunks across 13 record files * Ruff lint * PASS * Ruff formatting check * PASS * `git diff --check` * PASS * `python -m pip check` * no broken requirements * pinned extractor sanity check * `pypdf 6.15.0` * repeated extraction of all three pilot PDFs * deterministic * qpdf structural validation * PASS * Poppler manual passage/page-boundary comparison * PASS ## Scope No changes were made to: * pilot PDF originals; * source/document metadata records; * metadata schemas; * acquisition policy; * metadata validator; * Issue #11 integrity implementation; * UTF-8 plain-text adapter. No OCR, normalization, chunk generation, indexing, embeddings, layout reconstruction, or plugin discovery is introduced.

JackFrostbyte added 1 commit

2026-08-09 02:37:40 -04:00