Arkive/arkive
1
0
Fork 0
Code Issues 18 Pull requests 1 Projects Releases Packages Wiki Activity Actions

Implement born-digital PDF text extraction #13

New issue
Closed
opened 2026-07-24 19:38:16 -04:00 by JackFrostbyte · 2 comments
JackFrostbyte commented 2026-07-24 19:38:16 -04:00
Owner
Copy link

Goal

    Add a deterministic adapter for extracting text and citation-relevant page information from approved born-digital PDFs.

    ## Why

    The pilot corpus must exercise a common real-world format while preserving enough location information to support later exact citations.

    ## Scope

    - Evaluate and select one focused PDF text-extraction library.

  • Record the selected library and version in project dependencies and transformation metadata.

  • Implement the processing-adapter contract for born-digital PDFs.

  • Extract text while preserving page boundaries or equivalent page references required for citation.

  • Preserve original source and document identity and transformation lineage.

  • Define deterministic serialization and ordering of extracted page content.

  • Detect and clearly reject scanned PDFs without usable text, malformed PDFs, encrypted PDFs, and unsupported cases.

  • Add tests using the legally redistributable pilot PDFs and purpose-built failure fixtures.

  • Document known extraction limitations without silently repairing source meaning.

      ## Out of scope

  - OCR.

  • Layout reconstruction intended to reproduce the visual PDF.

  • Table understanding, image extraction, or figure interpretation.

  • Text normalization or chunk generation.

  • Support for every PDF producer or malformed file.

      ## Dependencies

  - #10 — Select and approve the five-document pilot corpus

  • #12 — Define the processing-adapter contract and implement UTF-8 plain-text ingestion

      ## Acceptance criteria

  - Every supported pilot PDF produces deterministic text and page references.

  • The extraction library and version are recorded.

  • Transformation metadata links the extraction to the correct document.

  • Scanned, malformed, encrypted, and otherwise unsupported PDFs fail clearly.

  • Repeated extraction of unchanged files produces identical serialized output.

  • No original PDF is modified.

      ## Validation

  - Run the complete unit-test suite.

  • Extract every approved pilot PDF and manually compare representative passages and page boundaries to the originals.

  • Exercise scanned/no-text, malformed, encrypted, and unsupported PDF fixtures.

  • Repeat extraction and compare outputs.

      ## Suggested branch

  `feature/issue-13-pdf-extraction`

  <!-- arkive-planning-automation:2026-07-24-v3:M1-08 -->
## Goal Add a deterministic adapter for extracting text and citation-relevant page information from approved born-digital PDFs. ## Why The pilot corpus must exercise a common real-world format while preserving enough location information to support later exact citations. ## Scope - Evaluate and select one focused PDF text-extraction library. - Record the selected library and version in project dependencies and transformation metadata. - Implement the processing-adapter contract for born-digital PDFs. - Extract text while preserving page boundaries or equivalent page references required for citation. - Preserve original source and document identity and transformation lineage. - Define deterministic serialization and ordering of extracted page content. - Detect and clearly reject scanned PDFs without usable text, malformed PDFs, encrypted PDFs, and unsupported cases. - Add tests using the legally redistributable pilot PDFs and purpose-built failure fixtures. - Document known extraction limitations without silently repairing source meaning. ## Out of scope - OCR. - Layout reconstruction intended to reproduce the visual PDF. - Table understanding, image extraction, or figure interpretation. - Text normalization or chunk generation. - Support for every PDF producer or malformed file. ## Dependencies - #10 — Select and approve the five-document pilot corpus - #12 — Define the processing-adapter contract and implement UTF-8 plain-text ingestion ## Acceptance criteria - Every supported pilot PDF produces deterministic text and page references. - The extraction library and version are recorded. - Transformation metadata links the extraction to the correct document. - Scanned, malformed, encrypted, and otherwise unsupported PDFs fail clearly. - Repeated extraction of unchanged files produces identical serialized output. - No original PDF is modified. ## Validation - Run the complete unit-test suite. - Extract every approved pilot PDF and manually compare representative passages and page boundaries to the originals. - Exercise scanned/no-text, malformed, encrypted, and unsupported PDF fixtures. - Repeat extraction and compare outputs. ## Suggested branch `feature/issue-13-pdf-extraction` <!-- arkive-planning-automation:2026-07-24-v3:M1-08 -->
JackFrostbyte commented 2026-08-02 00:35:55 -04:00
Author
Owner
Copy link

PDF tooling discovered during Issue #10

During selection and review of the five-document pilot corpus, the following external PDF tools were used successfully:

  • qpdf:

    • structural and stream validation through qpdf --check;
    • detection of encryption and malformed PDF structure.

  • Poppler utilities:

    • pdfinfo for document metadata and page information;
    • pdftotext for text-extraction inspection;
    • pdffonts for embedded-font review;
    • pdfimages for image inventory;
    • pdftoppm for deterministic page rendering during manual review.

Issue #13 should explicitly evaluate and record whether these tools are:

  1. production runtime requirements;
  2. development and validation requirements;
  3. optional human-review tools;
  4. or replaced by a focused Python PDF extraction library.

The implementation should also:

  • record the selected extractor and tested version;
  • document required system packages separately from Python dependencies;
  • fail clearly when a required external executable is missing;
  • record tool versions in transformation metadata where they can affect output;
  • handle qpdf warning and error exit statuses deliberately;
  • document installation and validation commands for Nobara/Linux;
  • preserve the distinction between structural validation, text extraction, rendering, and visual inspection.

No PDF tooling should be introduced as part of Issue #11 because PDF extraction is explicitly outside that issue's scope.

## PDF tooling discovered during Issue #10 During selection and review of the five-document pilot corpus, the following external PDF tools were used successfully: - qpdf: - structural and stream validation through `qpdf --check`; - detection of encryption and malformed PDF structure. - Poppler utilities: - `pdfinfo` for document metadata and page information; - `pdftotext` for text-extraction inspection; - `pdffonts` for embedded-font review; - `pdfimages` for image inventory; - `pdftoppm` for deterministic page rendering during manual review. Issue #13 should explicitly evaluate and record whether these tools are: 1. production runtime requirements; 2. development and validation requirements; 3. optional human-review tools; 4. or replaced by a focused Python PDF extraction library. The implementation should also: - record the selected extractor and tested version; - document required system packages separately from Python dependencies; - fail clearly when a required external executable is missing; - record tool versions in transformation metadata where they can affect output; - handle qpdf warning and error exit statuses deliberately; - document installation and validation commands for Nobara/Linux; - preserve the distinction between structural validation, text extraction, rendering, and visual inspection. No PDF tooling should be introduced as part of Issue #11 because PDF extraction is explicitly outside that issue's scope.
JackFrostbyte commented 2026-08-09 02:39:53 -04:00
Author
Owner
Copy link

Issue #13 is complete.

PR #58 implemented deterministic born-digital PDF text extraction on top of the existing approved-file verification and processing-adapter boundaries.

Completed requirements include:

  • selected and exactly pinned pypdf==6.15.0 as the production PDF extractor;
  • page-aware extraction using immutable 1-based physical page references;
  • deterministic page ordering and serialized output;
  • preservation of source/document/original lineage and verified SHA-256/byte-size identity;
  • explicit adapter and extraction-library version metadata;
  • stable failures for encrypted, malformed/unreadable, no-usable-text, and unsupported PDFs;
  • clear no-OCR behavior for scanned/image-only sources;
  • successful extraction and regression coverage for all three approved born-digital pilot PDFs;
  • representative page-local and page-boundary assertions;
  • repeated deterministic extraction;
  • verification that original PDFs remain unchanged;
  • documented extraction limitations without normalization or silent source repair;
  • documented qpdf/Poppler roles as development/reference tools rather than runtime dependencies;
  • documented controlled upgrade policy for the pinned PDF extractor.

Final validation passed with 142 unit tests, metadata validation, Ruff lint/format checks, git diff --check, pip check, deterministic all-three-pilot extraction, qpdf structural checks, and independent Poppler passage/page-boundary comparison.

Closing Issue #13 as completed.

Issue #13 is complete. PR #58 implemented deterministic born-digital PDF text extraction on top of the existing approved-file verification and processing-adapter boundaries. Completed requirements include: * selected and exactly pinned `pypdf==6.15.0` as the production PDF extractor; * page-aware extraction using immutable 1-based physical page references; * deterministic page ordering and serialized output; * preservation of source/document/original lineage and verified SHA-256/byte-size identity; * explicit adapter and extraction-library version metadata; * stable failures for encrypted, malformed/unreadable, no-usable-text, and unsupported PDFs; * clear no-OCR behavior for scanned/image-only sources; * successful extraction and regression coverage for all three approved born-digital pilot PDFs; * representative page-local and page-boundary assertions; * repeated deterministic extraction; * verification that original PDFs remain unchanged; * documented extraction limitations without normalization or silent source repair; * documented qpdf/Poppler roles as development/reference tools rather than runtime dependencies; * documented controlled upgrade policy for the pinned PDF extractor. Final validation passed with 142 unit tests, metadata validation, Ruff lint/format checks, `git diff --check`, `pip check`, deterministic all-three-pilot extraction, qpdf structural checks, and independent Poppler passage/page-boundary comparison. Closing Issue #13 as completed.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feature/issue-17-reproducibility-tests
develop
No results found.
Labels
Clear labels
  
area/deployment
Portable packaging and constrained-environment validation

  
area/governance
Project rules, policies, contributor workflow

  
area/metadata
Schemas, manifests, validation contracts

  
area/pipeline
Import, extraction, normalization, chunking

  
area/search
Indexing, retrieval, citation resolution

  
type/data
Source selection, metadata, fixtures, and corpus work

  
type/design
Architecture, policy, contracts, and ADR work

  
type/implementation
Production code or tooling

  
type/testing
Reproducibility, integration, evaluation, and benchmark work

No labels
area/deployment
area/governance
area/metadata
area/pipeline
area/search
type/data
type/design
type/implementation
type/testing
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
JackFrostbyte
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks
#14 Implement deterministic conservative text normalization
Arkive/arkive
#16 Implement the end-to-end pilot processing command
Arkive/arkive
Depends on
#10 Select and approve the five-document pilot corpus
Arkive/arkive
#12 Define the processing-adapter contract and implement UTF-8 plain-text ingestion
Arkive/arkive
Reference
Arkive/arkive#13
No description provided.