Define the processing-adapter contract and implement UTF-8 plain-text ingestion #12

New issue

Closed

opened 2026-07-24 19:38:15 -04:00 by JackFrostbyte · 1 comment

JackFrostbyte commented

2026-07-24 19:38:15 -04:00

Owner

Goal

    Define a minimal format-adapter boundary and implement deterministic ingestion of approved UTF-8 plain-text files.

    ## Why

    Arkive needs a small, explicit processing contract before adding format-specific extraction. Plain text provides the simplest implementation against which later PDF behavior can be compared.

    ## Scope

    - Define the input and output contract shared by format-specific processing adapters.

Carry source ID, document ID, original file identity, media type, and transformation metadata through the adapter result.
Implement strict or explicitly documented UTF-8 decoding behavior.
Preserve the input text exactly at this ingestion stage.
Record encoding assumptions and the adapter implementation/version used.
Produce deterministic output and deterministic errors.
Reject unsupported encodings or malformed input according to the documented policy.
Add tests using approved plain-text pilot files and focused invalid fixtures.
Keep the interface small enough that a born-digital PDF adapter can implement it without a plugin framework.
```
  ## Out of scope

  - Text normalization.
```
Chunk generation.
PDF extraction.
OCR.
Automatic format conversion or repair.

A dynamic plugin discovery system.

  ## Dependencies

  - #7 — Establish the Python project and development-tooling baseline

#9 — Align metadata schemas and validation with the intake and processing contracts

#11 — Implement approved-file import and integrity verification

  ## Acceptance criteria

  - The adapter contract is documented and exercised by the plain-text adapter.

Valid UTF-8 pilot files preserve exact text and identity.
Source and document lineage is present in the adapter output.
Encoding and transformation metadata are explicit.
Unsupported or malformed text fails clearly.

Repeated ingestion produces identical output.

  ## Validation

  - Run the complete unit-test suite.

Compare ingested text byte-for-byte or character-for-character with approved UTF-8 inputs as appropriate.
Exercise malformed UTF-8 and unsupported-encoding fixtures.

Repeat ingestion and compare serialized results.

  ## Suggested branch

  `feature/issue-12-plaintext-ingestion`

  <!-- arkive-planning-automation:2026-07-24-v3:M1-07 -->

## Goal Define a minimal format-adapter boundary and implement deterministic ingestion of approved UTF-8 plain-text files. ## Why Arkive needs a small, explicit processing contract before adding format-specific extraction. Plain text provides the simplest implementation against which later PDF behavior can be compared. ## Scope - Define the input and output contract shared by format-specific processing adapters. - Carry source ID, document ID, original file identity, media type, and transformation metadata through the adapter result. - Implement strict or explicitly documented UTF-8 decoding behavior. - Preserve the input text exactly at this ingestion stage. - Record encoding assumptions and the adapter implementation/version used. - Produce deterministic output and deterministic errors. - Reject unsupported encodings or malformed input according to the documented policy. - Add tests using approved plain-text pilot files and focused invalid fixtures. - Keep the interface small enough that a born-digital PDF adapter can implement it without a plugin framework. ## Out of scope - Text normalization. - Chunk generation. - PDF extraction. - OCR. - Automatic format conversion or repair. - A dynamic plugin discovery system. ## Dependencies - #7 — Establish the Python project and development-tooling baseline - #9 — Align metadata schemas and validation with the intake and processing contracts - #11 — Implement approved-file import and integrity verification ## Acceptance criteria - The adapter contract is documented and exercised by the plain-text adapter. - Valid UTF-8 pilot files preserve exact text and identity. - Source and document lineage is present in the adapter output. - Encoding and transformation metadata are explicit. - Unsupported or malformed text fails clearly. - Repeated ingestion produces identical output. ## Validation - Run the complete unit-test suite. - Compare ingested text byte-for-byte or character-for-character with approved UTF-8 inputs as appropriate. - Exercise malformed UTF-8 and unsupported-encoding fixtures. - Repeat ingestion and compare serialized results. ## Suggested branch `feature/issue-12-plaintext-ingestion`

JackFrostbyte added this to the Reproducible Acquisition & Processing v0.1 milestone

2026-07-24 19:38:15 -04:00

JackFrostbyte added the

area/pipeline

type/implementation

labels

2026-07-24 19:38:15 -04:00

JackFrostbyte referenced this issue

2026-07-24 19:38:16 -04:00

Implement born-digital PDF text extraction #13

JackFrostbyte referenced this issue

2026-07-24 19:38:17 -04:00

Implement the end-to-end pilot processing command #16