Arkive/arkive
1
0
Fork 0
Code Issues 18 Pull requests 1 Projects Releases Packages Wiki Activity Actions

Implement deterministic conservative text normalization #14

New issue
Closed
opened 2026-07-24 19:38:16 -04:00 by JackFrostbyte · 1 comment
JackFrostbyte commented 2026-07-24 19:38:16 -04:00
Owner
Copy link

Goal

    Implement a conservative, traceable normalization stage that produces deterministic text without silently changing substantive source meaning.

    ## Why

    Plain-text and PDF adapters can produce different newline, Unicode, whitespace, and extraction-artifact patterns. Arkive needs one documented normalization contract before deterministic chunking.

    ## Scope

    - Define the normalization input and output contract.

  • Normalize line endings deterministically.

  • Select and document an appropriate Unicode normalization form.

  • Apply predictable whitespace cleanup only where it does not alter substantive meaning.

  • Remove only explicitly identified and tested extraction artifacts.

  • Preserve page, section, heading, and source-location associations received from adapters.

  • Record each transformation and its implementation/version in document lineage metadata.

  • Provide a way to compare pre-normalized and normalized text during review.

  • Add positive, edge-case, and no-op regression fixtures.

      ## Out of scope

  - Semantic rewriting, summarization, spelling correction, or translation.

  • OCR cleanup.

  • Chunk generation.

  • Document-specific hand editing hidden inside the normalizer.

  • Lossy removal of content because it appears unimportant.

      ## Dependencies

  - #12 — Define the processing-adapter contract and implement UTF-8 plain-text ingestion

  • #13 — Implement born-digital PDF text extraction

      ## Acceptance criteria

  - The normalization rules and their order are documented.

  • Unchanged input produces stable, deterministic normalized text.

  • Substantive characters and source meaning are not silently rewritten.

  • Every transformation is represented in lineage metadata.

  • Location metadata remains associated with the correct normalized content.

  • No-op cases remain unchanged under the documented contract.

  • Regression tests cover each normalization rule and edge case.

      ## Validation

  - Run the complete unit-test suite.

  • Compare before/after fixtures for every normalization rule.

  • Manually review normalized output from all five pilot documents.

  • Repeat normalization and compare serialized output.

  • Verify that no content is silently omitted.

      ## Suggested branch

  `feature/issue-14-text-normalization`

  <!-- arkive-planning-automation:2026-07-24-v3:M1-09 -->
## Goal Implement a conservative, traceable normalization stage that produces deterministic text without silently changing substantive source meaning. ## Why Plain-text and PDF adapters can produce different newline, Unicode, whitespace, and extraction-artifact patterns. Arkive needs one documented normalization contract before deterministic chunking. ## Scope - Define the normalization input and output contract. - Normalize line endings deterministically. - Select and document an appropriate Unicode normalization form. - Apply predictable whitespace cleanup only where it does not alter substantive meaning. - Remove only explicitly identified and tested extraction artifacts. - Preserve page, section, heading, and source-location associations received from adapters. - Record each transformation and its implementation/version in document lineage metadata. - Provide a way to compare pre-normalized and normalized text during review. - Add positive, edge-case, and no-op regression fixtures. ## Out of scope - Semantic rewriting, summarization, spelling correction, or translation. - OCR cleanup. - Chunk generation. - Document-specific hand editing hidden inside the normalizer. - Lossy removal of content because it appears unimportant. ## Dependencies - #12 — Define the processing-adapter contract and implement UTF-8 plain-text ingestion - #13 — Implement born-digital PDF text extraction ## Acceptance criteria - The normalization rules and their order are documented. - Unchanged input produces stable, deterministic normalized text. - Substantive characters and source meaning are not silently rewritten. - Every transformation is represented in lineage metadata. - Location metadata remains associated with the correct normalized content. - No-op cases remain unchanged under the documented contract. - Regression tests cover each normalization rule and edge case. ## Validation - Run the complete unit-test suite. - Compare before/after fixtures for every normalization rule. - Manually review normalized output from all five pilot documents. - Repeat normalization and compare serialized output. - Verify that no content is silently omitted. ## Suggested branch `feature/issue-14-text-normalization` <!-- arkive-planning-automation:2026-07-24-v3:M1-09 -->
JackFrostbyte commented 2026-08-10 02:50:24 -04:00
Author
Owner
Copy link

Issue completed by merged PR #61.

Implemented:

  • Added the deterministic arkive.text_normalization v0.1.0 stage operating on existing ProcessingResult output.
  • Canonicalized CRLF and bare CR line endings to LF in a fixed first step.
  • Applied Unicode NFC normalization as the second normalization rule.
  • Preserved all other whitespace, Unicode compatibility distinctions, and extraction artifacts where no generic safe transformation could be justified.
  • Preserved upstream processing lineage separately from normalization transformation metadata.
  • Recorded deterministic normalization metadata including normalizer/version, ordered rule effects, Unicode normalization form/database version, line-ending policy, and explicit no-op whitespace/artifact policies.
  • Preserved plaintext results as page-less.
  • Normalized PDF pages independently while retaining 1-based physical page identity, ordering, empty-page positions, and deterministic U+000C aggregate page boundaries.
  • Added deterministic before/after review support and explicit idempotence coverage.
  • Documented the normalization contract and developer review procedure.

Verified:

  • Issue #14 normalization tests: 22 passed.
  • Issue #11 integrity tests: 37 passed.
  • Issue #12 processing/plaintext tests: 17 passed.
  • Issue #13 PDF tests: 12 passed.
  • Complete repository suite: 164 tests passed.
  • Metadata validator passed.
  • Ruff lint and formatting checks passed.
  • git diff --check passed.
  • python -m pip check passed.
  • All five approved pilot documents were manually reviewed before and after normalization.
  • All five pilots are true normalization no-ops under the v0.1 policy.
  • Original pilot hashes remained unchanged.
  • Plaintext page-less behavior and PDF physical-page associations remained unchanged.
  • Repeated normalization and review serialization are deterministic.
  • Normalization is idempotent across focused fixtures and all five pilots.

The intentionally conservative v0.1 implementation does not perform semantic rewriting, PDF cleanup/layout reconstruction, dehyphenation, whitespace collapsing, OCR cleanup, chunk generation, indexing, or orchestration. No schemas, policies, source metadata, originals, adapters, or dependencies were changed.

Issue #15 can now proceed with deterministic chunk generation against the normalized-text contract, and Issue #16 can subsequently integrate normalization into the end-to-end pilot processing workflow.

Closing #14 as completed.

Issue completed by merged PR #61. Implemented: * Added the deterministic `arkive.text_normalization` v0.1.0 stage operating on existing `ProcessingResult` output. * Canonicalized CRLF and bare CR line endings to LF in a fixed first step. * Applied Unicode NFC normalization as the second normalization rule. * Preserved all other whitespace, Unicode compatibility distinctions, and extraction artifacts where no generic safe transformation could be justified. * Preserved upstream processing lineage separately from normalization transformation metadata. * Recorded deterministic normalization metadata including normalizer/version, ordered rule effects, Unicode normalization form/database version, line-ending policy, and explicit no-op whitespace/artifact policies. * Preserved plaintext results as page-less. * Normalized PDF pages independently while retaining 1-based physical page identity, ordering, empty-page positions, and deterministic `U+000C` aggregate page boundaries. * Added deterministic before/after review support and explicit idempotence coverage. * Documented the normalization contract and developer review procedure. Verified: * Issue #14 normalization tests: 22 passed. * Issue #11 integrity tests: 37 passed. * Issue #12 processing/plaintext tests: 17 passed. * Issue #13 PDF tests: 12 passed. * Complete repository suite: 164 tests passed. * Metadata validator passed. * Ruff lint and formatting checks passed. * `git diff --check` passed. * `python -m pip check` passed. * All five approved pilot documents were manually reviewed before and after normalization. * All five pilots are true normalization no-ops under the v0.1 policy. * Original pilot hashes remained unchanged. * Plaintext page-less behavior and PDF physical-page associations remained unchanged. * Repeated normalization and review serialization are deterministic. * Normalization is idempotent across focused fixtures and all five pilots. The intentionally conservative v0.1 implementation does not perform semantic rewriting, PDF cleanup/layout reconstruction, dehyphenation, whitespace collapsing, OCR cleanup, chunk generation, indexing, or orchestration. No schemas, policies, source metadata, originals, adapters, or dependencies were changed. Issue #15 can now proceed with deterministic chunk generation against the normalized-text contract, and Issue #16 can subsequently integrate normalization into the end-to-end pilot processing workflow. Closing #14 as completed.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feature/issue-17-reproducibility-tests
develop
No results found.
Labels
Clear labels
  
area/deployment
Portable packaging and constrained-environment validation

  
area/governance
Project rules, policies, contributor workflow

  
area/metadata
Schemas, manifests, validation contracts

  
area/pipeline
Import, extraction, normalization, chunking

  
area/search
Indexing, retrieval, citation resolution

  
type/data
Source selection, metadata, fixtures, and corpus work

  
type/design
Architecture, policy, contracts, and ADR work

  
type/implementation
Production code or tooling

  
type/testing
Reproducibility, integration, evaluation, and benchmark work

No labels
area/deployment
area/governance
area/metadata
area/pipeline
area/search
type/data
type/design
type/implementation
type/testing
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
JackFrostbyte
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks
#15 Implement deterministic chunk generation
Arkive/arkive
#16 Implement the end-to-end pilot processing command
Arkive/arkive
Depends on
#12 Define the processing-adapter contract and implement UTF-8 plain-text ingestion
Arkive/arkive
#13 Implement born-digital PDF text extraction
Arkive/arkive
Reference
Arkive/arkive#14
No description provided.