Arkive/arkive
1
0
Fork 0
Code Issues 18 Pull requests 1 Projects Releases Packages Wiki Activity Actions

Implement deterministic chunk generation #15

New issue
Closed
opened 2026-07-24 19:38:16 -04:00 by JackFrostbyte · 1 comment
JackFrostbyte commented 2026-07-24 19:38:16 -04:00
Owner
Copy link

Goal

    Generate deterministic chunk records with stable identifiers, exact text, valid half-open character offsets, preserved lineage, and available citation-location metadata.

    ## Why

    Chunks are the common unit for direct search and future retrieval. They must be reproducible and resolve exactly to normalized document text before any index is built.

    ## Scope

    - Define a deterministic chunking strategy suitable for the pilot corpus and later lexical or vector retrieval.

  • Generate stable chunk IDs from documented stable inputs.

  • Preserve source ID and document ID lineage.

  • Produce deterministic sequence ordering.

  • Store exact chunk text and verified half-open character offsets into the normalized parent text.

  • Preserve page, section, heading, or equivalent location information when available.

  • Define and test behavior at headings, page boundaries, long sections, short documents, and empty or whitespace-only regions.

  • Reject overlapping, reversed, out-of-range, duplicate, or otherwise invalid chunk locations.

  • Detect silently omitted non-whitespace content.

  • Do not add embeddings.

      ## Out of scope

  - Vector embeddings or similarity search.

  • LLM-based semantic chunking.

  • Summarization or rewriting.

  • Index generation.

  • Safety-critical content-specific chunking.

      ## Dependencies

  - #9 — Align metadata schemas and validation with the intake and processing contracts

  • #14 — Implement deterministic conservative text normalization

      ## Acceptance criteria

  - Repeated chunking of identical normalized input produces byte-identical records.

  • Every chunk's text exactly matches its declared half-open offsets.

  • Chunk IDs and sequence ordering are stable.

  • All chunks preserve valid source and document lineage.

  • Available page, section, and heading metadata remains correctly associated.

  • Invalid or overlapping offsets are rejected.

  • The process detects unintended loss of substantive normalized content.

  • All records validate against the metadata schemas and cross-record rules.

      ## Validation

  - Run the complete unit-test suite.

  • Reconstruct or coverage-check normalized text according to the documented chunking contract.

  • Verify every chunk text against its declared offsets.

  • Exercise heading, page-boundary, long-section, short-document, empty-region, overlap, and out-of-range fixtures.

  • Repeat chunk generation and compare serialized output.

      ## Suggested branch

  `feature/issue-15-chunk-generation`

  <!-- arkive-planning-automation:2026-07-24-v3:M1-10 -->
## Goal Generate deterministic chunk records with stable identifiers, exact text, valid half-open character offsets, preserved lineage, and available citation-location metadata. ## Why Chunks are the common unit for direct search and future retrieval. They must be reproducible and resolve exactly to normalized document text before any index is built. ## Scope - Define a deterministic chunking strategy suitable for the pilot corpus and later lexical or vector retrieval. - Generate stable chunk IDs from documented stable inputs. - Preserve source ID and document ID lineage. - Produce deterministic sequence ordering. - Store exact chunk text and verified half-open character offsets into the normalized parent text. - Preserve page, section, heading, or equivalent location information when available. - Define and test behavior at headings, page boundaries, long sections, short documents, and empty or whitespace-only regions. - Reject overlapping, reversed, out-of-range, duplicate, or otherwise invalid chunk locations. - Detect silently omitted non-whitespace content. - Do not add embeddings. ## Out of scope - Vector embeddings or similarity search. - LLM-based semantic chunking. - Summarization or rewriting. - Index generation. - Safety-critical content-specific chunking. ## Dependencies - #9 — Align metadata schemas and validation with the intake and processing contracts - #14 — Implement deterministic conservative text normalization ## Acceptance criteria - Repeated chunking of identical normalized input produces byte-identical records. - Every chunk's text exactly matches its declared half-open offsets. - Chunk IDs and sequence ordering are stable. - All chunks preserve valid source and document lineage. - Available page, section, and heading metadata remains correctly associated. - Invalid or overlapping offsets are rejected. - The process detects unintended loss of substantive normalized content. - All records validate against the metadata schemas and cross-record rules. ## Validation - Run the complete unit-test suite. - Reconstruct or coverage-check normalized text according to the documented chunking contract. - Verify every chunk text against its declared offsets. - Exercise heading, page-boundary, long-section, short-document, empty-region, overlap, and out-of-range fixtures. - Repeat chunk generation and compare serialized output. ## Suggested branch `feature/issue-15-chunk-generation` <!-- arkive-planning-automation:2026-07-24-v3:M1-10 -->
JackFrostbyte commented 2026-08-15 01:58:45 -04:00
Author
Owner
Copy link

Issue #15 is complete via PR #63.

Deterministic chunk generation is now implemented on top of the normalized-text contract from Issue #14, including stable chunk identity, exact offsets/text, deterministic sequencing and serialization, PDF page boundaries, explicit distribution scope, substantive-content coverage validation, and schema/cross-record validation.

The implementation was validated against all five pilot documents and the full repository test suite.

Closing as completed.

Issue #15 is complete via PR #63. Deterministic chunk generation is now implemented on top of the normalized-text contract from Issue #14, including stable chunk identity, exact offsets/text, deterministic sequencing and serialization, PDF page boundaries, explicit distribution scope, substantive-content coverage validation, and schema/cross-record validation. The implementation was validated against all five pilot documents and the full repository test suite. Closing as completed.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feature/issue-17-reproducibility-tests
develop
No results found.
Labels
Clear labels
  
area/deployment
Portable packaging and constrained-environment validation

  
area/governance
Project rules, policies, contributor workflow

  
area/metadata
Schemas, manifests, validation contracts

  
area/pipeline
Import, extraction, normalization, chunking

  
area/search
Indexing, retrieval, citation resolution

  
type/data
Source selection, metadata, fixtures, and corpus work

  
type/design
Architecture, policy, contracts, and ADR work

  
type/implementation
Production code or tooling

  
type/testing
Reproducibility, integration, evaluation, and benchmark work

No labels
area/deployment
area/governance
area/metadata
area/pipeline
area/search
type/data
type/design
type/implementation
type/testing
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
JackFrostbyte
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks
#16 Implement the end-to-end pilot processing command
Arkive/arkive
Depends on
#9 Align metadata schemas and validation with the intake and processing contracts
Arkive/arkive
#14 Implement deterministic conservative text normalization
Arkive/arkive
Reference
Arkive/arkive#15
No description provided.