Implement deterministic chunk generation #63

Merged

JackFrostbyte merged 1 commit from feature/issue-15-chunk-generation into develop

2026-08-15 01:58:15 -04:00

JackFrostbyte commented

2026-08-15 01:48:52 -04:00

Owner

Summary

Implements deterministic chunk generation for Issue #15 on top of the normalized-text contract introduced by Issue #14.

The implementation adds a dedicated chunk-generation stage that consumes NormalizationResult without reopening, re-extracting, or re-normalizing source content.

Implementation

Add src/arkive/chunk_generation.py.
Generate immutable chunk records with:
- stable chunk_id
- authoritative document_id / source_id lineage
- zero-based contiguous sequence numbers
- exact half-open [char_start, char_end) offsets
- exact source text slices
- explicit distribution_scope
- physical PDF page location where applicable
Use a deterministic 2,000-character maximum chunk size.
Preserve complete LF-delimited lines when possible.
Hard-split only individual lines longer than the maximum.
Treat PDF physical pages as hard chunk boundaries.
Do not invent heading or section metadata.
Allow uncovered whitespace-only spans while rejecting omission of substantive content.
Generate stable IDs from committed document identity plus sequence:
chk_<document-key>-<six-digit-sequence>.
Add deterministic JSON serialization.
Add generator-level validation for offsets, overlap, duplicate ranges/IDs, sequence, lineage, scope, exact text, page locations, and substantive coverage.

No schema changes, dependency changes, or generic metadata-validator changes are required.

Tests

Added comprehensive chunk-generation tests covering:

deterministic generation and serialization
exact normalized-text offsets
stable IDs and sequence numbers
line-boundary behavior and long-line splitting
empty and whitespace-only inputs
PDF page boundaries and empty pages
distribution-scope and lineage enforcement
overlap, duplicate, invalid-range, and omission rejection
all five pilot documents
generated chunk compatibility with the existing schema and cross-record validator

Final validation:

207 unit tests passed
pipeline/validate/validate_records.py passed
full Ruff lint passed
full Ruff format check passed
git diff --check passed
pip check passed
test-harness compatibility check passed

Documentation

Updates the architecture overview with the implemented v0.1 chunk-generation contract and removes chunk generation from the future-work list.

Closes #15

## Summary Implements deterministic chunk generation for Issue #15 on top of the normalized-text contract introduced by Issue #14. The implementation adds a dedicated chunk-generation stage that consumes `NormalizationResult` without reopening, re-extracting, or re-normalizing source content. ## Implementation - Add `src/arkive/chunk_generation.py`. - Generate immutable chunk records with: - stable `chunk_id` - authoritative `document_id` / `source_id` lineage - zero-based contiguous sequence numbers - exact half-open `[char_start, char_end)` offsets - exact source text slices - explicit `distribution_scope` - physical PDF page location where applicable - Use a deterministic 2,000-character maximum chunk size. - Preserve complete LF-delimited lines when possible. - Hard-split only individual lines longer than the maximum. - Treat PDF physical pages as hard chunk boundaries. - Do not invent heading or section metadata. - Allow uncovered whitespace-only spans while rejecting omission of substantive content. - Generate stable IDs from committed document identity plus sequence: `chk_<document-key>-<six-digit-sequence>`. - Add deterministic JSON serialization. - Add generator-level validation for offsets, overlap, duplicate ranges/IDs, sequence, lineage, scope, exact text, page locations, and substantive coverage. No schema changes, dependency changes, or generic metadata-validator changes are required. ## Tests Added comprehensive chunk-generation tests covering: - deterministic generation and serialization - exact normalized-text offsets - stable IDs and sequence numbers - line-boundary behavior and long-line splitting - empty and whitespace-only inputs - PDF page boundaries and empty pages - distribution-scope and lineage enforcement - overlap, duplicate, invalid-range, and omission rejection - all five pilot documents - generated chunk compatibility with the existing schema and cross-record validator Final validation: - `207` unit tests passed - `pipeline/validate/validate_records.py` passed - full Ruff lint passed - full Ruff format check passed - `git diff --check` passed - `pip check` passed - test-harness compatibility check passed ## Documentation Updates the architecture overview with the implemented v0.1 chunk-generation contract and removes chunk generation from the future-work list. Closes #15

JackFrostbyte added 1 commit

2026-08-15 01:48:52 -04:00