Arkive/arkive
1
0
Fork 0
Code Issues 18 Pull requests 1 Projects Releases Packages Wiki Activity Actions

Add clean-checkout regeneration and reproducibility tests #65

Open
McJuniorstein wants to merge 1 commit from feature/issue-17-reproducibility-tests into develop
pull from: feature/issue-17-reproducibility-tests
merge into: Arkive:develop
Conversation 0 Commits 1 Files changed 3 +562
McJuniorstein commented 2026-08-17 20:10:58 -04:00
Member
Copy link

Summary

  • Added tests/test_reproducibility.py, proving automatically that the committed pilot corpus is reproducible rather than merely internally consistent.
  • Checked every committed document record against the bytes on disk — byte_size, SHA-256, and each chunk's text against the half-open range it claims. The canonical validator compares records to other records and never opens a document file, and chunk_generation.validate_chunks compares chunk text to its parent only in memory at generation time, so a stale checksum or a drifted offset would validate cleanly today.
  • Regenerated the corpus inside an isolated copy of data/: the documented command on an intact checkout changes nothing, removing every generated artifact and rebuilding reproduces the committed bytes exactly, two consecutive runs are identical, committed identifiers are preserved, originals and source records are never rewritten, and processing one source restores only that source.
  • Covered corpus-wide failure behaviour: a failure injected into the last source of a five-source run leaves nothing on disk for the four that already succeeded, and a final-validation failure — after every write has landed — rolls the whole corpus back to its exact prior bytes.
  • Verified regeneration with socket creation disabled.
  • Documented the reproducibility tests in docs/development.md and recorded the change in CHANGELOG.md.

No production behaviour, schema, source data, or record content changed. The only new file is a test module; the two documentation edits are additive.

Design notes for review

Identifiers are authoritative input, not recomputed output. Per the constraint recorded on #17, the intake contract makes a committed identifier immutable while deriving it from metadata that may later be corrected, so re-deriving one can legitimately differ from the committed value. The tests assert identifiers are preserved, never that they are rederivable. The second constraint from that comment — checksum_sha256 participating in document identity — was resolved under #9 and no longer applies.

Nondeterministic fields: none are excluded. Generated document and chunk records carry no timestamps (chunk.created_at is schema-optional and never emitted), so every comparison is whole-file byte equality rather than a filtered structural one. docs/development.md records this and its consequence: a dependency upgrade that alters extracted or normalized output will fail these tests. That is the intended signal, handled as the reviewed compatibility change already documented for the pypdf pin.

Clean checkout is modelled by copying data/ into a temporary root rather than shelling out to git archive or git worktree. This keeps the suite hermetic — it works from an exported tarball and needs no Git available to the test runner — at the cost of not exercising checkout itself, which is Git's behaviour rather than Arkive's.

Chunk contiguity is deliberately not asserted. Committed chunks leave intentional gaps at page boundaries, so the offsets test asserts ordering, non-overlap, and in-bounds only.

One test drives main() rather than the library call, because the documentation tells a contributor to run the command, and the existing CLI tests mock the pipeline out.

Test verification

Every test was checked by breaking the property it guards and confirming it fails. Two did not, and were rewritten before this PR:

  • Single-source scoping originally ran against an intact checkout, where the correct result and the no-op result are the same bytes. It would have passed even if the source selection were ignored entirely. It now strips every source first, regenerates one, and asserts only that one came back.
  • The rollback test had the same defect: with _restore_file_snapshot stubbed out it still passed, because the run rewrote bytes that were already there. It now perturbs a committed file first, so restored bytes and regenerated bytes actually differ.

Both were confirmed to fail against the corresponding injected defect after the rewrite.

Verification

  • git diff --check
  • python pipeline/validate/validate_records.py — PASS: 6 source(s), 14 document(s), 38 chunk(s), 0 rights profile(s), 0 recipe(s) across 18 record file(s)
  • python -m unittest discover -s tests -v — 304 tests pass (289 before)
  • python -m ruff check src pipeline tests
  • python -m ruff format --check src pipeline tests
  • python -m pip check
  • Coverage re-run — repository-wide 92%, unchanged; the new tests exercise production paths that were already covered, which is expected for a regression suite
  • Rendered Markdown reviewed

Scope confirmation

  • No unrelated production behaviour changed
  • No metadata contract changed
  • No unreviewed source or generated data was added

Linked issue

Closes #17

## Summary - Added `tests/test_reproducibility.py`, proving automatically that the committed pilot corpus is reproducible rather than merely internally consistent. - Checked every committed document record against the bytes on disk — `byte_size`, SHA-256, and each chunk's `text` against the half-open range it claims. The canonical validator compares records to other records and never opens a document file, and `chunk_generation.validate_chunks` compares chunk text to its parent only in memory at generation time, so a stale checksum or a drifted offset would validate cleanly today. - Regenerated the corpus inside an isolated copy of `data/`: the documented command on an intact checkout changes nothing, removing every generated artifact and rebuilding reproduces the committed bytes exactly, two consecutive runs are identical, committed identifiers are preserved, originals and source records are never rewritten, and processing one source restores only that source. - Covered corpus-wide failure behaviour: a failure injected into the last source of a five-source run leaves nothing on disk for the four that already succeeded, and a final-validation failure — after every write has landed — rolls the whole corpus back to its exact prior bytes. - Verified regeneration with socket creation disabled. - Documented the reproducibility tests in `docs/development.md` and recorded the change in `CHANGELOG.md`. No production behaviour, schema, source data, or record content changed. The only new file is a test module; the two documentation edits are additive. ## Design notes for review **Identifiers are authoritative input, not recomputed output.** Per the constraint recorded on #17, the intake contract makes a committed identifier immutable while deriving it from metadata that may later be corrected, so re-deriving one can legitimately differ from the committed value. The tests assert identifiers are *preserved*, never that they are *rederivable*. The second constraint from that comment — `checksum_sha256` participating in document identity — was resolved under #9 and no longer applies. **Nondeterministic fields: none are excluded.** Generated document and chunk records carry no timestamps (`chunk.created_at` is schema-optional and never emitted), so every comparison is whole-file byte equality rather than a filtered structural one. `docs/development.md` records this and its consequence: a dependency upgrade that alters extracted or normalized output will fail these tests. That is the intended signal, handled as the reviewed compatibility change already documented for the `pypdf` pin. **Clean checkout is modelled by copying `data/` into a temporary root** rather than shelling out to `git archive` or `git worktree`. This keeps the suite hermetic — it works from an exported tarball and needs no Git available to the test runner — at the cost of not exercising checkout itself, which is Git's behaviour rather than Arkive's. **Chunk contiguity is deliberately not asserted.** Committed chunks leave intentional gaps at page boundaries, so the offsets test asserts ordering, non-overlap, and in-bounds only. **One test drives `main()` rather than the library call**, because the documentation tells a contributor to run the command, and the existing CLI tests mock the pipeline out. ## Test verification Every test was checked by breaking the property it guards and confirming it fails. Two did not, and were rewritten before this PR: - Single-source scoping originally ran against an intact checkout, where the correct result and the no-op result are the same bytes. It would have passed even if the source selection were ignored entirely. It now strips every source first, regenerates one, and asserts only that one came back. - The rollback test had the same defect: with `_restore_file_snapshot` stubbed out it still passed, because the run rewrote bytes that were already there. It now perturbs a committed file first, so restored bytes and regenerated bytes actually differ. Both were confirmed to fail against the corresponding injected defect after the rewrite. ## Verification - [x] `git diff --check` - [x] `python pipeline/validate/validate_records.py` — `PASS: 6 source(s), 14 document(s), 38 chunk(s), 0 rights profile(s), 0 recipe(s) across 18 record file(s)` - [x] `python -m unittest discover -s tests -v` — 304 tests pass (289 before) - [x] `python -m ruff check src pipeline tests` - [x] `python -m ruff format --check src pipeline tests` - [x] `python -m pip check` - [x] Coverage re-run — repository-wide 92%, unchanged; the new tests exercise production paths that were already covered, which is expected for a regression suite - [x] Rendered Markdown reviewed ## Scope confirmation - [x] No unrelated production behaviour changed - [x] No metadata contract changed - [x] No unreviewed source or generated data was added ## Linked issue Closes #17 
Prove automatically that the committed pilot corpus is reproducible, not
merely internally consistent.

The gap this closes. The canonical validator compares records to other
records and never opens a document file, and chunk generation compares
chunk text to its parent only in memory at generation time. A wrong
byte_size, a stale checksum, or an offset that had drifted away from the
text it points at would all validate cleanly today.

Records against bytes:
- every document byte_size and SHA-256 is checked against the file on disk
- every chunk text is checked against document_text[char_start:char_end]
- chunk offsets are ordered, non-overlapping, and inside their document;
  contiguity is deliberately not asserted, because page boundaries leave
  intentional gaps

Regeneration against the committed corpus:
- the documented command on an intact checkout changes nothing; this test
  drives main() rather than the library call, because that is what the
  documentation tells a contributor to run
- removing every generated artifact and rebuilding reproduces the
  committed bytes exactly, which is what shows the outputs are a function
  of the originals and the pinned toolchain rather than bytes nobody can
  reproduce
- two consecutive runs are identical
- committed document and chunk identifiers are preserved, never recomputed
- originals, source.json, and licence evidence are not rewritten
- processing one source restores only that source

Failure behaviour:
- a failure injected into the last source of a five-source run leaves
  nothing on disk for the four that already succeeded
- a final-validation failure, after every write has landed, rolls the
  whole corpus back to its exact prior bytes

Offline:
- regeneration is verified with socket creation disabled

Identifiers are treated as authoritative input rather than recomputed
output, per the constraint recorded on Issue #17: the intake contract
makes a committed identifier immutable while deriving it from metadata
that may later be corrected, so re-deriving one can legitimately differ.
The checksum-in-document-identity constraint from the same comment was
resolved under Issue #9 and no longer applies.

Nondeterministic fields. None are excluded from the comparison. Generated
records carry no timestamps, chunk.created_at being schema-optional and
never emitted, so every comparison is whole-file byte equality rather
than a filtered structural one. docs/development.md records this, and
records the consequence: a dependency upgrade that alters extracted or
normalized output will fail these tests, which is the intended signal and
should be handled as the reviewed compatibility change already documented
for the pypdf pin.

Each test was checked by breaking the property it guards and confirming
it fails. Two did not, and were rewritten. Single-source scoping ran
against an intact checkout, where the correct result and the no-op result
are the same bytes, so it would have passed even if the source selection
were ignored entirely. The rollback test had the same defect: with
rollback disabled it still passed, because the run rewrote the bytes that
were already there. Both now perturb state first so a real difference
exists to detect.

304 tests pass. No production behaviour, schema, source data, or record
content was changed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feature/issue-17-reproducibility-tests:feature/issue-17-reproducibility-tests
git switch feature/issue-17-reproducibility-tests

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch develop
git merge --no-ff feature/issue-17-reproducibility-tests
git switch feature/issue-17-reproducibility-tests
git rebase develop
git switch develop
git merge --ff-only feature/issue-17-reproducibility-tests
git switch feature/issue-17-reproducibility-tests
git rebase develop
git switch develop
git merge --no-ff feature/issue-17-reproducibility-tests
git switch develop
git merge --squash feature/issue-17-reproducibility-tests
git switch develop
git merge --ff-only feature/issue-17-reproducibility-tests
git switch develop
git merge feature/issue-17-reproducibility-tests
git push origin develop
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
  
area/deployment
Portable packaging and constrained-environment validation

  
area/governance
Project rules, policies, contributor workflow

  
area/metadata
Schemas, manifests, validation contracts

  
area/pipeline
Import, extraction, normalization, chunking

  
area/search
Indexing, retrieval, citation resolution

  
type/data
Source selection, metadata, fixtures, and corpus work

  
type/design
Architecture, policy, contracts, and ADR work

  
type/implementation
Production code or tooling

  
type/testing
Reproducibility, integration, evaluation, and benchmark work

No labels
area/deployment
area/governance
area/metadata
area/pipeline
area/search
type/data
type/design
type/implementation
type/testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Arkive/arkive!65
No description provided.