Arkive/arkive

Fork 0

Add clean-checkout regeneration and reproducibility tests #17

New issue

Open

opened 2026-07-24 19:38:18 -04:00 by JackFrostbyte · 1 comment

JackFrostbyte commented

2026-07-24 19:38:18 -04:00

Owner

Goal

    Prove automatically that a clean checkout can regenerate deterministic pilot records with correct checksums, offsets, lineage, and failure behavior.

    ## Why

    The milestone is not complete merely because the pipeline runs once. Arkive needs regression tests that prove committed outputs are reproducible and exact.

    ## Scope

    - Add an automated clean-output or temporary-directory regeneration test for the complete pilot corpus.

Run the documented end-to-end processing command in the test workflow.
Compare regenerated records with committed expected outputs using a deterministic comparison.
Verify original-file byte size and SHA-256 values.
Verify every chunk text against its half-open offsets.
Verify source → document → chunk lineage and source consistency.
Verify repeated runs produce identical records and stable identifiers.
Verify representative failures are clear and do not alter successful committed output.
Document any intentionally excluded nondeterministic field and justify it.
```
  ## Out of scope

  - Selecting or legally reviewing sources.
```
Manual editorial approval of extracted text.
Search index generation.
Performance benchmarking.

OCR, embeddings, LLMs, translation, or speech.

  ## Dependencies

  - #16 — Implement the end-to-end pilot processing command

  ## Acceptance criteria

  - A clean-checkout-style test regenerates all committed pilot outputs.

Regenerated deterministic records match committed expected records.
All file checksums and sizes are correct.
All chunk texts and offsets are exact.
All lineage and cross-record validations pass.
Two consecutive runs produce identical deterministic output.
Failure tests prove final outputs remain unchanged.

The complete test suite passes without network access.

  ## Validation

  - Run the complete unit-test suite from a clean checkout or equivalent isolated copy.

Run the regeneration test twice and compare outputs.
Run with networking disabled.

Inspect the test workspace to confirm failures leave no partial final records.

  ## Suggested branch

  `feature/issue-17-reproducibility-tests`

  <!-- arkive-planning-automation:2026-07-24-v3:M1-12 -->

## Goal Prove automatically that a clean checkout can regenerate deterministic pilot records with correct checksums, offsets, lineage, and failure behavior. ## Why The milestone is not complete merely because the pipeline runs once. Arkive needs regression tests that prove committed outputs are reproducible and exact. ## Scope - Add an automated clean-output or temporary-directory regeneration test for the complete pilot corpus. - Run the documented end-to-end processing command in the test workflow. - Compare regenerated records with committed expected outputs using a deterministic comparison. - Verify original-file byte size and SHA-256 values. - Verify every chunk text against its half-open offsets. - Verify source → document → chunk lineage and source consistency. - Verify repeated runs produce identical records and stable identifiers. - Verify representative failures are clear and do not alter successful committed output. - Document any intentionally excluded nondeterministic field and justify it. ## Out of scope - Selecting or legally reviewing sources. - Manual editorial approval of extracted text. - Search index generation. - Performance benchmarking. - OCR, embeddings, LLMs, translation, or speech. ## Dependencies - #16 — Implement the end-to-end pilot processing command ## Acceptance criteria - A clean-checkout-style test regenerates all committed pilot outputs. - Regenerated deterministic records match committed expected records. - All file checksums and sizes are correct. - All chunk texts and offsets are exact. - All lineage and cross-record validations pass. - Two consecutive runs produce identical deterministic output. - Failure tests prove final outputs remain unchanged. - The complete test suite passes without network access. ## Validation - Run the complete unit-test suite from a clean checkout or equivalent isolated copy. - Run the regeneration test twice and compare outputs. - Run with networking disabled. - Inspect the test workspace to confirm failures leave no partial final records. ## Suggested branch `feature/issue-17-reproducibility-tests`

JackFrostbyte added this to the Reproducible Acquisition & Processing v0.1 milestone

2026-07-24 19:38:18 -04:00

JackFrostbyte added the

type/testing

area/pipeline

labels

2026-07-24 19:38:18 -04:00

JackFrostbyte referenced this issue

2026-07-24 19:38:18 -04:00

Process and review the complete five-document pilot corpus #18

JackFrostbyte added a new dependency

2026-07-24 20:10:49 -04:00

#16 Implement the end-to-end pilot processing command

JackFrostbyte added a new dependency

2026-07-24 20:10:49 -04:00

#18 Process and review the complete five-document pilot corpus

McJuniorstein referenced this issue

2026-07-29 19:18:46 -04:00

Align metadata schemas and validation with the intake and processing contracts #9

McJuniorstein commented

2026-07-29 19:22:27 -04:00

Member

Two constraints from the merged #8 identifier rules that affect how this issue's reproducibility tests can be written. Recording them here so they are known before the tests are designed.

1. Identifiers are not re-derivable after a metadata correction

docs/source-intake.md §6.1 derives the source key from the normalized title and a canonical identity built from title, authors, publisher, publication date, edition, and language. It also states that a committed identifier is immutable and that correcting metadata must not rename it.

Both rules are correct individually. Together they mean that once any of those six fields is corrected, re-deriving the identifier from the committed record produces a different key than the one committed.

Determinism therefore holds at first assignment only, not as a property that can be re-verified from the record afterwards.

Consequence for this issue: a regeneration test that recomputes source or document identifiers and compares them against committed values will produce false failures on any source whose metadata was ever corrected. Committed identifiers must be treated as authoritative input to regeneration, not as recomputed output to be checked.

If identifier derivation should remain auditable, one cheap option is to record the canonical-identity string, or its SHA-256, in the source record at assignment time. Regeneration could then verify that the stored identity still matches the stored identifier without requiring current metadata to reproduce it.

2. Document identity currently includes the file checksum, which couples identifiers to tool output

§6.2 builds the canonical document identity from source_id, artifact_key, role, language, derived_from, and checksum_sha256.

Because the checksum participates in identity, any change to derived output bytes produces a different canonical identity. Re-running OCR or normalization with a different tool version therefore yields a different identity, while the natural key <source-key>-<artifact-key> is already held by the previous identity. Rule 3 then applies the collision path and produces a second document identifier, and rule 9 forbids updating the existing record's identifier in place. Chunk records referencing the previous document_id become stale.

This means "clean-checkout regeneration produces identical records" holds only while every transformation tool and version is pinned. Any dependency upgrade that alters output bytes will change document identifiers rather than just checksums.

Two possible resolutions, both worth deciding before the tests are written:

pin transformation tool versions as part of the reproducibility contract, and treat an output-bytes change as an intentional, reviewed migration; or
remove checksum_sha256 from the document identity tuple, leaving (source_id, artifact_key, role, language, derived_from), so regeneration updates the checksum field while the identifier and all chunk references stay stable.

The second option is more consistent with §6.3, which argues that source identifiers should not be content hashes precisely because file integrity belongs in checksum fields. This has been raised as a finding on #9, since the identifier rules may be amended there.

Two constraints from the merged #8 identifier rules that affect how this issue's reproducibility tests can be written. Recording them here so they are known before the tests are designed. ## 1. Identifiers are not re-derivable after a metadata correction `docs/source-intake.md` §6.1 derives the source key from the normalized title and a canonical identity built from title, authors, publisher, publication date, edition, and language. It also states that a committed identifier is immutable and that correcting metadata must not rename it. Both rules are correct individually. Together they mean that once any of those six fields is corrected, re-deriving the identifier from the committed record produces a **different** key than the one committed. Determinism therefore holds at first assignment only, not as a property that can be re-verified from the record afterwards. Consequence for this issue: a regeneration test that recomputes source or document identifiers and compares them against committed values will produce false failures on any source whose metadata was ever corrected. Committed identifiers must be treated as authoritative input to regeneration, not as recomputed output to be checked. If identifier derivation should remain auditable, one cheap option is to record the canonical-identity string, or its SHA-256, in the source record at assignment time. Regeneration could then verify that the stored identity still matches the stored identifier without requiring current metadata to reproduce it. ## 2. Document identity currently includes the file checksum, which couples identifiers to tool output §6.2 builds the canonical document identity from `source_id`, `artifact_key`, `role`, `language`, `derived_from`, and `checksum_sha256`. Because the checksum participates in identity, any change to derived output bytes produces a different canonical identity. Re-running OCR or normalization with a different tool version therefore yields a different identity, while the natural key `<source-key>-<artifact-key>` is already held by the previous identity. Rule 3 then applies the collision path and produces a second document identifier, and rule 9 forbids updating the existing record's identifier in place. Chunk records referencing the previous `document_id` become stale. This means "clean-checkout regeneration produces identical records" holds only while every transformation tool and version is pinned. Any dependency upgrade that alters output bytes will change document identifiers rather than just checksums. Two possible resolutions, both worth deciding before the tests are written: - pin transformation tool versions as part of the reproducibility contract, and treat an output-bytes change as an intentional, reviewed migration; or - remove `checksum_sha256` from the document identity tuple, leaving `(source_id, artifact_key, role, language, derived_from)`, so regeneration updates the checksum field while the identifier and all chunk references stay stable. The second option is more consistent with §6.3, which argues that source identifiers should not be content hashes precisely because file integrity belongs in checksum fields. This has been raised as a finding on #9, since the identifier rules may be amended there.

McJuniorstein referenced this issue from a commit

2026-08-17 20:10:02 -04:00

Add clean-checkout regeneration and reproducibility tests

McJuniorstein referenced this issue from a pull request that will close it,

2026-08-17 20:10:58 -04:00

Add clean-checkout regeneration and reproducibility tests #65