Align metadata schemas and validation with the intake workflow #33

Merged

JackFrostbyte merged 1 commit from feature/issue-9-metadata-intake-alignment into develop

2026-07-30 00:47:40 -04:00

McJuniorstein commented

2026-07-29 21:16:32 -04:00

Member

Summary

Enforces the parts of the approved v0.1 intake workflow that fit the existing source → document → chunk model. Adds a source.review object for reviewer identity and review date, and license.jurisdiction plus license.legal_basis for public-domain and government-work claims.
Adds six validator cross-field rules, each commented with the workflow section it implements: license evidence required before bundle eligibility; reviewer and date required once a review decision is recorded; attribution text required with an attribution condition; jurisdiction and legal basis required for public-domain or government-work evidence; access date required on evidence carrying a URL; and no_derivatives inheritance blocking bundle eligibility while derived documents or chunks exist.
Corrects three identifier rules in docs/source-intake.md: removes the file checksum from document identity, preserves the artifact key when truncating a long document key, and records that identifier determinism applies to first assignment rather than to re-derivation.
Resulting behaviour: the prose requirements in sections 7.2, 7.3, 9.1, and 10 are now machine-enforced rather than advisory. A no_derivatives source with an OCR derivative and chunks marked bundleable previously validated clean; it now fails with specific errors.

Verification

git diff --check
python pipeline/validate/validate_records.py
python -m unittest discover -s tests -v — 21 tests, all passing (was 8)
python -m ruff check src pipeline tests
python -m ruff format --check src pipeline tests
Rendered Markdown reviewed — heading structure inspected, all relative links resolved
Complete diff inspected

Additional behavioural checks performed:

Each of the four record shapes prescribed by section 3.2 (pending review, metadata-only, rejected tombstone, accepted non-bundleable) validates once it carries the newly required review object.
A source with conditions: ["attribution", "no_derivatives"], bundleable: true, an ocr_text document derived from the original, and a chunk cut from that OCR text now fails validation. The same fixture passed before this change.
A no_derivatives source with only an unchanged original and no chunks still validates, confirming the rule does not over-block what section 9.1 permits.
A no_derivatives source with derived documents but bundleable: false still validates, confirming derivatives may exist for lawful local review.

Scope confirmation

No unrelated production behaviour changed
No metadata contract changed unless explicitly in scope — document.schema.json and chunk.schema.json are untouched
No unreviewed source or generated data was added
No visual document roles, per-document bundle eligibility, importing, extraction, normalization, or chunk-generation work included

Metadata v0.1 compatibility decision

review is optional in the schema but required by the validator once a decision exists. Section 3.2 permits a committed pending_review record, and such a record legitimately has no completed reviewer or review date. Making review schema-required would forbid a state the workflow explicitly allows. It is therefore enforced as a cross-field rule for review_status of accepted or rejected only, with a positive regression test asserting that a pending_review record may omit it.

The committed worked example was migrated, not grandfathered. data/sources/core/reference-library-sample/source.json gains a review block. Its reviewer is recorded as the contributor who authored the fixture, with the review date matching the commit that introduced it. No other field changed and its checksums, offsets, and exact text are unaffected.

Existing behaviour is preserved. All eight prior validations continue to work. One existing test fixture (test_valid_derived_chain_passes) was migrated because its source record predated these rules and was bundleable with neither evidence nor a review block. The shared source_record test helper now produces a valid record by default, with explicit switches for the negative cases, so each new rule has both a positive and a negative test.

Backward compatibility. Any source record written before this change that is accepted or rejected, or that is bundleable without evidence, will now fail validation. The only such record in the repository is the worked example, which is migrated here. No migration tooling is provided because no other records exist.

Two points for reviewer attention

The no_derivatives rule also blocks chunks, not only derived documents. Section 9.1 states that no_derivatives "permits consideration of the unchanged original only", and a chunk is not the unchanged original. The strict reading is implemented. If chunks cut directly from an unchanged original were intended to remain permissible, this rule needs narrowing to derived documents alone.

Three documentation corrections extend slightly beyond the schema and validator changes. The document-identity, key-truncation, and determinism clarifications in docs/source-intake.md were raised in the review comment on this issue and all concern the identifier contract this issue aligns, so they were judged in scope. They are documentation-only and change no committed record. Happy to split them into a follow-up if preferred.

Remaining follow-up

Per-document and per-asset bundle eligibility, visual document roles with page and caption linkage, and release rules for share_alike and non_commercial are not addressed here. The first two are structural changes to the document contract and are tracked in #32. Section 13 and the schema README both record the remaining gaps.

Linked issue

Closes #9

## Summary - Enforces the parts of the approved v0.1 intake workflow that fit the existing source → document → chunk model. Adds a `source.review` object for reviewer identity and review date, and `license.jurisdiction` plus `license.legal_basis` for public-domain and government-work claims. - Adds six validator cross-field rules, each commented with the workflow section it implements: license evidence required before bundle eligibility; reviewer and date required once a review decision is recorded; attribution text required with an attribution condition; jurisdiction and legal basis required for public-domain or government-work evidence; access date required on evidence carrying a URL; and `no_derivatives` inheritance blocking bundle eligibility while derived documents or chunks exist. - Corrects three identifier rules in `docs/source-intake.md`: removes the file checksum from document identity, preserves the artifact key when truncating a long document key, and records that identifier determinism applies to first assignment rather than to re-derivation. - Resulting behaviour: the prose requirements in sections 7.2, 7.3, 9.1, and 10 are now machine-enforced rather than advisory. A `no_derivatives` source with an OCR derivative and chunks marked bundleable previously validated clean; it now fails with specific errors. ## Verification - [x] `git diff --check` - [x] `python pipeline/validate/validate_records.py` - [x] `python -m unittest discover -s tests -v` — 21 tests, all passing (was 8) - [x] `python -m ruff check src pipeline tests` - [x] `python -m ruff format --check src pipeline tests` - [x] Rendered Markdown reviewed — heading structure inspected, all relative links resolved - [x] Complete diff inspected Additional behavioural checks performed: - Each of the four record shapes prescribed by section 3.2 (pending review, metadata-only, rejected tombstone, accepted non-bundleable) validates once it carries the newly required `review` object. - A source with `conditions: ["attribution", "no_derivatives"]`, `bundleable: true`, an `ocr_text` document derived from the original, and a chunk cut from that OCR text now fails validation. The same fixture passed before this change. - A `no_derivatives` source with only an unchanged original and no chunks still validates, confirming the rule does not over-block what section 9.1 permits. - A `no_derivatives` source with derived documents but `bundleable: false` still validates, confirming derivatives may exist for lawful local review. ## Scope confirmation - [x] No unrelated production behaviour changed - [x] No metadata contract changed unless explicitly in scope — `document.schema.json` and `chunk.schema.json` are untouched - [x] No unreviewed source or generated data was added - [x] No visual document roles, per-document bundle eligibility, importing, extraction, normalization, or chunk-generation work included ## Metadata v0.1 compatibility decision **`review` is optional in the schema but required by the validator once a decision exists.** Section 3.2 permits a committed `pending_review` record, and such a record legitimately has no completed reviewer or review date. Making `review` schema-required would forbid a state the workflow explicitly allows. It is therefore enforced as a cross-field rule for `review_status` of `accepted` or `rejected` only, with a positive regression test asserting that a `pending_review` record may omit it. **The committed worked example was migrated, not grandfathered.** `data/sources/core/reference-library-sample/source.json` gains a `review` block. Its reviewer is recorded as the contributor who authored the fixture, with the review date matching the commit that introduced it. No other field changed and its checksums, offsets, and exact text are unaffected. **Existing behaviour is preserved.** All eight prior validations continue to work. One existing test fixture (`test_valid_derived_chain_passes`) was migrated because its source record predated these rules and was bundleable with neither evidence nor a review block. The shared `source_record` test helper now produces a valid record by default, with explicit switches for the negative cases, so each new rule has both a positive and a negative test. **Backward compatibility.** Any source record written before this change that is `accepted` or `rejected`, or that is `bundleable` without evidence, will now fail validation. The only such record in the repository is the worked example, which is migrated here. No migration tooling is provided because no other records exist. ## Two points for reviewer attention **The `no_derivatives` rule also blocks chunks, not only derived documents.** Section 9.1 states that `no_derivatives` "permits consideration of the unchanged original only", and a chunk is not the unchanged original. The strict reading is implemented. If chunks cut directly from an unchanged original were intended to remain permissible, this rule needs narrowing to derived documents alone. **Three documentation corrections extend slightly beyond the schema and validator changes.** The document-identity, key-truncation, and determinism clarifications in `docs/source-intake.md` were raised in the review comment on this issue and all concern the identifier contract this issue aligns, so they were judged in scope. They are documentation-only and change no committed record. Happy to split them into a follow-up if preferred. ## Remaining follow-up Per-document and per-asset bundle eligibility, visual document roles with page and caption linkage, and release rules for `share_alike` and `non_commercial` are not addressed here. The first two are structural changes to the document contract and are tracked in #32. Section 13 and the schema README both record the remaining gaps. ## Linked issue Closes #9

McJuniorstein added 1 commit

2026-07-29 21:16:32 -04:00

Align metadata schemas and validation with the intake workflow 3e35dcb201

Enforce the parts of the approved v0.1 intake workflow that fit the
existing source -> document -> chunk model, without redesigning it.

Schema:
- add a source `review` object carrying reviewer identity and review date
- add `license.jurisdiction` and `license.legal_basis` for public-domain
  and government-work claims

Validator cross-field rules, each traceable to a workflow section:
- bundleable requires at least one license evidence entry
- an accepted or rejected review must record its reviewer and date
- an attribution condition requires attribution text
- public-domain or government-work evidence requires jurisdiction and
  legal basis
- license evidence carrying a url must record its access date
- a no_derivatives source cannot be bundleable while derived documents or
  chunks exist, because v0.1 carries only a source-level bundle decision

Identifier corrections in docs/source-intake.md:
- remove the file checksum from document identity, so regenerating a
  derived artifact updates its checksum without changing its identifier
  or stranding chunk references
- preserve the artifact key when truncating a long document key
- record that identifier determinism applies to first assignment, not to
  re-derivation after a metadata correction

Migrate the committed worked example to carry a review block, refresh the
schema README boundary section and section 13 gap list, and correct the
regression-test count to twenty-one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>