Arkive/arkive

Fork 0

Define acquisition recipe and local-use rights schemas #56

Merged

JackFrostbyte merged 4 commits from feature/issue-44-acquisition-schemas into develop

2026-08-09 01:16:03 -04:00

McJuniorstein commented

2026-08-04 03:30:49 -04:00

Member

Summary

Adds three schemas: local_use_rights (the eight eligibility dimensions, evaluated independently), acquisition_recipe (one reviewed route as declarative, versioned, withdrawable data), and local_acquisition (what a user actually obtained, belonging to the user_local domain).
Adds an optional distribution_scope to document and chunk records, defaulting to bundled_core, and enforces that it is inherited down the lineage so anything produced from a user acquisition stays user_local.
Adds nine validator rules covering fail-closed dimensions, not_applicable rationale and mode-binding, recipe/rights agreement, withdrawal propagation, checksum coherence, download-host allowlisting, and scope inheritance.
Records ADR-0003 explaining why provenance, permission, recipe, and runtime remain four separate records rather than one field, with five rejected alternatives.
Resulting behaviour: an eligibility determination and an acquisition route are now representable and machine-checked. Nothing is acquired, no target is selected, and no network or processing code exists.

Deliverables

Required by #44	Where
Versioned JSON Schemas	`meta/schemas/local_use_rights.schema.json`, `acquisition_recipe.schema.json`, `local_acquisition.schema.json`
Valid and invalid fixtures	`tests/fixtures/acquisition/` (valid); invalid cases are single-field mutations of a valid fixture in `tests/test_acquisition_schemas.py`
Schema documentation and examples	`meta/schemas/README.md`, new "User-side acquisition contracts" section
Migration/compatibility notes	`meta/schemas/README.md`, "Compatibility with the existing contracts"
ADR note on separating the four concepts	`governance/decision_records/ADR-0003-...md`

Acceptance criteria

Existing bundled-source records remain valid or have a documented migration — distribution_scope is optional on both document and chunk and defaults to bundled_core. The committed corpus validates with no edits; a regression test asserts that a record omitting the field entirely is still valid. The compatibility table in the schema README states this per field.
Unknown permission states fail closed — an undetermined result in a dimension required for the selected acquisition mode blocks eligibility. Required and exemptible dimensions are declared per mode in the validator rather than inferred.
Arbitrary code and secrets cannot be represented by a valid recipe — see the note below; this is structural, not a rule.
Recipe withdrawal and expiry are representable — both records carry review.last_reviewed, review.expires, and an optional review.withdrawn with date and reason. A withdrawn rights profile forces its recipes to be withdrawn; tests cover both the failure and the valid withdrawn state.
Derived local artifacts inherit local_only — a document derived from a user_local document must be user_local, a chunk must match its document's scope, and a document carrying a local_acquisition_id must be user_local. A local_acquisition record is pinned to "user_local" by const, so no value promotes it.
Validation tests cover contradictory states and missing evidence — mode mismatch between recipe and rights, jurisdiction not reviewed, unknown rights_id, source mismatch, withdrawn rights against an active recipe, checksum required without a checksum, download host outside approved_hosts, missing rationale, and exemption claimed where the mode forbids it.

Verification

git diff --check
python pipeline/validate/validate_records.py — PASS, committed corpus unchanged
python -m unittest discover -s tests -v — 83 tests passing, up from 55
python -m ruff check src pipeline tests
python -m ruff format --check src pipeline tests
Rendered Markdown reviewed — 9 relative links checked, 0 broken
Complete diff inspected

Scope confirmation

No unrelated production behaviour changed
No existing field changed; origin.acquisition_path is deliberately untouched
No unreviewed source or generated data added
No network access, downloader, or download execution
No processing, indexing, or search implementation
No production acquisition target selected or approved

The "arbitrary code and secrets" criterion

This is enforced structurally rather than by prohibition, which is worth checking during review because it is the strongest claim in the PR:

every object sets additionalProperties: false, so an unknown property is invalid rather than ignored;
the adapter allowlist is closed to manual and https_direct, and adapter parameters are per-type closed objects rather than a free-form map, so there is nowhere to put a command;
URLs are constrained to ^https:// with userinfo excluded by pattern, so file://, http://, and https://user:secret@host/ are all invalid;
hostnames are bare DNS names with no scheme, port, path, or userinfo, so a host entry cannot smuggle a URL.

Tests assert that a recipe carrying command, post_process_script, password, api_key, cookie, embedded userinfo, or a non-HTTPS scheme fails validation, and that a local acquisition record carrying account_id, order_receipt, or payment_method fails.

Points for reviewer attention

One rule may belong to #45 rather than here. The validator rejects a local_acquisition.json found anywhere under data/. It is a small contamination guard directly serving the inheritance criterion, but full storage-boundary enforcement is Issue #45's remit. Happy to move it if the scopes should stay cleaner.

Fixtures live under tests/fixtures/acquisition/, not data/. A recipe committed under data/ would read as an approved acquisition target, and Issue #43 explicitly selects none. Every host in the fixtures is example.org, and both fixture sources are marked metadata_only and non-bundleable.

Required and exemptible dimensions are declared per mode as explicit sets in the validator rather than derived. This is deliberate: a missing entry in a derived rule would silently permit an unreviewed dimension, which is the failure mode the fail-closed rule exists to prevent.

Only two acquisition modes and two adapters are defined. The issue mentions a possible future official API adapter; ADR-0002 defers authenticated APIs to a dedicated design, so adding the enum value now would be a speculative field. Both schemas note that a third mode requires an approved design and a version bump.

authentication_required: true forces manual_import. Since credentials can never appear in a recipe, an automated mode against an authenticated route is unrepresentable in principle; this rule makes that explicit rather than leaving it implied.

Linked issue

Closes #44

## Summary - Adds three schemas: `local_use_rights` (the eight eligibility dimensions, evaluated independently), `acquisition_recipe` (one reviewed route as declarative, versioned, withdrawable data), and `local_acquisition` (what a user actually obtained, belonging to the `user_local` domain). - Adds an optional `distribution_scope` to document and chunk records, defaulting to `bundled_core`, and enforces that it is inherited down the lineage so anything produced from a user acquisition stays `user_local`. - Adds nine validator rules covering fail-closed dimensions, `not_applicable` rationale and mode-binding, recipe/rights agreement, withdrawal propagation, checksum coherence, download-host allowlisting, and scope inheritance. - Records ADR-0003 explaining why provenance, permission, recipe, and runtime remain four separate records rather than one field, with five rejected alternatives. - Resulting behaviour: an eligibility determination and an acquisition route are now representable and machine-checked. Nothing is acquired, no target is selected, and no network or processing code exists. ## Deliverables | Required by #44 | Where | |---|---| | Versioned JSON Schemas | `meta/schemas/local_use_rights.schema.json`, `acquisition_recipe.schema.json`, `local_acquisition.schema.json` | | Valid and invalid fixtures | `tests/fixtures/acquisition/` (valid); invalid cases are single-field mutations of a valid fixture in `tests/test_acquisition_schemas.py` | | Schema documentation and examples | `meta/schemas/README.md`, new "User-side acquisition contracts" section | | Migration/compatibility notes | `meta/schemas/README.md`, "Compatibility with the existing contracts" | | ADR note on separating the four concepts | `governance/decision_records/ADR-0003-...md` | ## Acceptance criteria - [x] **Existing bundled-source records remain valid or have a documented migration** — `distribution_scope` is optional on both document and chunk and defaults to `bundled_core`. The committed corpus validates with no edits; a regression test asserts that a record omitting the field entirely is still valid. The compatibility table in the schema README states this per field. - [x] **Unknown permission states fail closed** — an `undetermined` result in a dimension required for the selected acquisition mode blocks eligibility. Required and exemptible dimensions are declared per mode in the validator rather than inferred. - [x] **Arbitrary code and secrets cannot be represented by a valid recipe** — see the note below; this is structural, not a rule. - [x] **Recipe withdrawal and expiry are representable** — both records carry `review.last_reviewed`, `review.expires`, and an optional `review.withdrawn` with date and reason. A withdrawn rights profile forces its recipes to be withdrawn; tests cover both the failure and the valid withdrawn state. - [x] **Derived local artifacts inherit `local_only`** — a document derived from a `user_local` document must be `user_local`, a chunk must match its document's scope, and a document carrying a `local_acquisition_id` must be `user_local`. A `local_acquisition` record is pinned to `"user_local"` by `const`, so no value promotes it. - [x] **Validation tests cover contradictory states and missing evidence** — mode mismatch between recipe and rights, jurisdiction not reviewed, unknown `rights_id`, source mismatch, withdrawn rights against an active recipe, checksum required without a checksum, download host outside `approved_hosts`, missing rationale, and exemption claimed where the mode forbids it. ## Verification - [x] `git diff --check` - [x] `python pipeline/validate/validate_records.py` — PASS, committed corpus unchanged - [x] `python -m unittest discover -s tests -v` — **83 tests** passing, up from 55 - [x] `python -m ruff check src pipeline tests` - [x] `python -m ruff format --check src pipeline tests` - [x] Rendered Markdown reviewed — 9 relative links checked, 0 broken - [x] Complete diff inspected ## Scope confirmation - [x] No unrelated production behaviour changed - [x] No existing field changed; `origin.acquisition_path` is deliberately untouched - [x] No unreviewed source or generated data added - [x] No network access, downloader, or download execution - [x] No processing, indexing, or search implementation - [x] No production acquisition target selected or approved ## The "arbitrary code and secrets" criterion This is enforced structurally rather than by prohibition, which is worth checking during review because it is the strongest claim in the PR: - every object sets `additionalProperties: false`, so an unknown property is invalid rather than ignored; - the adapter allowlist is closed to `manual` and `https_direct`, and adapter parameters are **per-type closed objects** rather than a free-form map, so there is nowhere to put a command; - URLs are constrained to `^https://` with userinfo excluded by pattern, so `file://`, `http://`, and `https://user:secret@host/` are all invalid; - hostnames are bare DNS names with no scheme, port, path, or userinfo, so a host entry cannot smuggle a URL. Tests assert that a recipe carrying `command`, `post_process_script`, `password`, `api_key`, `cookie`, embedded userinfo, or a non-HTTPS scheme fails validation, and that a local acquisition record carrying `account_id`, `order_receipt`, or `payment_method` fails. ## Points for reviewer attention **One rule may belong to #45 rather than here.** The validator rejects a `local_acquisition.json` found anywhere under `data/`. It is a small contamination guard directly serving the inheritance criterion, but full storage-boundary enforcement is Issue #45's remit. Happy to move it if the scopes should stay cleaner. **Fixtures live under `tests/fixtures/acquisition/`, not `data/`.** A recipe committed under `data/` would read as an approved acquisition target, and Issue #43 explicitly selects none. Every host in the fixtures is `example.org`, and both fixture sources are marked `metadata_only` and non-bundleable. **Required and exemptible dimensions are declared per mode as explicit sets** in the validator rather than derived. This is deliberate: a missing entry in a derived rule would silently permit an unreviewed dimension, which is the failure mode the fail-closed rule exists to prevent. **Only two acquisition modes and two adapters are defined.** The issue mentions a possible future official API adapter; ADR-0002 defers authenticated APIs to a dedicated design, so adding the enum value now would be a speculative field. Both schemas note that a third mode requires an approved design and a version bump. **`authentication_required: true` forces `manual_import`.** Since credentials can never appear in a recipe, an automated mode against an authenticated route is unrepresentable in principle; this rule makes that explicit rather than leaving it implied. ## Linked issue Closes #44

McJuniorstein added 1 commit

2026-08-04 03:30:50 -04:00

Define acquisition recipe and local-use rights schemas 7cbf39e24d

Turn the acquisition eligibility policy into machine-readable contracts
without making descriptive source provenance executable.

Three new schemas:
- local_use_rights records the eight eligibility dimensions independently,
  each resolving to permitted, not_permitted, not_applicable, or
  undetermined, with reviewed jurisdictions and a review lifecycle
- acquisition_recipe describes one reviewed route as declarative data,
  versioned, withdrawable, and consumed by a trusted adapter
- local_acquisition records what a user actually obtained, and belongs to
  the user_local domain rather than this repository

Arbitrary code and credentials are structurally unrepresentable rather
than merely forbidden: every object closes additionalProperties, the
adapter allowlist is closed, adapter parameters are per-type closed
objects instead of a free-form map, and URLs are constrained to https
with userinfo excluded. Regression tests assert that commands, scripts,
passwords, API keys, cookies, embedded userinfo, and non-https schemes
all fail validation.

Validator additions:
- an undetermined dimension required by the chosen mode fails closed
- not_applicable requires a rationale and is allowed only where the mode
  exempts the dimension
- a recipe must agree with its rights profile on mode and source, offer
  only reviewed jurisdictions, and be withdrawn when its rights profile is
- checksum enforcement requires a recorded checksum, and a direct download
  must target an approved host
- a local acquisition record found under data/ is rejected

Distribution scope:
- document and chunk records gain an optional distribution_scope that
  defaults to bundled_core, so every existing record stays valid
- scope is inherited and never relaxed: a document derived from a
  user_local document stays user_local, a chunk matches its document, and
  a document referencing a local acquisition must be user_local

Also records ADR-0003 explaining why provenance, permission, recipe, and
runtime remain four separate records, with five rejected alternatives.

Worked fixtures live under tests/fixtures/acquisition/ rather than data/,
because a recipe committed under data/ would read as an approved
acquisition target and Issue #43 selects none. No acquisition target is
selected, and no network, download, or processing code is added.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

JackFrostbyte commented

2026-08-05 01:24:53 -04:00

Owner

Review — changes required before merge

This is a substantial and well-structured implementation. The separation between rights profiles, acquisition recipes, and local acquisition records is appropriate, and the PR remains within Issue #44’s schema-and-validation scope.

I found several contract gaps that need to be corrected before the PR can be merged:

Required not_permitted dimensions do not invalidate the selected acquisition mode.

The validator currently blocks required dimensions only when their result is undetermined. Under the approved rights policy, not_permitted in a dimension required for the selected mode must also make that mode unavailable, except for the explicitly non-blocking redistribution dimension.

Please add mode-aware validation and regression tests covering:
- Automated acquisition with automation = not_permitted
- Automated acquisition with service_terms = not_permitted
- Manual or automated acquisition with a required access/jurisdiction/evidence dimension marked not_permitted
- redistribution = not_permitted remaining non-blocking for local acquisition
A rights profile can claim that evidence was recorded without containing evidence.

evidence is optional at the root, and an evidence entry only requires its type. Please require usable supporting evidence and validate its relationship with the evidence_recorded determination. A valid supporting entry should include an auditable reference or sufficiently concrete recorded detail.
Some recipe URLs still permit embedded credentials.

The adapter URL rejects URL user information, but target.landing_page and integrity.publisher_signature_url do not. Please apply a shared HTTPS-without-userinfo definition consistently to every URL-bearing field.
Local acquisition records can omit required provenance.

final_url, adapter, and user_acknowledgement are currently optional. Please add mode-aware requirements so the record captures:
- The actual final URL for direct-download acquisitions
- Adapter/tool identity and version
- User acknowledgement whenever required by the recipe
Please also add corresponding negative tests.
Jurisdiction must be explicit.

The generic value multi does not identify which jurisdictions were reviewed. Please replace it with explicit jurisdiction identifiers, preferably through a non-empty list where more than one jurisdiction applies, and keep recipe/profile matching machine-checkable.
Tighten expected-file and post-acquisition coherence.

Please reject contradictory or ineffective combinations, including:
- min_bytes greater than max_bytes
- A required checksum check without an expected checksum
- A required content-type check without expected media types
- Direct-download recipes lacking meaningful expected file constraints

Once these points and their regression tests are addressed, the PR should receive another review against Issues #43 and #44 before merge.

## Review — changes required before merge This is a substantial and well-structured implementation. The separation between rights profiles, acquisition recipes, and local acquisition records is appropriate, and the PR remains within Issue #44’s schema-and-validation scope. I found several contract gaps that need to be corrected before the PR can be merged: 1. **Required `not_permitted` dimensions do not invalidate the selected acquisition mode.** The validator currently blocks required dimensions only when their result is `undetermined`. Under the approved rights policy, `not_permitted` in a dimension required for the selected mode must also make that mode unavailable, except for the explicitly non-blocking redistribution dimension. Please add mode-aware validation and regression tests covering: * Automated acquisition with `automation = not_permitted` * Automated acquisition with `service_terms = not_permitted` * Manual or automated acquisition with a required access/jurisdiction/evidence dimension marked `not_permitted` * `redistribution = not_permitted` remaining non-blocking for local acquisition 2. **A rights profile can claim that evidence was recorded without containing evidence.** `evidence` is optional at the root, and an evidence entry only requires its type. Please require usable supporting evidence and validate its relationship with the `evidence_recorded` determination. A valid supporting entry should include an auditable reference or sufficiently concrete recorded detail. 3. **Some recipe URLs still permit embedded credentials.** The adapter URL rejects URL user information, but `target.landing_page` and `integrity.publisher_signature_url` do not. Please apply a shared HTTPS-without-userinfo definition consistently to every URL-bearing field. 4. **Local acquisition records can omit required provenance.** `final_url`, `adapter`, and `user_acknowledgement` are currently optional. Please add mode-aware requirements so the record captures: * The actual final URL for direct-download acquisitions * Adapter/tool identity and version * User acknowledgement whenever required by the recipe Please also add corresponding negative tests. 5. **Jurisdiction must be explicit.** The generic value `multi` does not identify which jurisdictions were reviewed. Please replace it with explicit jurisdiction identifiers, preferably through a non-empty list where more than one jurisdiction applies, and keep recipe/profile matching machine-checkable. 6. **Tighten expected-file and post-acquisition coherence.** Please reject contradictory or ineffective combinations, including: * `min_bytes` greater than `max_bytes` * A required checksum check without an expected checksum * A required content-type check without expected media types * Direct-download recipes lacking meaningful expected file constraints Once these points and their regression tests are addressed, the PR should receive another review against Issues #43 and #44 before merge.

McJuniorstein referenced this pull request from a commit

2026-08-08 12:21:12 -04:00

Block refused dimensions, require checkable evidence, tighten recipes

McJuniorstein added 2 commits

2026-08-08 12:21:12 -04:00

Merge remote-tracking branch 'origin/develop' into feature/issue-44-acquisition-schemas 3cb92f3fee

Block refused dimensions, require checkable evidence, tighten recipes afcdac9980

Addresses the review on PR #56.

Mode availability:
- a required dimension now blocks its acquisition mode when the result is
  `not_permitted`, not only when it is `undetermined`. Refusal and absence
  are different facts with the same consequence for that mode.
- `redistribution` stays explicitly non-blocking, since its refusal is the
  normal case for anything reaching this contract at all.

Evidence:
- a rights profile must carry at least one evidence entry
- each entry must offer something another reviewer can check: a URL with
  the date it was accessed, or a concrete recorded detail. A bare type is
  an assertion rather than evidence.

URLs:
- one shared `https_url` definition, excluding userinfo, now applies to
  every URL-bearing field: adapter URL, landing page, publisher signature
  URL, evidence URLs, and the recorded final URL. Previously only the
  adapter and final URLs excluded embedded credentials.

Jurisdictions:
- removed the `multi` catch-all. A determination names every jurisdiction
  it covers, because "everywhere" is not a reviewed position. Multiple
  jurisdictions are listed explicitly, and recipe/profile matching stays
  machine-checkable.

Local acquisition provenance:
- `adapter` is required, `final_url` is required for direct downloads, and
  acknowledgement must be recorded and accepted whenever the referenced
  recipe requires it.

Recipe coherence:
- reject `min_bytes` greater than `max_bytes`, media-type checking without
  expected media types, size checking without bounds, and direct-download
  recipes that state no expected media types. A flag that cannot fail
  reads as a guarantee it is not providing.

The existing rule rejecting checksum enforcement without a recorded
checksum already covered that case and is unchanged.

Also removes a cross-field evidence rule that became unreachable once
evidence was required at the schema level. Keeping it would have been the
same defect as an enforcement flag with no data behind it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

McJuniorstein commented

2026-08-08 12:21:42 -04:00

Author

Member

All six points addressed in afcdac9, with the branch updated from develop after PR #57.

1. Refused dimensions now block their mode. The gap was real, and it contradicted the policy this PR is supposed to enforce: docs/acquisition-eligibility.md section 3 already says a not_permitted result in a required dimension makes that mode unavailable, and the validator only checked undetermined. A required dimension now blocks on both — absence and refusal are different facts with the same consequence for that mode. redistribution stays explicitly non-blocking via a named NON_BLOCKING_DIMENSIONS set rather than an inline exception. Tests cover automated acquisition with automation and with service_terms refused, refusal of a required access/jurisdiction/evidence dimension across both modes, and redistribution = not_permitted still passing in both.

2. Evidence must now be checkable. evidence is required at the root, and each entry must satisfy anyOf: a URL together with date_accessed, or a detail of at least 20 characters. A bare {"type": ...} is an assertion, not evidence, and no longer validates. Tests cover the missing array, the empty array, a bare type, a URL with no access date, and a trivial detail.

One note on the second half of that point. Once evidence was required at schema level with minItems: 1, the cross-field rule checking evidence_recorded = permitted against evidence presence became unreachable — schema validation rejects the record before the rule runs. I removed it rather than leave it in place, because a rule that cannot fire is the same defect as point 6: it reads as enforcement while providing none. If you would rather see the relationship expressed as a cross-field rule, the way to do it is to make evidence optional again and enforce it there instead; happy to switch if you prefer that shape.

3. Userinfo excluded everywhere. There is now one shared https_url definition per schema, and every URL-bearing field references it: the adapter URL, target.landing_page, integrity.publisher_signature_url, rights-profile evidence URLs, and the recorded final_url. A subtest asserts that https://user:pw@… is rejected in each of those positions.

4. Local acquisition provenance is now mode-aware. adapter is required unconditionally; final_url is required when acquisition_mode is official_direct_download, via schema if/then; and acknowledgement is cross-checked against the referenced recipe, so a record must show accepted: true whenever the recipe sets user_acknowledgement.required. Negative tests cover each, including acknowledgement recorded but refused.

5. Jurisdictions are explicit. The multi catch-all is gone from both new schemas; the shared jurisdiction definition is now ^[A-Z]{2}$ with a comment stating why there is deliberately no catch-all. Multiple jurisdictions are listed explicitly, and recipe/profile subset matching is unchanged and still machine-checkable. Tests reject multi in both the profile and the recipe, and confirm a three-jurisdiction list validates.

One thing outside this PR's scope that you should decide on: source.schema.json still permits multi in license.jurisdiction, added under Issue #9. The same argument applies to it. No committed record uses the value, so removing it would break nothing, but it is a change to an existing contract outside #44 and I did not want to make it silently. Happy to include it here or raise a separate issue, whichever you prefer.

6. Recipe coherence. Now rejected: min_bytes greater than max_bytes; require_media_type_match without expected.media_types; require_size_within_bounds without either bound; and a direct-download recipe stating no expected media types. Each has a regression test.

The checksum sub-item was already implemented before this review — validate_records.py rejected require_checksum_match without a recorded integrity.sha256, and there was a test for it. It is unchanged, and I mention it only so the list reconciles.

Validation after the changes, with develop merged in:

git diff --check clean
metadata validator PASS, committed corpus unchanged
121 unit tests passing, up from 83
ruff check and ruff format --check clean
9 relative links checked, 0 broken

Both worked fixtures still validate, and the committed corpus needed no edits.

All six points addressed in `afcdac9`, with the branch updated from `develop` after PR #57. **1. Refused dimensions now block their mode.** The gap was real, and it contradicted the policy this PR is supposed to enforce: `docs/acquisition-eligibility.md` section 3 already says a `not_permitted` result in a required dimension makes that mode unavailable, and the validator only checked `undetermined`. A required dimension now blocks on both — absence and refusal are different facts with the same consequence for that mode. `redistribution` stays explicitly non-blocking via a named `NON_BLOCKING_DIMENSIONS` set rather than an inline exception. Tests cover automated acquisition with `automation` and with `service_terms` refused, refusal of a required access/jurisdiction/evidence dimension across both modes, and `redistribution = not_permitted` still passing in both. **2. Evidence must now be checkable.** `evidence` is required at the root, and each entry must satisfy `anyOf`: a URL together with `date_accessed`, or a `detail` of at least 20 characters. A bare `{"type": ...}` is an assertion, not evidence, and no longer validates. Tests cover the missing array, the empty array, a bare type, a URL with no access date, and a trivial detail. One note on the second half of that point. Once evidence was required at schema level with `minItems: 1`, the cross-field rule checking `evidence_recorded = permitted` against evidence presence became unreachable — schema validation rejects the record before the rule runs. I removed it rather than leave it in place, because a rule that cannot fire is the same defect as point 6: it reads as enforcement while providing none. If you would rather see the relationship expressed as a cross-field rule, the way to do it is to make `evidence` optional again and enforce it there instead; happy to switch if you prefer that shape. **3. Userinfo excluded everywhere.** There is now one shared `https_url` definition per schema, and every URL-bearing field references it: the adapter URL, `target.landing_page`, `integrity.publisher_signature_url`, rights-profile evidence URLs, and the recorded `final_url`. A subtest asserts that `https://user:pw@…` is rejected in each of those positions. **4. Local acquisition provenance is now mode-aware.** `adapter` is required unconditionally; `final_url` is required when `acquisition_mode` is `official_direct_download`, via schema `if/then`; and acknowledgement is cross-checked against the referenced recipe, so a record must show `accepted: true` whenever the recipe sets `user_acknowledgement.required`. Negative tests cover each, including acknowledgement recorded but refused. **5. Jurisdictions are explicit.** The `multi` catch-all is gone from both new schemas; the shared `jurisdiction` definition is now `^[A-Z]{2}$` with a comment stating why there is deliberately no catch-all. Multiple jurisdictions are listed explicitly, and recipe/profile subset matching is unchanged and still machine-checkable. Tests reject `multi` in both the profile and the recipe, and confirm a three-jurisdiction list validates. **One thing outside this PR's scope that you should decide on:** `source.schema.json` still permits `multi` in `license.jurisdiction`, added under Issue #9. The same argument applies to it. No committed record uses the value, so removing it would break nothing, but it is a change to an existing contract outside #44 and I did not want to make it silently. Happy to include it here or raise a separate issue, whichever you prefer. **6. Recipe coherence.** Now rejected: `min_bytes` greater than `max_bytes`; `require_media_type_match` without `expected.media_types`; `require_size_within_bounds` without either bound; and a direct-download recipe stating no expected media types. Each has a regression test. The checksum sub-item was already implemented before this review — `validate_records.py` rejected `require_checksum_match` without a recorded `integrity.sha256`, and there was a test for it. It is unchanged, and I mention it only so the list reconciles. **Validation after the changes**, with `develop` merged in: - `git diff --check` clean - metadata validator PASS, committed corpus unchanged - **121 unit tests** passing, up from 83 - `ruff check` and `ruff format --check` clean - 9 relative links checked, 0 broken Both worked fixtures still validate, and the committed corpus needed no edits.

JackFrostbyte added 1 commit

2026-08-09 01:14:06 -04:00