Arkive/arkive
1
0
Fork 0
Code Issues 18 Pull requests 1 Projects Releases Packages Wiki Activity Actions

Align metadata schemas and validation with the intake and processing contracts #9

New issue
Closed
opened 2026-07-24 19:38:14 -04:00 by JackFrostbyte · 2 comments
JackFrostbyte commented 2026-07-24 19:38:14 -04:00
Owner
Copy link

Goal

    Make the smallest schema, validator, documentation, and regression-test changes needed to support the approved intake workflow and upcoming deterministic processing pipeline.

    ## Why

    The intake workflow may reveal concrete fields or validation rules that the metadata v0.1 contracts do not yet express. Those changes should be implemented before the pilot corpus and importer rely on them.

    ## Scope

    - Compare the approved intake workflow with the current source, document, and chunk schemas.

  • Identify only the missing fields, enumerations, or cross-record rules required for intake and deterministic processing.

  • Update Draft 2020-12 schemas while preserving explicit source → document → chunk lineage.

  • Update cross-record validation for any newly required invariants.

  • Preserve or strengthen bundle eligibility restrictions.

  • Update schema documentation and worked-example records when required.

  • Add focused regression tests for every new rule and backward-incompatible change.

  • Document any metadata v0.1 compatibility decision made in the pull request.

      ## Out of scope

  - A wholesale metadata redesign.

  • Index, bundle, vector, translation, speech, or model manifests.

  • Trust scoring or broad controlled vocabularies.

  • File importing, PDF extraction, normalization, or chunk generation.

  • Selecting or processing the pilot corpus.

      ## Dependencies

  - #7 — Establish the Python project and development-tooling baseline

  • #8 — Define source-intake workflow and repository layout v0.1

      ## Acceptance criteria

  - Every schema change is directly traceable to an approved intake or processing requirement.

  • Existing lineage, duplicate-ID, source-consistency, bundle-eligibility, and offset validations continue to work.

  • New invariants have positive and negative regression tests.

  • The committed worked example remains valid or is intentionally migrated with documented reasoning.

  • All schema and cross-record validation tests pass.

  • No speculative fields for later milestones are added.

      ## Validation

  - Run the complete unit-test suite.

  • Validate every committed source, document, and chunk example.

  • Exercise negative fixtures for each new validation rule.

  • Run git diff --check.

      ## Suggested branch

  `feature/issue-9-metadata-intake-alignment`

  <!-- arkive-planning-automation:2026-07-24-v3:M1-04 -->
## Goal Make the smallest schema, validator, documentation, and regression-test changes needed to support the approved intake workflow and upcoming deterministic processing pipeline. ## Why The intake workflow may reveal concrete fields or validation rules that the metadata v0.1 contracts do not yet express. Those changes should be implemented before the pilot corpus and importer rely on them. ## Scope - Compare the approved intake workflow with the current source, document, and chunk schemas. - Identify only the missing fields, enumerations, or cross-record rules required for intake and deterministic processing. - Update Draft 2020-12 schemas while preserving explicit source → document → chunk lineage. - Update cross-record validation for any newly required invariants. - Preserve or strengthen bundle eligibility restrictions. - Update schema documentation and worked-example records when required. - Add focused regression tests for every new rule and backward-incompatible change. - Document any metadata v0.1 compatibility decision made in the pull request. ## Out of scope - A wholesale metadata redesign. - Index, bundle, vector, translation, speech, or model manifests. - Trust scoring or broad controlled vocabularies. - File importing, PDF extraction, normalization, or chunk generation. - Selecting or processing the pilot corpus. ## Dependencies - #7 — Establish the Python project and development-tooling baseline - #8 — Define source-intake workflow and repository layout v0.1 ## Acceptance criteria - Every schema change is directly traceable to an approved intake or processing requirement. - Existing lineage, duplicate-ID, source-consistency, bundle-eligibility, and offset validations continue to work. - New invariants have positive and negative regression tests. - The committed worked example remains valid or is intentionally migrated with documented reasoning. - All schema and cross-record validation tests pass. - No speculative fields for later milestones are added. ## Validation - Run the complete unit-test suite. - Validate every committed source, document, and chunk example. - Exercise negative fixtures for each new validation rule. - Run `git diff --check`. ## Suggested branch `feature/issue-9-metadata-intake-alignment` <!-- arkive-planning-automation:2026-07-24-v3:M1-04 -->
McJuniorstein commented 2026-07-29 19:18:46 -04:00
Member
Copy link

Review pass on the merged #8 workflow, filed here because every actionable follow-up lands in this issue.

Read docs/source-intake.md in full plus the diffs to both policies, AGENTS.md, CONTRIBUTING.md, meta/schemas/README.md, and the architecture overview. Ran the record shapes through the validator rather than reading only.

Verification performed

Current develop (aece00f) is green: metadata validator passes, 8/8 unit tests, Ruff lint and format clean, git diff --check clean. The "eight tests" claims in AGENTS.md and the architecture overview match reality.

All four record shapes prescribed by §3.2 validate against the current schema with no changes:

Shape Result
pending_review + license.status: pending_review + bundleable: false passes
accepted + metadata_only + bundleable: false passes
rejected + rejection_reason + bundleable: false passes
accepted + redistributable + bundleable: false (unresolved visual rights) passes

So the workflow does not prescribe any record the schema currently rejects. Good.

Limitation 4 confirmed concretely rather than assumed. A source with license.status: redistributable_with_conditions, conditions: ["attribution", "no_derivatives"], bundleable: true, plus an ocr_text document with derived_from pointing at the original and a chunk cut from that OCR text — the exact case §9.1 and §10 forbid — passes validation today. Nothing in validate_records.py reads conditions.

What holds up well

§2 keeping candidate stage, review status, license handling, and bundle eligibility as four separate dimensions is the right model, and §3.3's forbidden-interpretations list closes the reading loopholes that prose policy usually leaves open.

§6.1 is rigorous — explicit NFKC normalization, a fixed six-line identity serialization, and collision resolution that stops and reports rather than inventing a fallback. Step 11 refusing to fabricate an identifier is the correct failure mode.

§11.5 is the strongest section. Reasoning that a single source-level license.bundleable cannot safely express "original ships, embedded figure does not," and then keeping the source non-bundleable rather than weakening the boundary, is the right call. The per-document/per-visual gap it identifies was not in the earlier review input — it is a real modeling limitation and worth carrying forward deliberately.

Findings

1. Document identity includes the file checksum, which contradicts §6.3 and causes ID churn on regeneration

§6.3 argues that source identifiers should not be content hashes because "file integrity belongs in document-level fields such as checksums." But the canonical document identity in §6.2 includes checksum_sha256=<file-checksum>.

Consequence: re-running OCR with a different tool version produces different bytes, therefore a different checksum, therefore a different canonical document identity. The natural key doc_<source>-ocr-text is already held by the previous identity, so rule 3 fires and a second record is created as doc_<source>-ocr-text-<hash>. Rule 9 makes identifiers immutable, so updating the checksum in place is not permitted. Every chunk referencing the previous document_id becomes stale.

The workflow does not define regeneration semantics — whether the prior record is replaced, superseded, or retained. As written, proliferation is the only conforming path. This affects #14, #15, and particularly #17.

Suggested resolution: drop checksum_sha256 from the document identity tuple, leaving (source_id, artifact_key, role, language, derived_from). Re-derivation then updates the checksum field, the identifier stays stable, and chunk references survive — which is what §6.3 already argues for.

2. Identifiers are not re-derivable after a metadata correction

§6.1 derives the source key from the normalized title and canonical identity, and also states that identifiers are immutable and that correcting metadata must not rename them. Both rules are correct individually. Together they mean that after any title, publisher, or edition correction, re-deriving the identifier from the committed record produces a different key than the one committed.

Determinism therefore holds at first assignment only. #17 needs this stated explicitly: a regeneration test that recomputes identifiers and compares them will fail on any source whose metadata was ever corrected. Committed identifiers must be treated as authoritative input rather than recomputed output.

Cheap mitigation: say so in §6.1, and optionally record the canonical-identity string or its hash in the source record at assignment time so the derivation remains auditable.

3. The visual-rights rule narrows the #10 corpus more than #10 currently implies

§11.4 and §11.5 are correct. Following the consequence: most real-world manuals contain figures whose rights are not separately documented, so unless a license statement affirmatively covers all embedded content, the source stays non-bundleable.

Because #27 requires a bundleable proof-loop bundle, the five documents selected in #10 effectively must be born-digital, single-rights-holder, licensed at the whole-work level (CC0 or CC BY), or project-authored. That is a considerably narrower field than #10's current text suggests.

Not a flaw in the workflow — a scoping consequence worth recording in #10 before selection begins.

4. §6.2 rule 2 truncates the portion that distinguishes documents

The natural document key is <source-key>-<artifact-key>. A source key may be 64 characters, so <source-key>-original already exceeds the 64-character document limit. Rule 2 truncates "the descriptive portion" to 55 characters, which cuts the artifact key — the only part distinguishing original from ocr-text. Uniqueness survives through the hash suffix, but lineage becomes unreadable during debugging.

Suggested change: always preserve the artifact key and truncate only the source-key portion.

5. Minor — reviewer identity is coupled to the current host

§7.3 specifies the reviewer identifier as "normally the exact Forgejo username." If project hosting ever changes, historical reviewer identities become unresolvable. Recording the host alongside the username, or adopting a project-level contributor identifier, costs nothing at this stage.

Proposed scope for this issue

Limitations 1 through 4 from the #8 closing comment are implementable now without structural change:

  1. Add a review object carrying reviewer and review_date.
  2. Require non-empty license.evidence when license.bundleable is true. §7.2 and the license policy both require evidence; nothing enforces it.
  3. Add jurisdiction and legal_basis to the license object, supporting §7.2's public-domain and government-work requirement.
  4. Add a cross-field validator rule: no_derivatives combined with bundleable: true and any derived document must fail. This is precisely §9.1's final bullet and is implementable against the current schema.
  5. Require attribution_text when attribution appears in conditions.

Plus finding 1 above, which is a change to the identifier rules rather than the schema.

Limitations 5 and 6 — per-document and per-asset bundle eligibility, and visual document roles with page, figure, caption, and context linkage — are recommended for a separate issue. Both are structural changes to the document contract, and this issue's acceptance criteria exclude wholesale metadata redesign and speculative fields for later milestones. Visual extraction is not required until well past the proof loop, so splitting keeps this issue small and reviewable while allowing the artifact-level release contract to be designed properly on its own terms.

Happy to take this issue on that scope if it is useful, and to draft the split-out issue for artifact-level bundle eligibility and visual roles.

Review pass on the merged #8 workflow, filed here because every actionable follow-up lands in this issue. Read `docs/source-intake.md` in full plus the diffs to both policies, `AGENTS.md`, `CONTRIBUTING.md`, `meta/schemas/README.md`, and the architecture overview. Ran the record shapes through the validator rather than reading only. ## Verification performed Current `develop` (`aece00f`) is green: metadata validator passes, 8/8 unit tests, Ruff lint and format clean, `git diff --check` clean. The "eight tests" claims in `AGENTS.md` and the architecture overview match reality. All four record shapes prescribed by §3.2 validate against the current schema with no changes: | Shape | Result | |---|---| | `pending_review` + `license.status: pending_review` + `bundleable: false` | passes | | accepted + `metadata_only` + `bundleable: false` | passes | | `rejected` + `rejection_reason` + `bundleable: false` | passes | | accepted + `redistributable` + `bundleable: false` (unresolved visual rights) | passes | So the workflow does not prescribe any record the schema currently rejects. Good. Limitation 4 confirmed concretely rather than assumed. A source with `license.status: redistributable_with_conditions`, `conditions: ["attribution", "no_derivatives"]`, `bundleable: true`, plus an `ocr_text` document with `derived_from` pointing at the original and a chunk cut from that OCR text — the exact case §9.1 and §10 forbid — **passes validation today**. Nothing in `validate_records.py` reads `conditions`. ## What holds up well §2 keeping candidate stage, review status, license handling, and bundle eligibility as four separate dimensions is the right model, and §3.3's forbidden-interpretations list closes the reading loopholes that prose policy usually leaves open. §6.1 is rigorous — explicit NFKC normalization, a fixed six-line identity serialization, and collision resolution that stops and reports rather than inventing a fallback. Step 11 refusing to fabricate an identifier is the correct failure mode. §11.5 is the strongest section. Reasoning that a single source-level `license.bundleable` cannot safely express "original ships, embedded figure does not," and then keeping the source non-bundleable rather than weakening the boundary, is the right call. The per-document/per-visual gap it identifies was not in the earlier review input — it is a real modeling limitation and worth carrying forward deliberately. ## Findings ### 1. Document identity includes the file checksum, which contradicts §6.3 and causes ID churn on regeneration §6.3 argues that source identifiers should not be content hashes because "file integrity belongs in document-level fields such as checksums." But the canonical document identity in §6.2 includes `checksum_sha256=<file-checksum>`. Consequence: re-running OCR with a different tool version produces different bytes, therefore a different checksum, therefore a different canonical document identity. The natural key `doc_<source>-ocr-text` is already held by the previous identity, so rule 3 fires and a second record is created as `doc_<source>-ocr-text-<hash>`. Rule 9 makes identifiers immutable, so updating the checksum in place is not permitted. Every chunk referencing the previous `document_id` becomes stale. The workflow does not define regeneration semantics — whether the prior record is replaced, superseded, or retained. As written, proliferation is the only conforming path. This affects #14, #15, and particularly #17. Suggested resolution: drop `checksum_sha256` from the document identity tuple, leaving `(source_id, artifact_key, role, language, derived_from)`. Re-derivation then updates the checksum field, the identifier stays stable, and chunk references survive — which is what §6.3 already argues for. ### 2. Identifiers are not re-derivable after a metadata correction §6.1 derives the source key from the normalized title and canonical identity, and also states that identifiers are immutable and that correcting metadata must not rename them. Both rules are correct individually. Together they mean that after any title, publisher, or edition correction, re-deriving the identifier from the committed record produces a different key than the one committed. Determinism therefore holds at first assignment only. #17 needs this stated explicitly: a regeneration test that recomputes identifiers and compares them will fail on any source whose metadata was ever corrected. Committed identifiers must be treated as authoritative input rather than recomputed output. Cheap mitigation: say so in §6.1, and optionally record the canonical-identity string or its hash in the source record at assignment time so the derivation remains auditable. ### 3. The visual-rights rule narrows the #10 corpus more than #10 currently implies §11.4 and §11.5 are correct. Following the consequence: most real-world manuals contain figures whose rights are not separately documented, so unless a license statement affirmatively covers all embedded content, the source stays non-bundleable. Because #27 requires a bundleable proof-loop bundle, the five documents selected in #10 effectively must be born-digital, single-rights-holder, licensed at the whole-work level (CC0 or CC BY), or project-authored. That is a considerably narrower field than #10's current text suggests. Not a flaw in the workflow — a scoping consequence worth recording in #10 before selection begins. ### 4. §6.2 rule 2 truncates the portion that distinguishes documents The natural document key is `<source-key>-<artifact-key>`. A source key may be 64 characters, so `<source-key>-original` already exceeds the 64-character document limit. Rule 2 truncates "the descriptive portion" to 55 characters, which cuts the artifact key — the only part distinguishing `original` from `ocr-text`. Uniqueness survives through the hash suffix, but lineage becomes unreadable during debugging. Suggested change: always preserve the artifact key and truncate only the source-key portion. ### 5. Minor — reviewer identity is coupled to the current host §7.3 specifies the reviewer identifier as "normally the exact Forgejo username." If project hosting ever changes, historical reviewer identities become unresolvable. Recording the host alongside the username, or adopting a project-level contributor identifier, costs nothing at this stage. ## Proposed scope for this issue Limitations 1 through 4 from the #8 closing comment are implementable now without structural change: 1. Add a `review` object carrying `reviewer` and `review_date`. 2. Require non-empty `license.evidence` when `license.bundleable` is true. §7.2 and the license policy both require evidence; nothing enforces it. 3. Add `jurisdiction` and `legal_basis` to the license object, supporting §7.2's public-domain and government-work requirement. 4. Add a cross-field validator rule: `no_derivatives` combined with `bundleable: true` and any derived document must fail. This is precisely §9.1's final bullet and is implementable against the current schema. 5. Require `attribution_text` when `attribution` appears in `conditions`. Plus finding 1 above, which is a change to the identifier rules rather than the schema. Limitations 5 and 6 — per-document and per-asset bundle eligibility, and visual document roles with page, figure, caption, and context linkage — are recommended for a **separate issue**. Both are structural changes to the document contract, and this issue's acceptance criteria exclude wholesale metadata redesign and speculative fields for later milestones. Visual extraction is not required until well past the proof loop, so splitting keeps this issue small and reviewable while allowing the artifact-level release contract to be designed properly on its own terms. Happy to take this issue on that scope if it is useful, and to draft the split-out issue for artifact-level bundle eligibility and visual roles.
JackFrostbyte commented 2026-07-30 00:47:59 -04:00
Author
Owner
Copy link

Completed by PR #33.

The metadata v0.1 schemas, reference validator, documentation, regression tests, and committed worked example are now aligned with the approved source-intake workflow while preserving the existing source → document → chunk model.

The completed work includes:

  • dedicated reviewer identity and review-date metadata;
  • jurisdiction and legal-basis metadata for public-domain and government-work claims;
  • required license evidence for bundleable sources;
  • required access dates for URL-based license evidence;
  • required attribution text when attribution applies;
  • conservative enforcement of inherited no_derivatives restrictions;
  • stable document-identifier behavior when derived artifacts are regenerated;
  • documented first-assignment semantics for deterministic identifiers;
  • positive and negative regression coverage for every new invariant.

All existing validation behavior remains covered, the worked example was intentionally migrated, and the full suite now contains 21 passing tests.

Per-document and per-asset bundle eligibility, visual document roles, and visual linkage metadata are structural extensions rather than intake-alignment changes and remain tracked separately in issue #32.

Issue #9 is complete.

Completed by PR #33. The metadata v0.1 schemas, reference validator, documentation, regression tests, and committed worked example are now aligned with the approved source-intake workflow while preserving the existing source → document → chunk model. The completed work includes: * dedicated reviewer identity and review-date metadata; * jurisdiction and legal-basis metadata for public-domain and government-work claims; * required license evidence for bundleable sources; * required access dates for URL-based license evidence; * required attribution text when attribution applies; * conservative enforcement of inherited `no_derivatives` restrictions; * stable document-identifier behavior when derived artifacts are regenerated; * documented first-assignment semantics for deterministic identifiers; * positive and negative regression coverage for every new invariant. All existing validation behavior remains covered, the worked example was intentionally migrated, and the full suite now contains 21 passing tests. Per-document and per-asset bundle eligibility, visual document roles, and visual linkage metadata are structural extensions rather than intake-alignment changes and remain tracked separately in issue #32. Issue #9 is complete.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feature/issue-17-reproducibility-tests
develop
No results found.
Labels
Clear labels
  
area/deployment
Portable packaging and constrained-environment validation

  
area/governance
Project rules, policies, contributor workflow

  
area/metadata
Schemas, manifests, validation contracts

  
area/pipeline
Import, extraction, normalization, chunking

  
area/search
Indexing, retrieval, citation resolution

  
type/data
Source selection, metadata, fixtures, and corpus work

  
type/design
Architecture, policy, contracts, and ADR work

  
type/implementation
Production code or tooling

  
type/testing
Reproducibility, integration, evaluation, and benchmark work

No labels
area/deployment
area/governance
area/metadata
area/pipeline
area/search
type/data
type/design
type/implementation
type/testing
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
McJuniorstein
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks
#10 Select and approve the five-document pilot corpus
Arkive/arkive
#11 Implement approved-file import and integrity verification
Arkive/arkive
#12 Define the processing-adapter contract and implement UTF-8 plain-text ingestion
Arkive/arkive
#15 Implement deterministic chunk generation
Arkive/arkive
Depends on
#7 Establish the Python project and development-tooling baseline
Arkive/arkive
#8 Define source-intake workflow and repository layout v0.1
Arkive/arkive
Reference
Arkive/arkive#9
No description provided.