Add v0.1 metadata schemas, validator, and worked example (Issue #3)

JackFrostbyte commented

2026-07-21 00:57:09 -04:00

Owner

Introduce the three metadata contracts the proof loop needs — source,
document, and chunk — under meta/schemas/, with fields derived directly
from the committed source acceptance and license policies. License is an
embedded $defs object in the source schema, able to graduate to its own
schema later without field changes.

Add pipeline/validate/validate_records.py: a reference validator that runs
JSON Schema (draft 2020-12) checks plus two things schema alone can't
express — the cross-field rule that a source is only bundleable under a
redistributable license status, and referential integrity across the three
record types.

Add a worked example record set under data/sources/core/army-survival/
(a public-domain US Army field manual) that validates cleanly, so the
proof-loop spine has real data to exercise.

Defers trust_score, index_manifest, bundle_manifest, and embeddings until
real documents have been run through the loop.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com

Introduce the three metadata contracts the proof loop needs — source, document, and chunk — under meta/schemas/, with fields derived directly from the committed source acceptance and license policies. License is an embedded $defs object in the source schema, able to graduate to its own schema later without field changes. Add pipeline/validate/validate_records.py: a reference validator that runs JSON Schema (draft 2020-12) checks plus two things schema alone can't express — the cross-field rule that a source is only bundleable under a redistributable license status, and referential integrity across the three record types. Add a worked example record set under data/sources/core/army-survival/ (a public-domain US Army field manual) that validates cleanly, so the proof-loop spine has real data to exercise. Defers trust_score, index_manifest, bundle_manifest, and embeddings until real documents have been run through the loop. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

JackFrostbyte added 1 commit

2026-07-21 00:57:10 -04:00

Add v0.1 metadata schemas, validator, and worked example (Issue #3 ) 5239e173ab

Introduce the three metadata contracts the proof loop needs — source,
document, and chunk — under meta/schemas/, with fields derived directly
from the committed source acceptance and license policies. License is an
embedded $defs object in the source schema, able to graduate to its own
schema later without field changes.

Add pipeline/validate/validate_records.py: a reference validator that runs
JSON Schema (draft 2020-12) checks plus two things schema alone can't
express — the cross-field rule that a source is only bundleable under a
redistributable license status, and referential integrity across the three
record types.

Add a worked example record set under data/sources/core/army-survival/
(a public-domain US Army field manual) that validates cleanly, so the
proof-loop spine has real data to exercise.

Defers trust_score, index_manifest, bundle_manifest, and embeddings until
real documents have been run through the loop.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

JackFrostbyte commented

2026-07-21 01:10:40 -04:00

Author

Owner

Thanks — this is a strong and well-scoped implementation, and it satisfies the literal acceptance criteria from Issue #3.

I am requesting a few changes before merge because the metadata contracts are foundational and need to enforce Arkive’s project policies consistently:

license.bundleable: true must also require review_status: accepted. The current example is pending_review while marked bundleable, but the license policy says pending-review sources must not be included in redistributable releases.
Referential integrity must verify source consistency, not only ID existence:
a chunk’s source_id must match the source of its referenced document;
a derived document must have the same source_id as its parent document.
Duplicate chunk_id values should be rejected, just as duplicate source and document IDs are.
The worked FM 21-76 example currently uses a placeholder URL, checksums, file paths, offsets, and an unverified safety-critical passage/page citation. Please replace it with a small project-created sample file with a real checksum and exact offsets, or provide a fully verified source artifact and citation.

Once those points are addressed, the PR should be ready to merge into develop.

<writing-block variant="chat_message"> Thanks — this is a strong and well-scoped implementation, and it satisfies the literal acceptance criteria from Issue #3. I am requesting a few changes before merge because the metadata contracts are foundational and need to enforce Arkive’s project policies consistently: license.bundleable: true must also require review_status: accepted. The current example is pending_review while marked bundleable, but the license policy says pending-review sources must not be included in redistributable releases. Referential integrity must verify source consistency, not only ID existence: a chunk’s source_id must match the source of its referenced document; a derived document must have the same source_id as its parent document. Duplicate chunk_id values should be rejected, just as duplicate source and document IDs are. The worked FM 21-76 example currently uses a placeholder URL, checksums, file paths, offsets, and an unverified safety-critical passage/page citation. Please replace it with a small project-created sample file with a real checksum and exact offsets, or provide a fully verified source artifact and citation. Once those points are addressed, the PR should be ready to merge into develop. </writing-block>

JackFrostbyte added 1 commit

2026-07-21 02:28:40 -04:00

fix(metadata): enforce lineage and bundle eligibility 48f798919e

JackFrostbyte merged commit 4e7442c58f into develop

2026-07-21 02:34:04 -04:00

JackFrostbyte deleted branch feature/issue-3-metadata-schemas-v0.1

2026-07-21 02:34:04 -04:00

JackFrostbyte referenced this pull request from a commit

2026-07-21 02:34:05 -04:00

Merge pull request 'Add v0.1 metadata schemas, validator, and worked example (Issue #3)' (#5) from feature/issue-3-metadata-schemas-v0.1 into develop

JackFrostbyte referenced this pull request

2026-07-21 02:34:38 -04:00

Proposal: Metadata contracts v0.1 — source/document/chunk schemas for the proof loop #3

Rows
Columns

Add v0.1 metadata schemas, validator, and worked example (Issue #3) #5