Define the lexical index manifest schema v0.1 #69

Merged

JackFrostbyte merged 2 commits from feature/issue-21-index-manifest into develop 2026-09-16 02:12:17 -04:00

Conversation 2 Commits 2 Files changed 13 +2213 -9

McJuniorstein commented 2026-09-05 17:08:11 -04:00

Member

Copy link

Summary

• Adds meta/schemas/index_manifest.schema.json, the v0.1 contract for one built SQLite FTS5 index per distribution domain. Six closed sections: backend (engine + SQLite version), build (table, columns, unicode61 options, a total insertion order, BM25 with chunk_id tie-break, per ADR-0004's determinism boundary), compatibility (FTS5 required, minimum SQLite, record schema versions, indexed languages), inputs (explicit included and excluded records, per-document checksum + chunk count + chunk digest), artifact (size + SHA-256 per file), and optional provenance.
• Adds arkive.index_manifest with the canonical digest form (arkive-json-canonical-v1: the indexed projection of each chunk, sorted keys, compact, UTF-8, (sequence, chunk_id) order) and verify_manifest, a pure function over loaded records.
• Wires index_manifest.json into the validator. It is verified against the records in the same run and reports missing, duplicate, stale, incompatible, and checksum-invalid conditions with JSON pointers. A user_local manifest under data/ is rejected as contamination (storage boundary §7.1).
• Adds docs/index-manifest.md and updates the schema README, storage-boundary §6.4, development docs, and CHANGELOG.

Design decisions worth a look

1. Timestamps. provenance is the only section allowed to hold one, and it is excluded from reproducibility comparison. Every other section is additionalProperties: false, so a timestamp cannot creep in without a schema change. Tests assert this for each section.
2. Staleness is per-document, over the indexed projection. Digesting only the fields the index reads means a future embedding_ref or token_count on a chunk does not make a current index stale, while any change to text, identity, offsets, citation anchors, or order does. Reordering/reformatting chunks.json is a no-op. Issue #22 should call chunks_digest rather than reimplement it.
3. Completeness is enforced. Every document in the manifest's domain must be included, excluded, or belong to an excluded source. An unlisted document is an error. This is what makes "a new document was added after the build" detectable, which §6.4's stale definition needs.
4. Exclusion reasons are checked claims. no_chunk_records, not_bundleable, review_not_accepted are each verified against the records; operator_exclusion requires a note. Same spirit as the recipe rules: a reason that cannot be wrong is not a reason.
5. Byte identity is bounded, not claimed. backend.sqlite_version plus provenance.environment let a reader tell whether comparing two artifacts byte-for-byte is meaningful at all. minimum_sqlite_version must be ≥ 3.27.0 when remove_diacritics is 2, because that option did not exist earlier.
6. v0.1 has no partial documents. Chunk membership follows document membership. A citation into a document whose other passages are silently missing seemed worse than the flexibility.

The fixture

tests/fixtures/index/pilot-core/ is a real FTS5 database (163,840 bytes, same size as the Issue #20 build) over all 38 committed chunks, built with the ADR-0004 configuration by tests/fixtures/index/make_pilot_core_fixture.py. Two consecutive runs here were byte-identical. It was built under SQLite 3.50.2; the manifest records that. Rebuilding under 3.51.2 will legitimately produce different bytes, which is exactly the boundary the manifest exists to make explicit. The generator is test scaffolding, not the Issue #22 builder, and #22 should supersede it. The fixture is deliberately tied to the committed corpus: adding a source under data/sources/ without regenerating it fails the completeness check.

Honest caveat: this commits a 160 KB binary. If you would rather not, the alternative is to generate the artifact in-test, but then the committed checksum cannot be stable across SQLite versions, so the checksum-invalid tests would have to compare against a checksum computed in the same test run rather than a committed value. I chose the committed artifact so the positive example is a real reviewable file. Easy to flip.

Verification

• git diff --check
• python pipeline/validate/validate_records.py (unchanged corpus still passes, 0 manifests)
• python pipeline/validate/validate_records.py data/sources tests/fixtures/index/pilot-core (PASS, 1 manifest)
• python -m unittest discover -s tests -v — 375 tests, 63 new
• python -m ruff check src pipeline tests
• python -m ruff format --check src pipeline tests
• Mutation pass: each verification rule and each closed schema constraint was disabled in turn and its test confirmed to fail. One constraint (compatibility.requires must contain fts5) is currently unkillable because the enum has only that value; it is kept for when the enum grows.
• Coverage of arkive/index_manifest.py: 97% statements; the uncovered branches are the defensive source is None guards on exclusion-reason checks, reachable only when a document references an unknown source, which the validator already reports separately.
• Rendered Markdown reviewed

Scope confirmation

• No unrelated production behavior changed (validator gains one record type; existing corpus validates unchanged)
• No existing metadata contract changed
• No unreviewed source or generated data was added (the fixture is derived from committed, accepted records)
• No index builder, search runtime, stale-index runtime behavior, or federation

Linked issue

Closes #21

🤖 Generated with Claude Code

https://claude.ai/code/session_015JFC5ueshFKDspps6NRMFU

## Summary - Adds `meta/schemas/index_manifest.schema.json`, the v0.1 contract for one built SQLite FTS5 index per distribution domain. Six closed sections: `backend` (engine + SQLite version), `build` (table, columns, `unicode61` options, a total insertion order, BM25 with `chunk_id` tie-break, per ADR-0004's determinism boundary), `compatibility` (FTS5 required, minimum SQLite, record schema versions, indexed languages), `inputs` (explicit included **and** excluded records, per-document checksum + chunk count + chunk digest), `artifact` (size + SHA-256 per file), and optional `provenance`. - Adds `arkive.index_manifest` with the canonical digest form (`arkive-json-canonical-v1`: the indexed projection of each chunk, sorted keys, compact, UTF-8, `(sequence, chunk_id)` order) and `verify_manifest`, a pure function over loaded records. - Wires `index_manifest.json` into the validator. It is verified against the records in the same run and reports missing, duplicate, stale, incompatible, and checksum-invalid conditions with JSON pointers. A `user_local` manifest under `data/` is rejected as contamination (storage boundary §7.1). - Adds `docs/index-manifest.md` and updates the schema README, storage-boundary §6.4, development docs, and CHANGELOG. ## Design decisions worth a look 1. **Timestamps.** `provenance` is the only section allowed to hold one, and it is excluded from reproducibility comparison. Every other section is `additionalProperties: false`, so a timestamp cannot creep in without a schema change. Tests assert this for each section. 2. **Staleness is per-document, over the indexed projection.** Digesting only the fields the index reads means a future `embedding_ref` or `token_count` on a chunk does not make a current index stale, while any change to text, identity, offsets, citation anchors, or order does. Reordering/reformatting `chunks.json` is a no-op. Issue #22 should call `chunks_digest` rather than reimplement it. 3. **Completeness is enforced.** Every document in the manifest's domain must be included, excluded, or belong to an excluded source. An unlisted document is an error. This is what makes "a new document was added after the build" detectable, which §6.4's stale definition needs. 4. **Exclusion reasons are checked claims.** `no_chunk_records`, `not_bundleable`, `review_not_accepted` are each verified against the records; `operator_exclusion` requires a `note`. Same spirit as the recipe rules: a reason that cannot be wrong is not a reason. 5. **Byte identity is bounded, not claimed.** `backend.sqlite_version` plus `provenance.environment` let a reader tell whether comparing two artifacts byte-for-byte is meaningful at all. `minimum_sqlite_version` must be ≥ 3.27.0 when `remove_diacritics` is 2, because that option did not exist earlier. 6. **v0.1 has no partial documents.** Chunk membership follows document membership. A citation into a document whose other passages are silently missing seemed worse than the flexibility. ## The fixture `tests/fixtures/index/pilot-core/` is a real FTS5 database (163,840 bytes, same size as the Issue #20 build) over all 38 committed chunks, built with the ADR-0004 configuration by `tests/fixtures/index/make_pilot_core_fixture.py`. Two consecutive runs here were byte-identical. It was built under SQLite 3.50.2; the manifest records that. Rebuilding under 3.51.2 will legitimately produce different bytes, which is exactly the boundary the manifest exists to make explicit. The generator is test scaffolding, not the Issue #22 builder, and #22 should supersede it. The fixture is deliberately tied to the committed corpus: adding a source under `data/sources/` without regenerating it fails the completeness check. Honest caveat: this commits a 160 KB binary. If you would rather not, the alternative is to generate the artifact in-test, but then the committed checksum cannot be stable across SQLite versions, so the checksum-invalid tests would have to compare against a checksum computed in the same test run rather than a committed value. I chose the committed artifact so the positive example is a real reviewable file. Easy to flip. ## Verification - [x] `git diff --check` - [x] `python pipeline/validate/validate_records.py` (unchanged corpus still passes, 0 manifests) - [x] `python pipeline/validate/validate_records.py data/sources tests/fixtures/index/pilot-core` (PASS, 1 manifest) - [x] `python -m unittest discover -s tests -v` — 375 tests, 63 new - [x] `python -m ruff check src pipeline tests` - [x] `python -m ruff format --check src pipeline tests` - [x] Mutation pass: each verification rule and each closed schema constraint was disabled in turn and its test confirmed to fail. One constraint (`compatibility.requires` must contain `fts5`) is currently unkillable because the enum has only that value; it is kept for when the enum grows. - [x] Coverage of `arkive/index_manifest.py`: 97% statements; the uncovered branches are the defensive `source is None` guards on exclusion-reason checks, reachable only when a document references an unknown source, which the validator already reports separately. - [x] Rendered Markdown reviewed ## Scope confirmation - [x] No unrelated production behavior changed (validator gains one record type; existing corpus validates unchanged) - [x] No existing metadata contract changed - [x] No unreviewed source or generated data was added (the fixture is derived from committed, accepted records) - [x] No index builder, search runtime, stale-index runtime behavior, or federation ## Linked issue Closes #21 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_015JFC5ueshFKDspps6NRMFU

McJuniorstein added 1 commit 2026-09-05 17:08:11 -04:00

Define the lexical index manifest schema v0.1 ... 8279f243de

Add meta/schemas/index_manifest.schema.json describing one built SQLite FTS5
index per distribution domain: the backend and SQLite version that wrote it,
the deterministic build configuration ADR-0004 requires, the compatibility a
runtime must check before opening it, explicit included and excluded records
with checked exclusion reasons and per-document chunk digests, and artifact
checksums. `provenance` is the only section that may hold a timestamp and is
excluded from reproducibility comparison.

arkive.index_manifest defines the canonical digest form and the cross-record
verification; the validator discovers index_manifest.json and reports missing,
duplicate, stale, incompatible, and checksum-invalid conditions against the
records validated in the same run. A user_local manifest under data/ is
rejected as contamination.

The positive fixture is a real FTS5 database over every committed chunk plus
its manifest, regenerated by tests/fixtures/index/make_pilot_core_fixture.py.
63 tests cover the digest form, each verification rule, and the schema's
closed sections; every rule was mutation-tested against its test.

Closes #21

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015JFC5ueshFKDspps6NRMFU

JackFrostbyte commented 2026-09-13 23:00:12 -04:00

Owner

Copy link

Thanks for the work on this — the implementation looks very solid overall, and the general manifest/reproducibility design is in good shape.

I found one issue I would like corrected before merging, plus one contract point that should be clarified.

Required change: builder tool/version must not be optional

Issue #21 explicitly requires the manifest to record the build tool, backend, and relevant dependency versions.

Right now the builder identity/version is stored under:

provenance.builder.tool
provenance.builder.version

but provenance is optional, and the test suite explicitly verifies that a manifest without provenance is still valid.

That means a schema-valid manifest can currently omit the build tool entirely, which does not satisfy the issue requirement.

Please move the builder identity/version into required reproducible metadata, for example under the existing build section:

build:
  builder:
    tool: ...
    version: ...

or use another required top-level structure if you think that is cleaner.

The optional provenance section can still contain non-reproducible/runtime information such as build timestamp, environment, repository commit, notes, etc.

Please also add a negative validation test confirming that a manifest missing the required builder tool/version is rejected.

Clarify chunk membership semantics

Issue #21 says that included and excluded source, document, and chunk records must be explicit.

The current implementation intentionally does not enumerate individual chunk IDs. Instead, inclusion of a document means all of its chunks are included, with chunk_count and a canonical chunks_digest protecting the exact chunk set.

I actually prefer this design because it will scale much better than storing potentially huge lists of chunk IDs, and the digest covers the important indexed/citation fields.

However, please make this contract explicit in the documentation/schema description so there is no ambiguity for later issues:

In manifest v0.1, document inclusion implies inclusion of all chunks belonging to that document. Exact chunk membership is represented and validated through the document chunk count and canonical chunk-set digest. Partial-document indexing is not supported.

That should make it clear that the current approach intentionally satisfies the chunk-membership requirement without enumerating every chunk.

Other than those points, the implementation looks very strong. The staleness checks, artifact verification, FTS5 build contract, inclusion/exclusion validation, citation fields, and test coverage all look appropriate for what #21 is trying to establish.

Once the required builder metadata issue is fixed and the chunk-membership contract is made explicit, I should be comfortable merging this.

Thanks for the work on this — the implementation looks very solid overall, and the general manifest/reproducibility design is in good shape. I found one issue I would like corrected before merging, plus one contract point that should be clarified. ### Required change: builder tool/version must not be optional Issue #21 explicitly requires the manifest to record the **build tool, backend, and relevant dependency versions**. Right now the builder identity/version is stored under: `provenance.builder.tool` `provenance.builder.version` but `provenance` is optional, and the test suite explicitly verifies that a manifest without provenance is still valid. That means a schema-valid manifest can currently omit the build tool entirely, which does not satisfy the issue requirement. Please move the builder identity/version into required reproducible metadata, for example under the existing `build` section: ```yaml build: builder: tool: ... version: ... ``` or use another required top-level structure if you think that is cleaner. The optional `provenance` section can still contain non-reproducible/runtime information such as build timestamp, environment, repository commit, notes, etc. Please also add a negative validation test confirming that a manifest missing the required builder tool/version is rejected. ### Clarify chunk membership semantics Issue #21 says that included and excluded **source, document, and chunk records** must be explicit. The current implementation intentionally does not enumerate individual chunk IDs. Instead, inclusion of a document means all of its chunks are included, with `chunk_count` and a canonical `chunks_digest` protecting the exact chunk set. I actually prefer this design because it will scale much better than storing potentially huge lists of chunk IDs, and the digest covers the important indexed/citation fields. However, please make this contract explicit in the documentation/schema description so there is no ambiguity for later issues: > In manifest v0.1, document inclusion implies inclusion of all chunks belonging to that document. Exact chunk membership is represented and validated through the document chunk count and canonical chunk-set digest. Partial-document indexing is not supported. That should make it clear that the current approach intentionally satisfies the chunk-membership requirement without enumerating every chunk. Other than those points, the implementation looks very strong. The staleness checks, artifact verification, FTS5 build contract, inclusion/exclusion validation, citation fields, and test coverage all look appropriate for what #21 is trying to establish. Once the required builder metadata issue is fixed and the chunk-membership contract is made explicit, I should be comfortable merging this.

McJuniorstein referenced this pull request from a commit 2026-09-16 01:25:49 -04:00

Require the builder tool and version and state the chunk-membership contract

McJuniorstein added 1 commit 2026-09-16 01:25:49 -04:00

Require the builder tool and version and state the chunk-membership contract ... 5da32c0bf7

Review of PR #69 found that `provenance.builder` was the only record of the
build tool, and `provenance` is optional, so a schema-valid manifest could omit
the tool Issue #21 requires it to record. Move `builder` (tool, version) into
the required `build` section; `provenance` keeps only runtime detail
(timestamp, environment, commit, notes). Three negative tests reject a manifest
with no builder, with a missing or empty tool or version, or with the builder
recorded only under `provenance`; the existing provenance-optional test now
also checks the builder survives dropping that section.

State the chunk-membership contract explicitly in the schema and the design
doc: in manifest v0.1, document inclusion implies inclusion of all of that
document's chunks, exact membership is represented and validated through
`chunk_count` and the canonical `chunks_digest`, and partial-document indexing
is not supported. Chunk IDs are deliberately never enumerated.

Fixture manifest regenerated; the SQLite artifact is byte-identical.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NnHRo83GGQgHxpY6Qzo8ES

McJuniorstein commented 2026-09-16 01:26:04 -04:00

Author

Member

Copy link

Thanks for the review. Both points are addressed in 5da32c0.

Builder tool/version is now required. builder (tool, version, both non-empty) moved out of provenance into the required build section, so it is part of the reproducible metadata and cannot be dropped with the optional section. provenance keeps only runtime detail: built_at, environment, repository_commit, notes. Three negative tests cover it: a manifest with no build.builder, one with a missing or empty tool or version, and one that records the builder only under provenance (rejected on both counts). The existing provenance-optional test now also asserts the builder survives dropping that section. Each new test was mutation-checked against the schema rule it guards.

Chunk-membership contract is explicit. Your wording is in the schema's inputs description verbatim, restated on inputs.documents, and is now its own subsection (2.1) of docs/index-manifest.md, with a note that Issues #22 and #25 must not assume a per-chunk membership list exists. Section 3 explains why build.builder lives in build rather than provenance, and section 7 tells #22 to record itself there.

Fixture manifest regenerated; the SQLite artifact is byte-identical. 378 tests pass, ruff clean.

Thanks for the review. Both points are addressed in 5da32c0. **Builder tool/version is now required.** `builder` (`tool`, `version`, both non-empty) moved out of `provenance` into the required `build` section, so it is part of the reproducible metadata and cannot be dropped with the optional section. `provenance` keeps only runtime detail: `built_at`, `environment`, `repository_commit`, `notes`. Three negative tests cover it: a manifest with no `build.builder`, one with a missing or empty `tool` or `version`, and one that records the builder only under `provenance` (rejected on both counts). The existing provenance-optional test now also asserts the builder survives dropping that section. Each new test was mutation-checked against the schema rule it guards. **Chunk-membership contract is explicit.** Your wording is in the schema's `inputs` description verbatim, restated on `inputs.documents`, and is now its own subsection (2.1) of `docs/index-manifest.md`, with a note that Issues #22 and #25 must not assume a per-chunk membership list exists. Section 3 explains why `build.builder` lives in `build` rather than `provenance`, and section 7 tells #22 to record itself there. Fixture manifest regenerated; the SQLite artifact is byte-identical. 378 tests pass, ruff clean.

JackFrostbyte merged commit 9038952f44 into develop 2026-09-16 02:12:17 -04:00

JackFrostbyte deleted branch feature/issue-21-index-manifest 2026-09-16 02:12:18 -04:00

JackFrostbyte referenced this pull request from a commit 2026-09-16 02:12:19 -04:00

Merge pull request 'Define the lexical index manifest schema v0.1' (#69) from feature/issue-21-index-manifest into develop

JackFrostbyte referenced this pull request 2026-09-16 02:12:47 -04:00

Define index manifest schema v0.1 #21

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

  

area/deployment

Portable packaging and constrained-environment validation

  

area/governance

Project rules, policies, contributor workflow

  

area/metadata

Schemas, manifests, validation contracts

  

area/pipeline

Import, extraction, normalization, chunking

  

area/search

Indexing, retrieval, citation resolution

  

type/data

Source selection, metadata, fixtures, and corpus work

  

type/design

Architecture, policy, contracts, and ADR work

  

type/implementation

Production code or tooling

  

type/testing

Reproducibility, integration, evaluation, and benchmark work

No labels

area/deployment

area/governance

area/metadata

area/pipeline

area/search

type/data

type/design

type/implementation

type/testing

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Arkive/arkive!69

No description provided.