Detect stale, mismatched, missing, or corrupted indexes #25

New issue

Open

opened 2026-07-24 19:38:22 -04:00 by JackFrostbyte · 0 comments

JackFrostbyte commented

2026-07-24 19:38:22 -04:00

Owner

Goal

    Make the search runtime verify index-manifest compatibility and integrity before returning results.

    ## Why

    An offline index can outlive or become separated from the records that created it. Returning stale results with apparently valid citations would violate Arkive's source-traceability requirements.

    ## Scope

    - Verify the index manifest schema and compatibility before opening the index.

Compare recorded source, document, and chunk input hashes with current validated data.
Verify the index artifact checksum or artifact-set integrity.
Detect changed source records, document checksums, chunk content, chunk identifiers, and build configuration.
Detect missing index files, missing manifests, partial artifact sets, and corruption.
Define explicit failure messages and exit behavior.
Prevent searches when integrity or compatibility cannot be established.

Add focused fixtures for each stale or corrupted state.

  ## Out of scope

  - Automatically rebuilding or updating the index.

Remote index fetching.
Cryptographic signing or public-release verification.
Vector-index integrity.

Generated-answer safeguards.

  ## Dependencies

  - #21 — Define index manifest schema v0.1

#22 — Implement deterministic full-text index generation
#23 — Implement the offline search library and CLI

#24 — Implement citation resolution and exact passage display

  ## Acceptance criteria

  - A valid current index opens and searches normally.

Every specified stale, mismatched, missing, incompatible, or corrupted state is detected.
The runtime refuses to return results when verification fails.
Errors identify the mismatched component without exposing misleading passages.

Integrity checks operate fully offline.

  ## Validation

  - Run the complete unit-test suite.

Exercise changed source metadata, document checksum, chunk text, chunk ID, build configuration, missing artifact, missing manifest, partial artifact, and corrupted index fixtures.

Confirm the CLI returns a non-zero status and no search results for each invalid state.

  ## Suggested branch

  `feature/issue-25-index-integrity`

  <!-- arkive-planning-automation:2026-07-24-v3:M2-07 -->

## Goal Make the search runtime verify index-manifest compatibility and integrity before returning results. ## Why An offline index can outlive or become separated from the records that created it. Returning stale results with apparently valid citations would violate Arkive's source-traceability requirements. ## Scope - Verify the index manifest schema and compatibility before opening the index. - Compare recorded source, document, and chunk input hashes with current validated data. - Verify the index artifact checksum or artifact-set integrity. - Detect changed source records, document checksums, chunk content, chunk identifiers, and build configuration. - Detect missing index files, missing manifests, partial artifact sets, and corruption. - Define explicit failure messages and exit behavior. - Prevent searches when integrity or compatibility cannot be established. - Add focused fixtures for each stale or corrupted state. ## Out of scope - Automatically rebuilding or updating the index. - Remote index fetching. - Cryptographic signing or public-release verification. - Vector-index integrity. - Generated-answer safeguards. ## Dependencies - #21 — Define index manifest schema v0.1 - #22 — Implement deterministic full-text index generation - #23 — Implement the offline search library and CLI - #24 — Implement citation resolution and exact passage display ## Acceptance criteria - A valid current index opens and searches normally. - Every specified stale, mismatched, missing, incompatible, or corrupted state is detected. - The runtime refuses to return results when verification fails. - Errors identify the mismatched component without exposing misleading passages. - Integrity checks operate fully offline. ## Validation - Run the complete unit-test suite. - Exercise changed source metadata, document checksum, chunk text, chunk ID, build configuration, missing artifact, missing manifest, partial artifact, and corrupted index fixtures. - Confirm the CLI returns a non-zero status and no search results for each invalid state. ## Suggested branch `feature/issue-25-index-integrity`

JackFrostbyte added this to the Offline Search Proof Loop v0.1 milestone

2026-07-24 19:38:22 -04:00

JackFrostbyte added the

area/search

type/implementation

labels

2026-07-24 19:38:22 -04:00

JackFrostbyte referenced this issue

2026-07-24 19:38:22 -04:00

Add the complete AI-independent offline proof-loop test #26

JackFrostbyte added a new dependency

2026-07-24 20:10:50 -04:00