Implement deterministic full-text index generation #22

New issue

Open

opened 2026-07-24 19:38:20 -04:00 by JackFrostbyte · 0 comments

JackFrostbyte commented

2026-07-24 19:38:20 -04:00

Owner

Goal

    Build a deterministic offline lexical index from validated, accepted, bundleable pilot chunks and emit a valid index manifest.

    ## Why

    The pilot corpus needs a reproducible index that preserves Arkive lineage and refuses invalid or ineligible inputs before a search runtime can be trusted.

    ## Scope

    - Read validated source, document, and chunk records through a documented input boundary.

Reject schema-invalid or cross-record-invalid metadata.
Exclude unaccepted, metadata-only, rejected, or otherwise non-bundleable sources.
Preserve chunk, document, and source IDs in the index.
Apply the backend configuration recorded by the index manifest.
Detect duplicate, missing, or stale inputs.
Build the index offline without network calls.
Emit the index artifact and a validated manifest.
Produce deterministic output or explicitly isolate and document unavoidable backend variation.

Use non-destructive or atomic output handling.

  ## Out of scope

  - Search UI or generated answers.

Vector or hybrid indexes.
Automatic source processing.
A deployment bundle.

Large-corpus scaling work.

  ## Dependencies

  - #21 — Define index manifest schema v0.1

  ## Acceptance criteria

  - All eligible pilot chunks are indexed exactly once.

Ineligible or invalid records are rejected or excluded according to documented rules.
Source, document, and chunk identifiers are recoverable from every result record.
The emitted manifest validates and matches the built artifact.
Repeated builds are identical under the documented determinism contract.
A failed build does not replace a valid existing index.

The builder operates without network access.

  ## Validation

  - Run the complete unit-test suite.

Build the pilot index twice and compare artifacts and manifests.
Exercise invalid, duplicate, stale, and non-bundleable input fixtures.
Run with networking disabled.

Verify index integrity against the manifest.

  ## Suggested branch

  `feature/issue-22-index-builder`

  <!-- arkive-planning-automation:2026-07-24-v3:M2-04 -->

## Goal Build a deterministic offline lexical index from validated, accepted, bundleable pilot chunks and emit a valid index manifest. ## Why The pilot corpus needs a reproducible index that preserves Arkive lineage and refuses invalid or ineligible inputs before a search runtime can be trusted. ## Scope - Read validated source, document, and chunk records through a documented input boundary. - Reject schema-invalid or cross-record-invalid metadata. - Exclude unaccepted, metadata-only, rejected, or otherwise non-bundleable sources. - Preserve chunk, document, and source IDs in the index. - Apply the backend configuration recorded by the index manifest. - Detect duplicate, missing, or stale inputs. - Build the index offline without network calls. - Emit the index artifact and a validated manifest. - Produce deterministic output or explicitly isolate and document unavoidable backend variation. - Use non-destructive or atomic output handling. ## Out of scope - Search UI or generated answers. - Vector or hybrid indexes. - Automatic source processing. - A deployment bundle. - Large-corpus scaling work. ## Dependencies - #21 — Define index manifest schema v0.1 ## Acceptance criteria - All eligible pilot chunks are indexed exactly once. - Ineligible or invalid records are rejected or excluded according to documented rules. - Source, document, and chunk identifiers are recoverable from every result record. - The emitted manifest validates and matches the built artifact. - Repeated builds are identical under the documented determinism contract. - A failed build does not replace a valid existing index. - The builder operates without network access. ## Validation - Run the complete unit-test suite. - Build the pilot index twice and compare artifacts and manifests. - Exercise invalid, duplicate, stale, and non-bundleable input fixtures. - Run with networking disabled. - Verify index integrity against the manifest. ## Suggested branch `feature/issue-22-index-builder`

JackFrostbyte added this to the Offline Search Proof Loop v0.1 milestone

2026-07-24 19:38:20 -04:00

JackFrostbyte added the

type/implementation

area/search

labels

2026-07-24 19:38:20 -04:00

JackFrostbyte referenced this issue

2026-07-24 19:38:22 -04:00

Detect stale, mismatched, missing, or corrupted indexes #25

JackFrostbyte referenced this issue

2026-07-24 19:38:22 -04:00