Arkive/arkive
1
0
Fork 0
Code Issues 14 Pull requests Projects Releases Packages Wiki Activity Actions

Implement deterministic full-text index generation #22

New issue
Closed
opened 2026-07-24 19:38:20 -04:00 by JackFrostbyte · 2 comments
JackFrostbyte commented 2026-07-24 19:38:20 -04:00
Owner
Copy link

Goal

    Build a deterministic offline lexical index from validated, accepted, bundleable pilot chunks and emit a valid index manifest.

    ## Why

    The pilot corpus needs a reproducible index that preserves Arkive lineage and refuses invalid or ineligible inputs before a search runtime can be trusted.

    ## Scope

    - Read validated source, document, and chunk records through a documented input boundary.

  • Reject schema-invalid or cross-record-invalid metadata.

  • Exclude unaccepted, metadata-only, rejected, or otherwise non-bundleable sources.

  • Preserve chunk, document, and source IDs in the index.

  • Apply the backend configuration recorded by the index manifest.

  • Detect duplicate, missing, or stale inputs.

  • Build the index offline without network calls.

  • Emit the index artifact and a validated manifest.

  • Produce deterministic output or explicitly isolate and document unavoidable backend variation.

  • Use non-destructive or atomic output handling.

      ## Out of scope

  - Search UI or generated answers.

  • Vector or hybrid indexes.

  • Automatic source processing.

  • A deployment bundle.

  • Large-corpus scaling work.

      ## Dependencies

  - #21 — Define index manifest schema v0.1

  ## Acceptance criteria

  - All eligible pilot chunks are indexed exactly once.

  • Ineligible or invalid records are rejected or excluded according to documented rules.

  • Source, document, and chunk identifiers are recoverable from every result record.

  • The emitted manifest validates and matches the built artifact.

  • Repeated builds are identical under the documented determinism contract.

  • A failed build does not replace a valid existing index.

  • The builder operates without network access.

      ## Validation

  - Run the complete unit-test suite.

  • Build the pilot index twice and compare artifacts and manifests.

  • Exercise invalid, duplicate, stale, and non-bundleable input fixtures.

  • Run with networking disabled.

  • Verify index integrity against the manifest.

      ## Suggested branch

  `feature/issue-22-index-builder`

  <!-- arkive-planning-automation:2026-07-24-v3:M2-04 -->
## Goal Build a deterministic offline lexical index from validated, accepted, bundleable pilot chunks and emit a valid index manifest. ## Why The pilot corpus needs a reproducible index that preserves Arkive lineage and refuses invalid or ineligible inputs before a search runtime can be trusted. ## Scope - Read validated source, document, and chunk records through a documented input boundary. - Reject schema-invalid or cross-record-invalid metadata. - Exclude unaccepted, metadata-only, rejected, or otherwise non-bundleable sources. - Preserve chunk, document, and source IDs in the index. - Apply the backend configuration recorded by the index manifest. - Detect duplicate, missing, or stale inputs. - Build the index offline without network calls. - Emit the index artifact and a validated manifest. - Produce deterministic output or explicitly isolate and document unavoidable backend variation. - Use non-destructive or atomic output handling. ## Out of scope - Search UI or generated answers. - Vector or hybrid indexes. - Automatic source processing. - A deployment bundle. - Large-corpus scaling work. ## Dependencies - #21 — Define index manifest schema v0.1 ## Acceptance criteria - All eligible pilot chunks are indexed exactly once. - Ineligible or invalid records are rejected or excluded according to documented rules. - Source, document, and chunk identifiers are recoverable from every result record. - The emitted manifest validates and matches the built artifact. - Repeated builds are identical under the documented determinism contract. - A failed build does not replace a valid existing index. - The builder operates without network access. ## Validation - Run the complete unit-test suite. - Build the pilot index twice and compare artifacts and manifests. - Exercise invalid, duplicate, stale, and non-bundleable input fixtures. - Run with networking disabled. - Verify index integrity against the manifest. ## Suggested branch `feature/issue-22-index-builder` <!-- arkive-planning-automation:2026-07-24-v3:M2-04 -->
McJuniorstein commented 2026-09-19 11:44:59 -04:00
Member
Copy link

Opened PR #70 on feature/issue-22-index-builder implementing the builder as arkive.index_build (installed as arkive-build-index). It records itself under build.builder, uses arkive.index_manifest.chunks_digest, and honours the all-chunks-per-document membership contract from #21. Eligibility rules, the eight fail-closed stages, and the determinism claims are in docs/index-build.md. The pilot-core fixture is now produced by the builder itself; its artifact bytes are unchanged.

Opened PR #70 on `feature/issue-22-index-builder` implementing the builder as `arkive.index_build` (installed as `arkive-build-index`). It records itself under `build.builder`, uses `arkive.index_manifest.chunks_digest`, and honours the all-chunks-per-document membership contract from #21. Eligibility rules, the eight fail-closed stages, and the determinism claims are in `docs/index-build.md`. The pilot-core fixture is now produced by the builder itself; its artifact bytes are unchanged.
JackFrostbyte commented 2026-10-02 22:40:10 -04:00
Author
Owner
Copy link

Completed by PR #70.

The deterministic lexical index builder is now implemented using SQLite FTS5, following the backend decision from ADR-0004 and the index-manifest contract from Issue #21.

The implementation includes validated record-tree input, explicit eligibility/exclusion handling, document and chunk integrity verification, deterministic insertion/build behavior, manifest generation and validation, offline execution, safe replacement with rollback, symlink-aware storage-boundary enforcement, and fail-closed handling of unexpected SQLite side files.

The final branch was synchronized with develop and the full validation suite passed before merge.

Closing as complete.

Completed by PR #70. The deterministic lexical index builder is now implemented using SQLite FTS5, following the backend decision from ADR-0004 and the index-manifest contract from Issue #21. The implementation includes validated record-tree input, explicit eligibility/exclusion handling, document and chunk integrity verification, deterministic insertion/build behavior, manifest generation and validation, offline execution, safe replacement with rollback, symlink-aware storage-boundary enforcement, and fail-closed handling of unexpected SQLite side files. The final branch was synchronized with `develop` and the full validation suite passed before merge. Closing as complete.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
develop
No results found.
Labels
Clear labels
  
area/deployment
Portable packaging and constrained-environment validation

  
area/governance
Project rules, policies, contributor workflow

  
area/metadata
Schemas, manifests, validation contracts

  
area/pipeline
Import, extraction, normalization, chunking

  
area/search
Indexing, retrieval, citation resolution

  
type/data
Source selection, metadata, fixtures, and corpus work

  
type/design
Architecture, policy, contracts, and ADR work

  
type/implementation
Production code or tooling

  
type/testing
Reproducibility, integration, evaluation, and benchmark work

No labels
area/deployment
area/governance
area/metadata
area/pipeline
area/search
type/data
type/design
type/implementation
type/testing
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
McJuniorstein
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks
#23 Implement the offline search library and CLI
Arkive/arkive
#25 Detect stale, mismatched, missing, or corrupted indexes
Arkive/arkive
#26 Add the complete AI-independent offline proof-loop test
Arkive/arkive
#48 Generate and validate a separate local-content index
Arkive/arkive
Depends on
#21 Define index manifest schema v0.1
Arkive/arkive
Reference
Arkive/arkive#22
No description provided.