Define retrieval evaluation fixtures and success criteria #67

Merged

JackFrostbyte merged 1 commit from feature/issue-19-retrieval-fixtures into develop

2026-08-31 01:16:49 -04:00

JackFrostbyte commented

2026-08-31 01:14:28 -04:00

Owner

Closes #19

Summary

Defines the first deterministic, human-reviewed retrieval evaluation fixture for Arkive's completed five-document Milestone 1 pilot corpus.

The fixture establishes backend-neutral retrieval success criteria before Issue #20 compares lexical search implementations.

Changes
Add tests/fixtures/retrieval/pilot-v0.1.json
10 reviewed retrieval cases
single obvious results
multiple relevant passages
wording variations
source_id metadata filtering
genuine no-result queries
required vs acceptable relevance relationships
stable source/document/chunk/page expectations
Add tests/test_retrieval_evaluation.py
validates the exact approved five-source corpus
validates fixture structure and required case coverage
resolves every positive expectation to committed source/document/chunk records
verifies source → document → chunk lineage
verifies expected passages against committed chunk text
verifies chunk text against cleaned-text [char_start, char_end) offsets
verifies physical page locations where available
validates metadata filters and no-result review requirements
Add docs/retrieval-evaluation.md
documents the v0.1 fixture contract
defines required vs acceptable relevance
defines top-5 success criteria without prescribing exact ranking
documents filter and no-result behavior
documents citation/lineage integrity requirements
explicitly keeps backend selection, embeddings, LLMs, and answer generation out of scope
Validation
canonical record validator: PASS
full unit-test suite: 312 tests PASS
Issue #19 retrieval fixture tests: 8/8 PASS
reproducibility suite: 15/15 PASS
Ruff check: PASS
Ruff format check: PASS
coverage: 92%
pip check: PASS
git diff --check: PASS
untracked-file whitespace validation before commit: PASS

The branch was based on and rechecked against current origin/develop before final validation and commit.

Scope

This PR defines what good retrieval means for the current pilot corpus.

It deliberately does not select or implement a search backend, vector/embedding retrieval, semantic search, LLM answer generation, or a network search service. Those remain follow-up work for Issue #20 and later milestones.

Closes #19 Summary Defines the first deterministic, human-reviewed retrieval evaluation fixture for Arkive's completed five-document Milestone 1 pilot corpus. The fixture establishes backend-neutral retrieval success criteria before Issue #20 compares lexical search implementations. Changes Add tests/fixtures/retrieval/pilot-v0.1.json 10 reviewed retrieval cases single obvious results multiple relevant passages wording variations source_id metadata filtering genuine no-result queries required vs acceptable relevance relationships stable source/document/chunk/page expectations Add tests/test_retrieval_evaluation.py validates the exact approved five-source corpus validates fixture structure and required case coverage resolves every positive expectation to committed source/document/chunk records verifies source → document → chunk lineage verifies expected passages against committed chunk text verifies chunk text against cleaned-text [char_start, char_end) offsets verifies physical page locations where available validates metadata filters and no-result review requirements Add docs/retrieval-evaluation.md documents the v0.1 fixture contract defines required vs acceptable relevance defines top-5 success criteria without prescribing exact ranking documents filter and no-result behavior documents citation/lineage integrity requirements explicitly keeps backend selection, embeddings, LLMs, and answer generation out of scope Validation canonical record validator: PASS full unit-test suite: 312 tests PASS Issue #19 retrieval fixture tests: 8/8 PASS reproducibility suite: 15/15 PASS Ruff check: PASS Ruff format check: PASS coverage: 92% pip check: PASS git diff --check: PASS untracked-file whitespace validation before commit: PASS The branch was based on and rechecked against current origin/develop before final validation and commit. Scope This PR defines what good retrieval means for the current pilot corpus. It deliberately does not select or implement a search backend, vector/embedding retrieval, semantic search, LLM answer generation, or a network search service. Those remain follow-up work for Issue #20 and later milestones.

JackFrostbyte added 1 commit

2026-08-31 01:14:28 -04:00