Define retrieval evaluation fixtures and success criteria #19

New issue

Open

opened 2026-07-24 19:38:19 -04:00 by JackFrostbyte · 0 comments

JackFrostbyte commented

2026-07-24 19:38:19 -04:00

Owner

Goal

    Create a small human-reviewed retrieval evaluation set for the completed pilot corpus before selecting a full-text search backend.

    ## Why

    Backend selection and later search changes need objective corpus-grounded queries, expected passages, filters, and no-result cases rather than subjective demonstrations.

    ## Scope

    - Create queries with one obvious correct passage.

Create queries with multiple relevant passages and documented relevance expectations.
Include wording variations and non-exact terminology.
Include language, domain, source, or other metadata-filter cases supported by the pilot records.
Include queries that should return no result.
Record expected source, document, chunk, and citation-location identifiers.
Define deterministic evaluation output and success criteria appropriate to a five-document corpus.
Keep the fixtures human-readable and reviewable.
Document that the evaluation measures retrieval only and does not involve an LLM.
```
  ## Out of scope

  - Selecting or implementing a search backend.
```
Embeddings or vector evaluation.
Generated answers.
Large benchmark datasets.

Safety-critical retrieval evaluation.

  ## Dependencies

  - #18 — Process and review the complete five-document pilot corpus

Milestone 1 must be complete before this issue is implemented.

  ## Acceptance criteria

  - The fixture set covers single-answer, multiple-relevant, filter, wording-variation, and no-result cases.

Every expected result resolves to a validated pilot chunk and source passage.
Relevance expectations are human-reviewed and documented.
Evaluation success criteria are deterministic and backend-neutral.

No LLM or network service is required.

  ## Validation

  - Validate every referenced source, document, and chunk ID.

Manually open every expected citation passage.
Review every no-result query for accidental relevant content.

Run the complete Milestone 1 test suite.

  ## Suggested branch

  `feature/issue-19-retrieval-fixtures`

  <!-- arkive-planning-automation:2026-07-24-v3:M2-01 -->

## Goal Create a small human-reviewed retrieval evaluation set for the completed pilot corpus before selecting a full-text search backend. ## Why Backend selection and later search changes need objective corpus-grounded queries, expected passages, filters, and no-result cases rather than subjective demonstrations. ## Scope - Create queries with one obvious correct passage. - Create queries with multiple relevant passages and documented relevance expectations. - Include wording variations and non-exact terminology. - Include language, domain, source, or other metadata-filter cases supported by the pilot records. - Include queries that should return no result. - Record expected source, document, chunk, and citation-location identifiers. - Define deterministic evaluation output and success criteria appropriate to a five-document corpus. - Keep the fixtures human-readable and reviewable. - Document that the evaluation measures retrieval only and does not involve an LLM. ## Out of scope - Selecting or implementing a search backend. - Embeddings or vector evaluation. - Generated answers. - Large benchmark datasets. - Safety-critical retrieval evaluation. ## Dependencies - #18 — Process and review the complete five-document pilot corpus - Milestone 1 must be complete before this issue is implemented. ## Acceptance criteria - The fixture set covers single-answer, multiple-relevant, filter, wording-variation, and no-result cases. - Every expected result resolves to a validated pilot chunk and source passage. - Relevance expectations are human-reviewed and documented. - Evaluation success criteria are deterministic and backend-neutral. - No LLM or network service is required. ## Validation - Validate every referenced source, document, and chunk ID. - Manually open every expected citation passage. - Review every no-result query for accidental relevant content. - Run the complete Milestone 1 test suite. ## Suggested branch `feature/issue-19-retrieval-fixtures`

JackFrostbyte added this to the Offline Search Proof Loop v0.1 milestone

2026-07-24 19:38:19 -04:00

JackFrostbyte added the

area/search

type/testing

labels

2026-07-24 19:38:19 -04:00

JackFrostbyte added a new dependency

2026-07-24 20:10:49 -04:00

#18 Process and review the complete five-document pilot corpus

JackFrostbyte added a new dependency

2026-07-24 20:10:49 -04:00