Arkive/arkive
1
0
Fork 0
Code Issues 14 Pull requests Projects Releases Packages Wiki Activity Actions

Define retrieval evaluation fixtures and success criteria #19

New issue
Closed
opened 2026-07-24 19:38:19 -04:00 by JackFrostbyte · 1 comment
JackFrostbyte commented 2026-07-24 19:38:19 -04:00
Owner
Copy link

Goal

    Create a small human-reviewed retrieval evaluation set for the completed pilot corpus before selecting a full-text search backend.

    ## Why

    Backend selection and later search changes need objective corpus-grounded queries, expected passages, filters, and no-result cases rather than subjective demonstrations.

    ## Scope

    - Create queries with one obvious correct passage.

  • Create queries with multiple relevant passages and documented relevance expectations.

  • Include wording variations and non-exact terminology.

  • Include language, domain, source, or other metadata-filter cases supported by the pilot records.

  • Include queries that should return no result.

  • Record expected source, document, chunk, and citation-location identifiers.

  • Define deterministic evaluation output and success criteria appropriate to a five-document corpus.

  • Keep the fixtures human-readable and reviewable.

  • Document that the evaluation measures retrieval only and does not involve an LLM.

      ## Out of scope

  - Selecting or implementing a search backend.

  • Embeddings or vector evaluation.

  • Generated answers.

  • Large benchmark datasets.

  • Safety-critical retrieval evaluation.

      ## Dependencies

  - #18 — Process and review the complete five-document pilot corpus

  • Milestone 1 must be complete before this issue is implemented.

      ## Acceptance criteria

  - The fixture set covers single-answer, multiple-relevant, filter, wording-variation, and no-result cases.

  • Every expected result resolves to a validated pilot chunk and source passage.

  • Relevance expectations are human-reviewed and documented.

  • Evaluation success criteria are deterministic and backend-neutral.

  • No LLM or network service is required.

      ## Validation

  - Validate every referenced source, document, and chunk ID.

  • Manually open every expected citation passage.

  • Review every no-result query for accidental relevant content.

  • Run the complete Milestone 1 test suite.

      ## Suggested branch

  `feature/issue-19-retrieval-fixtures`

  <!-- arkive-planning-automation:2026-07-24-v3:M2-01 -->
## Goal Create a small human-reviewed retrieval evaluation set for the completed pilot corpus before selecting a full-text search backend. ## Why Backend selection and later search changes need objective corpus-grounded queries, expected passages, filters, and no-result cases rather than subjective demonstrations. ## Scope - Create queries with one obvious correct passage. - Create queries with multiple relevant passages and documented relevance expectations. - Include wording variations and non-exact terminology. - Include language, domain, source, or other metadata-filter cases supported by the pilot records. - Include queries that should return no result. - Record expected source, document, chunk, and citation-location identifiers. - Define deterministic evaluation output and success criteria appropriate to a five-document corpus. - Keep the fixtures human-readable and reviewable. - Document that the evaluation measures retrieval only and does not involve an LLM. ## Out of scope - Selecting or implementing a search backend. - Embeddings or vector evaluation. - Generated answers. - Large benchmark datasets. - Safety-critical retrieval evaluation. ## Dependencies - #18 — Process and review the complete five-document pilot corpus - Milestone 1 must be complete before this issue is implemented. ## Acceptance criteria - The fixture set covers single-answer, multiple-relevant, filter, wording-variation, and no-result cases. - Every expected result resolves to a validated pilot chunk and source passage. - Relevance expectations are human-reviewed and documented. - Evaluation success criteria are deterministic and backend-neutral. - No LLM or network service is required. ## Validation - Validate every referenced source, document, and chunk ID. - Manually open every expected citation passage. - Review every no-result query for accidental relevant content. - Run the complete Milestone 1 test suite. ## Suggested branch `feature/issue-19-retrieval-fixtures` <!-- arkive-planning-automation:2026-07-24-v3:M2-01 -->
JackFrostbyte commented 2026-08-31 01:17:10 -04:00
Author
Owner
Copy link

Completed by PR #67.

Issue #19 now provides the human-reviewed retrieval benchmark for the completed five-document Milestone 1 corpus.

The delivered fixture contains 10 deterministic cases covering:

  • single obvious relevant passages
  • multiple relevant passages with documented required / acceptable relationships
  • wording variations that do not simply reproduce source terminology
  • metadata filtering using the existing source_id contract
  • genuine no-result queries manually checked against the full 36-chunk pilot corpus

Every positive expectation identifies real committed source, document, and chunk IDs, with physical page locations where available. Tests verify lineage, literal expected passages, cleaned-text [char_start, char_end) integrity, and citation/page identity.

Success is defined backend-neutrally: required results must appear within the top 5, filters must be respected, and no-result cases must return zero results. Exact ranking, scoring, tokenization, search implementation, embeddings, LLMs, and generated answers remain deliberately out of scope.

This establishes the common evaluation set needed by Issue #20 to compare lexical search backends without selecting or optimizing for one in advance.

Closing as complete.

Completed by PR #67. Issue #19 now provides the human-reviewed retrieval benchmark for the completed five-document Milestone 1 corpus. The delivered fixture contains 10 deterministic cases covering: * single obvious relevant passages * multiple relevant passages with documented `required` / `acceptable` relationships * wording variations that do not simply reproduce source terminology * metadata filtering using the existing `source_id` contract * genuine no-result queries manually checked against the full 36-chunk pilot corpus Every positive expectation identifies real committed source, document, and chunk IDs, with physical page locations where available. Tests verify lineage, literal expected passages, cleaned-text `[char_start, char_end)` integrity, and citation/page identity. Success is defined backend-neutrally: required results must appear within the top 5, filters must be respected, and no-result cases must return zero results. Exact ranking, scoring, tokenization, search implementation, embeddings, LLMs, and generated answers remain deliberately out of scope. This establishes the common evaluation set needed by Issue #20 to compare lexical search backends without selecting or optimizing for one in advance. Closing as complete.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
develop
No results found.
Labels
Clear labels
  
area/deployment
Portable packaging and constrained-environment validation

  
area/governance
Project rules, policies, contributor workflow

  
area/metadata
Schemas, manifests, validation contracts

  
area/pipeline
Import, extraction, normalization, chunking

  
area/search
Indexing, retrieval, citation resolution

  
type/data
Source selection, metadata, fixtures, and corpus work

  
type/design
Architecture, policy, contracts, and ADR work

  
type/implementation
Production code or tooling

  
type/testing
Reproducibility, integration, evaluation, and benchmark work

No labels
area/deployment
area/governance
area/metadata
area/pipeline
area/search
type/data
type/design
type/implementation
type/testing
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
JackFrostbyte
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks
#20 Evaluate offline full-text search backends and record the decision
Arkive/arkive
#26 Add the complete AI-independent offline proof-loop test
Arkive/arkive
Depends on
#18 Process and review the complete five-document pilot corpus
Arkive/arkive
Reference
Arkive/arkive#19
No description provided.