Arkive/arkive
1
0
Fork 0
Code Issues 18 Pull requests 1 Projects Releases Packages Wiki Activity Actions

Select and approve the five-document pilot corpus #10

New issue
Closed
opened 2026-07-24 19:38:14 -04:00 by JackFrostbyte · 2 comments
JackFrostbyte commented 2026-07-24 19:38:14 -04:00
Owner
Copy link

Goal

    Commit five legally redistributable, non-safety-critical pilot source files with complete provenance, license evidence, review status, and verified integrity metadata.

    ## Why

    The processing pipeline needs a small but varied real corpus that can exercise plain-text and born-digital PDF handling without introducing safety-critical review risk.

    ## Scope

    - Select exactly five small documents with legally verifiable redistribution permission.

  • Use only non-safety-critical subjects suitable for an early processing pilot.

  • Include both UTF-8 plain text and born-digital PDF.

  • Exercise meaningful variation in document length, headings or sections, and harmless practical subject matter.

  • Record provenance, publisher or author information when available, acquisition details, license evidence, reviewer identity, review date, and accepted status.

  • Commit each real original file under the approved intake layout.

  • Record and independently verify byte size and SHA-256 for every file.

  • Ensure the existing Arkive-created reference-library sample remains a separate regression fixture and is not counted among the five documents.

      ## Out of scope

  - Medical, water-treatment, food-preservation, electrical, chemical, structural, or other safety-critical content.

  • OCR or scanned PDFs.

  • Unknown-license, fair-use-only, or merely web-accessible material.

  • PDF extraction, normalization, chunking, or index generation.

  • Wikipedia-scale or scraped corpora.

      ## Dependencies

  - #8 — Define source-intake workflow and repository layout v0.1

  • #9 — Align metadata schemas and validation with the intake and processing contracts

      ## Acceptance criteria

  - Exactly five pilot documents are accepted and committed.

  • Every file has verifiable provenance and explicit redistribution evidence.

  • Every corresponding source record is accepted and legally bundleable under the existing policy and schema rules.

  • Recorded byte sizes and SHA-256 values match the committed files.

  • The corpus contains at least one UTF-8 text file and at least one born-digital PDF.

  • No pilot document is safety-critical or requires OCR.

      ## Validation

  - Independently open every original file and confirm it is readable and is the reviewed work.

  • Recalculate byte size and SHA-256 for all five files.

  • Validate the source metadata and cross-record relationships.

  • Manually review every license-evidence reference.

  • Run the complete unit-test suite.

      ## Suggested branch

  `feature/issue-10-pilot-corpus`

  <!-- arkive-planning-automation:2026-07-24-v3:M1-05 -->
## Goal Commit five legally redistributable, non-safety-critical pilot source files with complete provenance, license evidence, review status, and verified integrity metadata. ## Why The processing pipeline needs a small but varied real corpus that can exercise plain-text and born-digital PDF handling without introducing safety-critical review risk. ## Scope - Select exactly five small documents with legally verifiable redistribution permission. - Use only non-safety-critical subjects suitable for an early processing pilot. - Include both UTF-8 plain text and born-digital PDF. - Exercise meaningful variation in document length, headings or sections, and harmless practical subject matter. - Record provenance, publisher or author information when available, acquisition details, license evidence, reviewer identity, review date, and accepted status. - Commit each real original file under the approved intake layout. - Record and independently verify byte size and SHA-256 for every file. - Ensure the existing Arkive-created reference-library sample remains a separate regression fixture and is not counted among the five documents. ## Out of scope - Medical, water-treatment, food-preservation, electrical, chemical, structural, or other safety-critical content. - OCR or scanned PDFs. - Unknown-license, fair-use-only, or merely web-accessible material. - PDF extraction, normalization, chunking, or index generation. - Wikipedia-scale or scraped corpora. ## Dependencies - #8 — Define source-intake workflow and repository layout v0.1 - #9 — Align metadata schemas and validation with the intake and processing contracts ## Acceptance criteria - Exactly five pilot documents are accepted and committed. - Every file has verifiable provenance and explicit redistribution evidence. - Every corresponding source record is accepted and legally bundleable under the existing policy and schema rules. - Recorded byte sizes and SHA-256 values match the committed files. - The corpus contains at least one UTF-8 text file and at least one born-digital PDF. - No pilot document is safety-critical or requires OCR. ## Validation - Independently open every original file and confirm it is readable and is the reviewed work. - Recalculate byte size and SHA-256 for all five files. - Validate the source metadata and cross-record relationships. - Manually review every license-evidence reference. - Run the complete unit-test suite. ## Suggested branch `feature/issue-10-pilot-corpus` <!-- arkive-planning-automation:2026-07-24-v3:M1-05 -->
McJuniorstein commented 2026-07-29 19:22:26 -04:00
Member
Copy link

Scoping consequence from the merged #8 workflow that is worth recording here before source selection begins.

docs/source-intake.md §11.4 and §11.5 establish that visual material carries its own rights, and that where the documented redistribution basis does not clearly cover embedded photographs, illustrations, maps, figures, and tables, the source must not be treated as bundleable. §11.5 goes further: because v0.1 provides only a single source-level license.bundleable value and no per-asset decision, an original file containing unresolved embedded visual rights remains non-bundleable as a whole.

That is the correct conservative rule. The consequence for this issue is that the practical candidate pool is considerably narrower than the current scope text implies.

Most real-world field manuals, repair guides, and scanned references contain figures, diagrams, or photographs whose rights are not separately documented — frequently third-party material licensed into the work rather than owned by the publisher. Under §11.5 those sources cannot be bundled, only accepted as non-bundleable or metadata-only.

Because #27 requires a portable proof-loop bundle, the five documents selected here effectively need to satisfy one of:

  • born-digital works with a single rights holder and no embedded third-party visuals;
  • works carrying a whole-work open license that explicitly covers all content, not just the text (CC0, CC BY, or equivalent);
  • works whose publisher rights statement affirmatively addresses embedded material;
  • Arkive-created or otherwise project-authored material.

US federal government publications deserve specific caution here rather than being treated as a safe default. 17 U.S.C. §105 covers works produced by federal employees within the scope of employment; it does not extend to contractor-produced content, and government field manuals commonly embed third-party diagrams and figures under license. The FM 21-76 fixture was replaced in 48f7989 for exactly this reason.

Suggested addition to this issue's scope: record the explicit bundle-eligibility selection criteria before candidate review begins, so that a candidate's visual-rights position is assessed during selection rather than discovered during #18. Otherwise this issue risks selecting five documents and then finding that several cannot be bundled, which would block #27.

If the intent is instead that the pilot corpus need not be fully bundleable at this stage, stating that explicitly here would also resolve the ambiguity — but #27 should then record that it may need different material.

Scoping consequence from the merged #8 workflow that is worth recording here before source selection begins. `docs/source-intake.md` §11.4 and §11.5 establish that visual material carries its own rights, and that where the documented redistribution basis does not clearly cover embedded photographs, illustrations, maps, figures, and tables, the source must not be treated as bundleable. §11.5 goes further: because v0.1 provides only a single source-level `license.bundleable` value and no per-asset decision, an original file containing unresolved embedded visual rights **remains non-bundleable as a whole**. That is the correct conservative rule. The consequence for this issue is that the practical candidate pool is considerably narrower than the current scope text implies. Most real-world field manuals, repair guides, and scanned references contain figures, diagrams, or photographs whose rights are not separately documented — frequently third-party material licensed into the work rather than owned by the publisher. Under §11.5 those sources cannot be bundled, only accepted as non-bundleable or metadata-only. Because #27 requires a portable proof-loop bundle, the five documents selected here effectively need to satisfy one of: - born-digital works with a single rights holder and no embedded third-party visuals; - works carrying a whole-work open license that explicitly covers all content, not just the text (CC0, CC BY, or equivalent); - works whose publisher rights statement affirmatively addresses embedded material; - Arkive-created or otherwise project-authored material. US federal government publications deserve specific caution here rather than being treated as a safe default. 17 U.S.C. §105 covers works produced by federal employees within the scope of employment; it does not extend to contractor-produced content, and government field manuals commonly embed third-party diagrams and figures under license. The FM 21-76 fixture was replaced in 48f7989 for exactly this reason. Suggested addition to this issue's scope: record the explicit bundle-eligibility selection criteria before candidate review begins, so that a candidate's visual-rights position is assessed during selection rather than discovered during #18. Otherwise this issue risks selecting five documents and then finding that several cannot be bundled, which would block #27. If the intent is instead that the pilot corpus need not be fully bundleable at this stage, stating that explicitly here would also resolve the ambiguity — but #27 should then record that it may need different material.
JackFrostbyte commented 2026-08-01 02:44:52 -04:00
Author
Owner
Copy link

Issue #10 is complete.

Pull request #51 was reviewed and merged into develop.

The repository now contains the approved five-document pilot corpus:

  • two pinned UTF-8 Markdown originals;
  • three born-digital PDF originals;
  • complete source and document metadata;
  • provenance, licensing, and human-review evidence for every source;
  • verified checksums and byte-for-byte original-file integrity;
  • accepted and bundleable decisions for all five sources;
  • non-safety-critical classifications for all five sources;
  • preserved Carpentries, DM Sans, and DM Mono license notices;
  • consistent binary handling for PDF files through .gitattributes.

The existing Arkive-created reference sample remains separate and is not counted among the five pilot documents.

Final validation passed:

  • 6 sources, 6 documents, and 2 fixture chunks across 13 record files;
  • all 21 unit tests;
  • Ruff lint and formatting;
  • Python dependency validation;
  • structural checks for all three PDFs;
  • complete pre-staging audit with zero failures.

No OCR, extracted text, generated chunks, indexes, embeddings, or other downstream processing artifacts were added.

Closing Issue #10 as completed.

Issue #10 is complete. Pull request #51 was reviewed and merged into `develop`. The repository now contains the approved five-document pilot corpus: - two pinned UTF-8 Markdown originals; - three born-digital PDF originals; - complete source and document metadata; - provenance, licensing, and human-review evidence for every source; - verified checksums and byte-for-byte original-file integrity; - accepted and bundleable decisions for all five sources; - non-safety-critical classifications for all five sources; - preserved Carpentries, DM Sans, and DM Mono license notices; - consistent binary handling for PDF files through `.gitattributes`. The existing Arkive-created reference sample remains separate and is not counted among the five pilot documents. Final validation passed: - 6 sources, 6 documents, and 2 fixture chunks across 13 record files; - all 21 unit tests; - Ruff lint and formatting; - Python dependency validation; - structural checks for all three PDFs; - complete pre-staging audit with zero failures. No OCR, extracted text, generated chunks, indexes, embeddings, or other downstream processing artifacts were added. Closing Issue #10 as completed.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feature/issue-17-reproducibility-tests
develop
No results found.
Labels
Clear labels
  
area/deployment
Portable packaging and constrained-environment validation

  
area/governance
Project rules, policies, contributor workflow

  
area/metadata
Schemas, manifests, validation contracts

  
area/pipeline
Import, extraction, normalization, chunking

  
area/search
Indexing, retrieval, citation resolution

  
type/data
Source selection, metadata, fixtures, and corpus work

  
type/design
Architecture, policy, contracts, and ADR work

  
type/implementation
Production code or tooling

  
type/testing
Reproducibility, integration, evaluation, and benchmark work

No labels
area/deployment
area/governance
area/metadata
area/pipeline
area/search
type/data
type/design
type/implementation
type/testing
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
JackFrostbyte
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks
#11 Implement approved-file import and integrity verification
Arkive/arkive
#13 Implement born-digital PDF text extraction
Arkive/arkive
#18 Process and review the complete five-document pilot corpus
Arkive/arkive
Depends on
#8 Define source-intake workflow and repository layout v0.1
Arkive/arkive
#9 Align metadata schemas and validation with the intake and processing contracts
Arkive/arkive
Reference
Arkive/arkive#10
No description provided.