Arkive/arkive

Fork 0

Add approved five-document pilot corpus #51

Merged

JackFrostbyte merged 1 commit from feature/issue-10-pilot-corpus into develop

2026-08-01 02:43:49 -04:00

JackFrostbyte commented

2026-08-01 02:36:55 -04:00

Owner

Summary

Adds the approved five-document pilot corpus for Issue #10.

The corpus contains exactly five legally redistributable, non-safety-critical source documents:

Naming files, folders and other things
Introducing the Shell
Ten Simple Rules for Taking Advantage of Git and GitHub
Attributions: Use TASL
Summary: Licensing Best Practices for Sharing Scientific Data

The corpus includes:

two pinned UTF-8 Markdown originals;
three born-digital PDF originals;
complete source and document metadata;
provenance, license, and human-review evidence for every source;
verified byte sizes and SHA-256 checksums;
accepted human-review and bundle-eligibility decisions;
locally preserved third-party license notices where required.

The existing Arkive-created reference-library sample remains separate and is not counted among the five pilot documents.

Rights and review

Each source was individually reviewed for:

authoritative provenance;
explicit redistribution permission;
embedded visual and third-party rights;
attribution and license-preservation requirements;
safety classification;
suitability for inclusion in a redistributable Arkive bundle.

All five pilot sources are recorded as:

review_status: accepted
license.bundleable: true
safety_critical: false

Local license snapshots are preserved for:

the Carpentries CC BY 4.0 and MIT terms;
the embedded DM Sans SIL Open Font License 1.1 notice;
the embedded DM Mono SIL Open Font License 1.1 notice.

The original source documents are committed unchanged and match the reviewed candidate files byte for byte.

Repository handling

Adds a root .gitattributes rule:

*.pdf binary

This ensures Git and Forgejo consistently treat committed PDF originals as binary files.

Validation

The following checks completed successfully:

Metadata validation:
- 6 source(s), 6 document(s), 2 chunk(s) across 13 record file(s) — all valid
Complete unit-test suite:
- 21 tests passed
Ruff lint:
- passed
Ruff formatting:
- passed
Python dependency check:
- no broken requirements
qpdf --check for all three pilot PDFs:
- passed
Direct byte-for-byte comparisons against approved candidate files:
- passed
Final pre-staging corpus audit:
- 0 audit failures

Scope boundaries

This pull request does not add:

OCR or scanned documents;
extracted or normalized PDF text;
generated chunks;
indexes or embeddings;
safety-critical source material;
unknown-license or fair-use-only material.

Extraction, normalization, chunking, indexing, and RAG processing remain assigned to downstream issues.

Issue

Implements #10.

## Summary Adds the approved five-document pilot corpus for Issue #10. The corpus contains exactly five legally redistributable, non-safety-critical source documents: - **Naming files, folders and other things** - **Introducing the Shell** - **Ten Simple Rules for Taking Advantage of Git and GitHub** - **Attributions: Use TASL** - **Summary: Licensing Best Practices for Sharing Scientific Data** The corpus includes: - two pinned UTF-8 Markdown originals; - three born-digital PDF originals; - complete source and document metadata; - provenance, license, and human-review evidence for every source; - verified byte sizes and SHA-256 checksums; - accepted human-review and bundle-eligibility decisions; - locally preserved third-party license notices where required. The existing Arkive-created reference-library sample remains separate and is not counted among the five pilot documents. ## Rights and review Each source was individually reviewed for: - authoritative provenance; - explicit redistribution permission; - embedded visual and third-party rights; - attribution and license-preservation requirements; - safety classification; - suitability for inclusion in a redistributable Arkive bundle. All five pilot sources are recorded as: - `review_status: accepted` - `license.bundleable: true` - `safety_critical: false` Local license snapshots are preserved for: - the Carpentries CC BY 4.0 and MIT terms; - the embedded DM Sans SIL Open Font License 1.1 notice; - the embedded DM Mono SIL Open Font License 1.1 notice. The original source documents are committed unchanged and match the reviewed candidate files byte for byte. ## Repository handling Adds a root `.gitattributes` rule: `*.pdf binary` This ensures Git and Forgejo consistently treat committed PDF originals as binary files. ## Validation The following checks completed successfully: - Metadata validation: - `6 source(s), 6 document(s), 2 chunk(s) across 13 record file(s) — all valid` - Complete unit-test suite: - `21 tests passed` - Ruff lint: - passed - Ruff formatting: - passed - Python dependency check: - no broken requirements - `qpdf --check` for all three pilot PDFs: - passed - Direct byte-for-byte comparisons against approved candidate files: - passed - Final pre-staging corpus audit: - `0 audit failures` ## Scope boundaries This pull request does not add: - OCR or scanned documents; - extracted or normalized PDF text; - generated chunks; - indexes or embeddings; - safety-critical source material; - unknown-license or fair-use-only material. Extraction, normalization, chunking, indexing, and RAG processing remain assigned to downstream issues. ## Issue Implements #10.

JackFrostbyte added 1 commit

2026-08-01 02:36:55 -04:00

feat(data): add approved five-document pilot corpus a23a277cfb

JackFrostbyte merged commit fec6453865 into develop

2026-08-01 02:43:49 -04:00

JackFrostbyte deleted branch feature/issue-10-pilot-corpus

2026-08-01 02:43:49 -04:00

JackFrostbyte referenced this pull request from a commit

2026-08-01 02:43:50 -04:00

Merge pull request 'Add approved five-document pilot corpus' (#51) from feature/issue-10-pilot-corpus into develop

JackFrostbyte commented

2026-08-01 02:44:02 -04:00

Author

Owner

Merged successfully into develop.

The approved five-document pilot corpus, supporting evidence, preserved license notices, binary PDF handling, and Issue #10 changelog entry are now part of the development branch.

Post-merge verification can now be completed before closing Issue #10.

Merged successfully into `develop`. The approved five-document pilot corpus, supporting evidence, preserved license notices, binary PDF handling, and Issue #10 changelog entry are now part of the development branch. Post-merge verification can now be completed before closing Issue #10.

JackFrostbyte referenced this pull request

2026-08-02 00:31:38 -04:00

Select and approve the five-document pilot corpus #10