Arkive/arkive

Fork 0

Define source intake workflow and repository layout v0.1 #31

Merged

JackFrostbyte merged 2 commits from feature/issue-8-source-intake-workflow into develop

2026-07-29 17:18:03 -04:00

JackFrostbyte commented

2026-07-29 17:00:10 -04:00

Owner

Summary

Define Arkive’s source-intake workflow and repository layout v0.1.

This pull request establishes the operational rules governing how candidate source material moves through review, accepted storage, metadata-only handling, rejection, bundle eligibility, and later processing.

It adds a dedicated source-intake document and makes targeted updates to contributor guidance, architecture documentation, metadata documentation, and the existing source-acceptance and license policies.

State model

The workflow separates four related but distinct dimensions:

candidate stage;
review status;
license handling;
bundle eligibility.

It defines handling for:

candidate;
pending review;
accepted and bundleable;
accepted but non-bundleable;
metadata-only;
rejected.

The workflow explicitly prevents candidate, pending-review, unknown-license, metadata-only, or rejected material from being interpreted as automatically redistributable or bundleable.

Repository layout

Committed sources use the existing stable path convention:

data/sources/<category>/<source-key>/

Review state is stored in records and evidence rather than encoded through directories that would require sources to move when their status changes.

The documented layout covers:

original source files;
source records;
document records;
generated chunk records;
derived text;
derived visual artifacts;
provenance evidence;
license evidence;
review evidence.

Unreviewed candidate files are kept in the ignored local-only directory:

local-candidates/

No empty directory trees or .gitkeep files were added.

Stable identifiers

The workflow defines deterministic rules for:

src_<source-key>
doc_<source-key>-<artifact-key>

The rules cover:

canonical identity strings;
lowercase ASCII normalization;
deterministic handling of non-ASCII-only identities;
schema-compatible length limits;
deterministic truncation;
collision suffixes;
identifier immutability;
distinction between source identity and document-file integrity.

File checksums remain document-level integrity evidence rather than becoming source identifiers.

Evidence requirements

The workflow defines objective evidence requirements for:

provenance;
exact work and edition identification;
acquisition path;
license state;
redistribution rights;
derivative rights;
embedded third-party material;
reviewer identity;
review date;
review rationale;
bundle eligibility;
unresolved concerns and required conditions.

It also includes a legal-commit checklist that must be completed before original source bytes may be committed.

Visual material

Visual material is explicitly treated as part of the authoritative source when it contributes to meaning or procedure.

The workflow covers:

diagrams;
photographs;
maps;
figures;
tables;
charts;
page-layout context;
later derived visual artifacts;
captions, OCR, and contextual linkage;
future text-linked and multimodal retrieval;
the requirement to surface essential visuals with relevant answers.

It also documents that embedded visuals may have rights different from the surrounding text and must not be assumed bundleable.

Under the current source-level licensing contract, a source containing unresolved visual rights remains non-bundleable until a later schema and packaging design can safely represent per-artifact eligibility.

Hypothetical walkthroughs

The documentation validates the workflow using three clearly labeled hypothetical cases:

an accepted and bundleable source;
an accepted metadata-only source;
a rejected source.

No real candidate or source file was added.

Policy and documentation updates

This pull request:

links contributor guidance to the source-intake workflow;
requires agents to read the workflow before source work;
extends the source-acceptance policy without replacing it;
extends the license policy to cover embedded visuals and evidence uncertainty;
documents the current architecture boundary;
records current schema and validator limitations;
updates the documented unit-test count;
adds the change to the changelog.

Validation

The following checks passed:

git diff --check
python pipeline/validate/validate_records.py
python -m unittest discover -s tests -v
python -m ruff check src pipeline tests
python -m ruff format --check src pipeline tests
python -m pip check
git check-ignore -v local-candidates/hypothetical-candidate.pdf

Results:

metadata validator passed for the committed worked example;
all 8 unit tests passed;
Ruff lint passed;
Ruff formatting check passed;
installed dependencies passed consistency checks;
the local candidate directory is correctly ignored;
Markdown links and code fences were inspected;
no trailing whitespace or conflict markers were found.

Scope confirmation

This pull request does not:

add real source files;
select pilot documents;
add safety-critical content;
change metadata schemas;
add import, extraction, OCR, normalization, chunking, indexing, or download code;
add a database, web application, or task queue;
change the Arkive project license.

Known follow-up limitations

The workflow intentionally documents several requirements that are not yet machine-enforced:

reviewer identity, review date, and evidence paths are not dedicated schema fields;
visual document roles and page/figure linkage are not yet represented by the schemas;
per-document and per-visual bundle eligibility is not currently representable;
deterministic identifier generation is specified but not yet implemented as code;
import, extraction, OCR, chunking, visual indexing, and runtime visual display remain future work.

Issue #9 should evaluate the smallest schema and validator changes needed to support the approved workflow without weakening the existing source → document → chunk lineage model.

Closes #8

## Summary Define Arkive’s source-intake workflow and repository layout v0.1. This pull request establishes the operational rules governing how candidate source material moves through review, accepted storage, metadata-only handling, rejection, bundle eligibility, and later processing. It adds a dedicated source-intake document and makes targeted updates to contributor guidance, architecture documentation, metadata documentation, and the existing source-acceptance and license policies. ## State model The workflow separates four related but distinct dimensions: 1. candidate stage; 2. review status; 3. license handling; 4. bundle eligibility. It defines handling for: * candidate; * pending review; * accepted and bundleable; * accepted but non-bundleable; * metadata-only; * rejected. The workflow explicitly prevents candidate, pending-review, unknown-license, metadata-only, or rejected material from being interpreted as automatically redistributable or bundleable. ## Repository layout Committed sources use the existing stable path convention: ```text data/sources/<category>/<source-key>/ ``` Review state is stored in records and evidence rather than encoded through directories that would require sources to move when their status changes. The documented layout covers: * original source files; * source records; * document records; * generated chunk records; * derived text; * derived visual artifacts; * provenance evidence; * license evidence; * review evidence. Unreviewed candidate files are kept in the ignored local-only directory: ```text local-candidates/ ``` No empty directory trees or `.gitkeep` files were added. ## Stable identifiers The workflow defines deterministic rules for: ```text src_<source-key> doc_<source-key>-<artifact-key> ``` The rules cover: * canonical identity strings; * lowercase ASCII normalization; * deterministic handling of non-ASCII-only identities; * schema-compatible length limits; * deterministic truncation; * collision suffixes; * identifier immutability; * distinction between source identity and document-file integrity. File checksums remain document-level integrity evidence rather than becoming source identifiers. ## Evidence requirements The workflow defines objective evidence requirements for: * provenance; * exact work and edition identification; * acquisition path; * license state; * redistribution rights; * derivative rights; * embedded third-party material; * reviewer identity; * review date; * review rationale; * bundle eligibility; * unresolved concerns and required conditions. It also includes a legal-commit checklist that must be completed before original source bytes may be committed. ## Visual material Visual material is explicitly treated as part of the authoritative source when it contributes to meaning or procedure. The workflow covers: * diagrams; * photographs; * maps; * figures; * tables; * charts; * page-layout context; * later derived visual artifacts; * captions, OCR, and contextual linkage; * future text-linked and multimodal retrieval; * the requirement to surface essential visuals with relevant answers. It also documents that embedded visuals may have rights different from the surrounding text and must not be assumed bundleable. Under the current source-level licensing contract, a source containing unresolved visual rights remains non-bundleable until a later schema and packaging design can safely represent per-artifact eligibility. ## Hypothetical walkthroughs The documentation validates the workflow using three clearly labeled hypothetical cases: 1. an accepted and bundleable source; 2. an accepted metadata-only source; 3. a rejected source. No real candidate or source file was added. ## Policy and documentation updates This pull request: * links contributor guidance to the source-intake workflow; * requires agents to read the workflow before source work; * extends the source-acceptance policy without replacing it; * extends the license policy to cover embedded visuals and evidence uncertainty; * documents the current architecture boundary; * records current schema and validator limitations; * updates the documented unit-test count; * adds the change to the changelog. ## Validation The following checks passed: ```text git diff --check python pipeline/validate/validate_records.py python -m unittest discover -s tests -v python -m ruff check src pipeline tests python -m ruff format --check src pipeline tests python -m pip check git check-ignore -v local-candidates/hypothetical-candidate.pdf ``` Results: * metadata validator passed for the committed worked example; * all 8 unit tests passed; * Ruff lint passed; * Ruff formatting check passed; * installed dependencies passed consistency checks; * the local candidate directory is correctly ignored; * Markdown links and code fences were inspected; * no trailing whitespace or conflict markers were found. ## Scope confirmation This pull request does not: * add real source files; * select pilot documents; * add safety-critical content; * change metadata schemas; * add import, extraction, OCR, normalization, chunking, indexing, or download code; * add a database, web application, or task queue; * change the Arkive project license. ## Known follow-up limitations The workflow intentionally documents several requirements that are not yet machine-enforced: * reviewer identity, review date, and evidence paths are not dedicated schema fields; * visual document roles and page/figure linkage are not yet represented by the schemas; * per-document and per-visual bundle eligibility is not currently representable; * deterministic identifier generation is specified but not yet implemented as code; * import, extraction, OCR, chunking, visual indexing, and runtime visual display remain future work. Issue #9 should evaluate the smallest schema and validator changes needed to support the approved workflow without weakening the existing source → document → chunk lineage model. Closes #8

JackFrostbyte added 1 commit

2026-07-29 17:00:10 -04:00