Arkive/arkive

Fork 0

Define source-intake workflow and repository layout v0.1 #8

New issue

Closed

opened 2026-07-24 19:38:13 -04:00 by JackFrostbyte · 2 comments

JackFrostbyte commented

2026-07-24 19:38:13 -04:00

Owner

Goal

    Define the review states, evidence requirements, repository layout, and stable naming rules that govern how source candidates enter Arkive.

    ## Why

    The existing policies and schemas define important legal and metadata concepts, but contributors do not yet have an operational workflow for moving a candidate through review or determining where files and evidence belong.

    ## Scope

    - Define the lifecycle and meaning of candidate, pending review, accepted, metadata-only, rejected, and bundleable material.

Specify directories and naming conventions for original files, source records, document records, generated chunks, license evidence, and review evidence.
Define whether unreviewed candidates may temporarily exist outside committed data/, and how that area is handled.
Define rejected-source handling without retaining material that Arkive is not permitted to store.
Define stable source and document identifier generation rules.
Document how contributors verify that a file may legally be committed and redistributed.
Define the evidence required for provenance, license state, reviewer identity, review date, and bundle eligibility.
Describe how accepted files move into processing without bypassing review.

Extend the existing policies rather than replacing them.

  ## Out of scope

  - Implementing import, extraction, normalization, or chunking code.

Selecting the five pilot documents.
Changing the project's license.
Automated downloading or scraping.
Safety-critical source review.

A generalized editorial workflow application.

  ## Dependencies

  - #6 — Add AGENTS.md project rules and align current-state documentation

  ## Acceptance criteria

  - Every supported intake state has an unambiguous definition and permitted repository locations.

The documented workflow prevents unknown-license or unaccepted content from becoming bundleable.
Stable naming and identifier rules are deterministic and documented.
License and review evidence requirements are objective enough for another contributor to follow.
Rejected and metadata-only cases are handled without implying redistribution permission.

The workflow remains consistent with the current source, document, and chunk lineage model.

  ## Validation

  - Walk through at least one hypothetical accepted source, one metadata-only source, and one rejected source using the documented workflow.

Review the design against the source-acceptance and license policies.
Confirm that no source file or production pipeline code is added.

Run git diff --check.

  ## Suggested branch

  `feature/issue-8-source-intake-workflow`

  <!-- arkive-planning-automation:2026-07-24-v3:M1-03 -->

## Goal Define the review states, evidence requirements, repository layout, and stable naming rules that govern how source candidates enter Arkive. ## Why The existing policies and schemas define important legal and metadata concepts, but contributors do not yet have an operational workflow for moving a candidate through review or determining where files and evidence belong. ## Scope - Define the lifecycle and meaning of candidate, pending review, accepted, metadata-only, rejected, and bundleable material. - Specify directories and naming conventions for original files, source records, document records, generated chunks, license evidence, and review evidence. - Define whether unreviewed candidates may temporarily exist outside committed `data/`, and how that area is handled. - Define rejected-source handling without retaining material that Arkive is not permitted to store. - Define stable source and document identifier generation rules. - Document how contributors verify that a file may legally be committed and redistributed. - Define the evidence required for provenance, license state, reviewer identity, review date, and bundle eligibility. - Describe how accepted files move into processing without bypassing review. - Extend the existing policies rather than replacing them. ## Out of scope - Implementing import, extraction, normalization, or chunking code. - Selecting the five pilot documents. - Changing the project's license. - Automated downloading or scraping. - Safety-critical source review. - A generalized editorial workflow application. ## Dependencies - #6 — Add AGENTS.md project rules and align current-state documentation ## Acceptance criteria - Every supported intake state has an unambiguous definition and permitted repository locations. - The documented workflow prevents unknown-license or unaccepted content from becoming bundleable. - Stable naming and identifier rules are deterministic and documented. - License and review evidence requirements are objective enough for another contributor to follow. - Rejected and metadata-only cases are handled without implying redistribution permission. - The workflow remains consistent with the current source, document, and chunk lineage model. ## Validation - Walk through at least one hypothetical accepted source, one metadata-only source, and one rejected source using the documented workflow. - Review the design against the source-acceptance and license policies. - Confirm that no source file or production pipeline code is added. - Run `git diff --check`. ## Suggested branch `feature/issue-8-source-intake-workflow`

JackFrostbyte added this to the Reproducible Acquisition & Processing v0.1 milestone

2026-07-24 19:38:13 -04:00

JackFrostbyte added the

type/design

area/governance

labels

2026-07-24 19:38:13 -04:00

JackFrostbyte referenced this issue

2026-07-24 19:38:14 -04:00

Align metadata schemas and validation with the intake and processing contracts #9

JackFrostbyte added a new dependency

2026-07-24 20:10:47 -04:00

#6 Add AGENTS.md project rules and align current-state documentation

JackFrostbyte added a new dependency

2026-07-24 20:10:47 -04:00

#9 Align metadata schemas and validation with the intake and processing contracts

JackFrostbyte added a new dependency

2026-07-24 20:10:47 -04:00

#10 Select and approve the five-document pilot corpus

JackFrostbyte referenced this issue

2026-07-28 01:46:20 -04:00

Add AGENTS.md project rules and align current-state documentation #6

JackFrostbyte self-assigned this

2026-07-29 02:25:42 -04:00

McJuniorstein commented

2026-07-29 11:28:43 -04:00

Member

Design input for #8 — not a competing draft, you own this one. Two flags first because they affect what you're writing right now, then the longer notes.

⚠️ Flag 1 — `reviewer` and `review_date` don't exist in the schema yet

This issue's scope commits to defining "the evidence required for provenance, license state, reviewer identity, review date, and bundle eligibility."

source.schema.json currently has review_status (pending_review / accepted / rejected) and rejection_reason — but no field recording who reviewed a source or when. There is nowhere to put reviewer identity or review date today.

That's fine, it just means #9 has to add them. The reason to know now: if #8 is written assuming those fields exist, and the pilot corpus (#10, #18) gets recorded before #9 lands, every source record needs a retrofit pass. Cheap to sequence correctly, annoying to fix after the fact.

Suggested shape, minimal and non-speculative:

"review": {
  "reviewer": "string",
  "review_date": "date",
  "notes": "string"
}

⚠️ Flag 2 — derivatives of no-derivatives sources aren't covered by the acceptance criteria

The current acceptance criterion reads:

The documented workflow prevents unknown-license or unaccepted content from becoming bundleable.

That catches unknown-license material and unreviewed material. It does not catch this case:

A source is reviewed properly and accepted — review_status: accepted
Its license is correctly recorded as redistributable_with_conditions with conditions: ["no_derivatives"]
It gets OCR'd, cleaned, and/or translated — all permitted by document.schema.json, which offers transformation.method of ocr, cleanup, translate
The derived documents and their chunks are marked bundleable

Nothing in the current rules stops step 4. The parent is legitimately accepted and legitimately licensed; the derivative is the problem. license_policy.md already states the principle ("A translation may be a derivative work. If the original license does not allow derivative works, Arkive may not be allowed to redistribute the translation") — but it's prose, and the workflow doesn't operationalize it.

Worth checking: no_derivatives, share_alike, and non_commercial currently exist only as enum strings in source.schema.json. Nothing in validate_records.py reads any of them. The vocabulary is defined; the enforcement isn't written yet.

Proposed one-line policy rule for #8, which #9 can then enforce mechanically:

A derived document inherits the redistribution constraints of its source. If the source carries no_derivatives, no document derived from it — and no chunk of such a document — may be bundleable, regardless of its own record.

This is a policy decision, so it belongs here rather than in #9. Worth settling before #9 implements against it.

Design notes

1. Intake states are a matrix, not a third enum

The scope lists "candidate, pending review, accepted, metadata-only, rejected, and bundleable" as one lifecycle. I'd argue they're three different things:

metadata_only is a license status
accepted is a review status
bundleable is a consequence of the other two
candidate isn't a record state at all — it's a file with no committed record yet

The schema already models review and license as orthogonal axes, which is the right call and worth preserving. Collapsing them into a single enum would lose information — you'd no longer be able to express "reviewed and accepted as useful, but not redistributable."

So rather than a new enum, a permission matrix where the cell determines what may exist on disk:

review_status	license.status	file committed?	bundleable
(no record yet)	—	no — quarantine only	no
`pending_review`	any	no	no
`accepted`	`redistributable`	yes	yes
`accepted`	`redistributable_with_conditions`	yes	yes, conditions tracked
`accepted`	`metadata_only`	no — record only	no
`accepted`	`unknown`	no	no
`rejected`	any	no — tombstone only	no

bundleable stays a derived assertion the validator cross-checks, never something a contributor sets by hand in isolation. validate_records.py:158 already enforces exactly this for the license-status and review-status pair — the matrix just makes the full rule explicit and gives the remaining cells somewhere to live.

2. Quarantine — and the reason is git history

On "whether unreviewed candidates may temporarily exist outside committed data/": they must, and I'd put this at the top of the document as the single most important operational rule.

Committing an unreviewed file to git puts it in history permanently. git rm doesn't remove it. .gitignore doesn't help retroactively. If Arkive later receives a takedown for something that shouldn't have been committed, the remedy is a history rewrite propagated to every clone and fork.

This repo is public, so a commit isn't just permanent — it's published the moment it's pushed, and can be cloned or mirrored before anyone notices the mistake.

Practical consequence: the review gate has to sit before the first commit, not before release. An ignored quarantine/ directory where candidates land for review, with files moving into data/sources/ only when the matrix permits. This one rule does more legal risk-reduction than everything downstream of it.

3. Rejected and metadata-only as tombstones

"Handle rejection without retaining material Arkive isn't permitted to store" resolves cleanly using machinery that already exists:

The file is deleted from quarantine and never committed
What's kept is a source.json with review_status: rejected, rejection_reason, origin URL, and reviewer
No documents.json, no chunks.json

The schema and validator already support this shape, so it costs nothing new. The same pattern serves metadata_only: a record with no file. This also satisfies the criterion about not implying redistribution permission — there's simply no artifact to imply it about.

4. An objective evidence bar

To make "objective enough for another contributor to follow" testable, a per-status requirement table:

license.status	required evidence
`redistributable`	≥1 `evidence` entry with a URL or in-document quote, plus `date_accessed`
`redistributable_with_conditions`	the above, `conditions` enumerated, and `attribution_text` populated when `attribution` is among them
`metadata_only` / `pending_review` / `unknown` / `rejected`	reason recorded; no file committed

Two gaps this closes, both currently passing validation:

A source can be redistributable + bundleable: true with evidence: []. license_policy.md:126 says "Do not rely only on memory or assumptions" — but nothing enforces it.
A source can carry conditions: ["attribution"] with no attribution_text. Attribution obligations that exist only as a checkbox tend to get dropped at packaging time.

5. Jurisdiction on public-domain claims

license_policy.md:26 says "government works that are legally public domain in the relevant jurisdiction" — but there's no field recording which jurisdiction, and the contributor checklist doesn't ask.

Two traps worth naming explicitly in the workflow doc:

US federal works (17 U.S.C. §105) are public domain only when produced by federal employees within scope of employment. It does not extend to contractor-produced content, and US government manuals commonly embed third-party figures and diagrams under license. Swapping the FM 21-76 sample for the CC0 fixture in 48f7989 sidestepped exactly this — good call.
Crown copyright (UK, Canada, Australia, NZ) means government works are not public domain by default. Applying US-style reasoning to Commonwealth government documents is probably the most likely way this project ends up holding material it believed was clean.

A jurisdiction field on any public-domain-by-government claim would make this checkable rather than assumed.

Proposed split so we don't collide

None of the above touches #9's territory — it's all policy and workflow definition, so it should merge cleanly with whatever you're drafting.

Looking at the dependency lines across the milestone, M1 is nearly serial with one real fan-out:

#8 ──▶ #9 ──┬──▶ #10 ──┐
            ├──▶ #11    ├──▶ #13 ──▶ #14 ──▶ #15 ──▶ #16/#17 ──▶ #18
            └──▶ #12 ──┘

Worth noting #19 depends on #18 and states Milestone 1 must be complete, so there's no way to start search work early. Everything genuinely is behind #8 right now.

Suggested division, if it's useful:

#8 — yours. You're assigned and mid-draft; the notes above are input, not a competing draft.
#9 — I can take it once #8 merges. It's the sole blocker for three downstream issues, the gaps above are already identified so it's mostly execution, and it stays inside meta/schemas/, pipeline/validate/, and tests/.
After #9 — the fan-out splits naturally. #10 (pilot corpus selection) is judgment and source review, which seems closer to the part you're actually looking forward to. #11 (import + integrity) and #12 (adapter contract) are deterministic code in pipeline/, disjoint from each other and from #10. Then #13 converges on both #10 and #12.

For coordination I'd just follow what AGENTS.md already specifies — assignment on the issue as the visible lock, feature branches pushed early, and each of us staying inside the directory boundaries the protected-boundaries section already carves out. Happy to do it differently if you'd rather keep the schema work yourself.

Design input for #8 — not a competing draft, you own this one. Two flags first because they affect what you're writing right now, then the longer notes. --- ## ⚠️ Flag 1 — `reviewer` and `review_date` don't exist in the schema yet This issue's scope commits to defining "the evidence required for provenance, license state, **reviewer identity, review date**, and bundle eligibility." `source.schema.json` currently has `review_status` (`pending_review` / `accepted` / `rejected`) and `rejection_reason` — but **no field recording who reviewed a source or when**. There is nowhere to put reviewer identity or review date today. That's fine, it just means #9 has to add them. The reason to know now: if #8 is written assuming those fields exist, and the pilot corpus (#10, #18) gets recorded before #9 lands, every source record needs a retrofit pass. Cheap to sequence correctly, annoying to fix after the fact. Suggested shape, minimal and non-speculative: ```json "review": { "reviewer": "string", "review_date": "date", "notes": "string" } ``` ## ⚠️ Flag 2 — derivatives of no-derivatives sources aren't covered by the acceptance criteria The current acceptance criterion reads: > The documented workflow prevents unknown-license or unaccepted content from becoming bundleable. That catches unknown-license material and unreviewed material. It does **not** catch this case: 1. A source is reviewed properly and accepted — `review_status: accepted` 2. Its license is correctly recorded as `redistributable_with_conditions` with `conditions: ["no_derivatives"]` 3. It gets OCR'd, cleaned, and/or translated — all permitted by `document.schema.json`, which offers `transformation.method` of `ocr`, `cleanup`, `translate` 4. The derived documents and their chunks are marked bundleable Nothing in the current rules stops step 4. The parent is legitimately accepted and legitimately licensed; the *derivative* is the problem. `license_policy.md` already states the principle ("A translation may be a derivative work. If the original license does not allow derivative works, Arkive may not be allowed to redistribute the translation") — but it's prose, and the workflow doesn't operationalize it. Worth checking: `no_derivatives`, `share_alike`, and `non_commercial` currently exist **only as enum strings** in `source.schema.json`. Nothing in `validate_records.py` reads any of them. The vocabulary is defined; the enforcement isn't written yet. Proposed one-line policy rule for #8, which #9 can then enforce mechanically: > A derived document inherits the redistribution constraints of its source. If the source carries `no_derivatives`, no document derived from it — and no chunk of such a document — may be bundleable, regardless of its own record. This is a policy decision, so it belongs here rather than in #9. Worth settling before #9 implements against it. --- ## Design notes ### 1. Intake states are a matrix, not a third enum The scope lists "candidate, pending review, accepted, metadata-only, rejected, and bundleable" as one lifecycle. I'd argue they're three different things: - `metadata_only` is a **license** status - `accepted` is a **review** status - `bundleable` is a **consequence** of the other two - `candidate` isn't a record state at all — it's a file with no committed record yet The schema already models review and license as orthogonal axes, which is the right call and worth preserving. Collapsing them into a single enum would lose information — you'd no longer be able to express "reviewed and accepted as useful, but not redistributable." So rather than a new enum, a permission matrix where the cell determines what may exist on disk: | review_status | license.status | file committed? | bundleable | |---|---|---|---| | *(no record yet)* | — | **no — quarantine only** | no | | `pending_review` | any | no | no | | `accepted` | `redistributable` | yes | **yes** | | `accepted` | `redistributable_with_conditions` | yes | yes, conditions tracked | | `accepted` | `metadata_only` | **no — record only** | no | | `accepted` | `unknown` | no | no | | `rejected` | any | **no — tombstone only** | no | `bundleable` stays a derived assertion the validator cross-checks, never something a contributor sets by hand in isolation. `validate_records.py:158` already enforces exactly this for the license-status and review-status pair — the matrix just makes the full rule explicit and gives the remaining cells somewhere to live. ### 2. Quarantine — and the reason is git history On "whether unreviewed candidates may temporarily exist outside committed `data/`": they must, and I'd put this at the top of the document as the single most important operational rule. **Committing an unreviewed file to git puts it in history permanently.** `git rm` doesn't remove it. `.gitignore` doesn't help retroactively. If Arkive later receives a takedown for something that shouldn't have been committed, the remedy is a history rewrite propagated to every clone and fork. **This repo is public**, so a commit isn't just permanent — it's published the moment it's pushed, and can be cloned or mirrored before anyone notices the mistake. Practical consequence: the review gate has to sit **before the first commit**, not before release. An ignored `quarantine/` directory where candidates land for review, with files moving into `data/sources/` only when the matrix permits. This one rule does more legal risk-reduction than everything downstream of it. ### 3. Rejected and metadata-only as tombstones "Handle rejection without retaining material Arkive isn't permitted to store" resolves cleanly using machinery that already exists: - The file is deleted from quarantine and never committed - What's kept is a `source.json` with `review_status: rejected`, `rejection_reason`, origin URL, and reviewer - **No** `documents.json`, **no** `chunks.json` The schema and validator already support this shape, so it costs nothing new. The same pattern serves `metadata_only`: a record with no file. This also satisfies the criterion about not implying redistribution permission — there's simply no artifact to imply it about. ### 4. An objective evidence bar To make "objective enough for another contributor to follow" testable, a per-status requirement table: | license.status | required evidence | |---|---| | `redistributable` | ≥1 `evidence` entry with a URL or in-document quote, **plus `date_accessed`** | | `redistributable_with_conditions` | the above, `conditions` enumerated, and `attribution_text` populated when `attribution` is among them | | `metadata_only` / `pending_review` / `unknown` / `rejected` | reason recorded; no file committed | Two gaps this closes, both currently passing validation: - A source can be `redistributable` + `bundleable: true` with `evidence: []`. `license_policy.md:126` says "Do not rely only on memory or assumptions" — but nothing enforces it. - A source can carry `conditions: ["attribution"]` with no `attribution_text`. Attribution obligations that exist only as a checkbox tend to get dropped at packaging time. ### 5. Jurisdiction on public-domain claims `license_policy.md:26` says "government works that are legally public domain **in the relevant jurisdiction**" — but there's no field recording which jurisdiction, and the contributor checklist doesn't ask. Two traps worth naming explicitly in the workflow doc: - **US federal works** (17 U.S.C. §105) are public domain only when produced by federal *employees within scope of employment*. It does not extend to contractor-produced content, and US government manuals commonly embed third-party figures and diagrams under license. Swapping the FM 21-76 sample for the CC0 fixture in 48f7989 sidestepped exactly this — good call. - **Crown copyright** (UK, Canada, Australia, NZ) means government works are **not** public domain by default. Applying US-style reasoning to Commonwealth government documents is probably the most likely way this project ends up holding material it believed was clean. A `jurisdiction` field on any public-domain-by-government claim would make this checkable rather than assumed. --- ## Proposed split so we don't collide None of the above touches #9's territory — it's all policy and workflow definition, so it should merge cleanly with whatever you're drafting. Looking at the dependency lines across the milestone, M1 is nearly serial with one real fan-out: ``` #8 ──▶ #9 ──┬──▶ #10 ──┐ ├──▶ #11 ├──▶ #13 ──▶ #14 ──▶ #15 ──▶ #16/#17 ──▶ #18 └──▶ #12 ──┘ ``` Worth noting #19 depends on #18 *and* states Milestone 1 must be complete, so there's no way to start search work early. Everything genuinely is behind #8 right now. Suggested division, if it's useful: - **#8 — yours.** You're assigned and mid-draft; the notes above are input, not a competing draft. - **#9 — I can take it** once #8 merges. It's the sole blocker for three downstream issues, the gaps above are already identified so it's mostly execution, and it stays inside `meta/schemas/`, `pipeline/validate/`, and `tests/`. - **After #9 — the fan-out splits naturally.** #10 (pilot corpus selection) is judgment and source review, which seems closer to the part you're actually looking forward to. #11 (import + integrity) and #12 (adapter contract) are deterministic code in `pipeline/`, disjoint from each other and from #10. Then #13 converges on both #10 and #12. For coordination I'd just follow what `AGENTS.md` already specifies — assignment on the issue as the visible lock, feature branches pushed early, and each of us staying inside the directory boundaries the protected-boundaries section already carves out. Happy to do it differently if you'd rather keep the schema work yourself.

JackFrostbyte referenced this issue from a pull request that will close it,

2026-07-29 17:00:10 -04:00

Define source intake workflow and repository layout v0.1 #31

JackFrostbyte closed this issue

2026-07-29 17:18:03 -04:00

JackFrostbyte referenced this issue from a commit

2026-07-29 17:18:05 -04:00

Merge pull request 'Define source intake workflow and repository layout v0.1' (#31) from feature/issue-8-source-intake-workflow into develop

JackFrostbyte commented

2026-07-29 17:18:29 -04:00

Author

Owner

Completed through PR #31.

The merged work establishes Arkive’s source-intake workflow and repository-layout baseline v0.1, including:

candidate quarantine outside committed data/;
pending-review, accepted, accepted non-bundleable, metadata-only, rejected, and bundleable handling;
deterministic source and document identifier rules;
stable source-directory and evidence-layout conventions;
objective provenance, license, reviewer, review-date, and bundle-decision requirements;
legal checks that must occur before original source bytes are committed;
explicit source → document → chunk lineage and later-processing gates;
safe metadata-only and rejected-source tombstone handling;
treatment of diagrams, maps, tables, photographs, figures, and other visuals as potentially authoritative source content;
visual-rights review requirements;
license inheritance for derived documents and chunks;
explicit handling of no_derivatives, attribution, share-alike, non-commercial, and notice conditions;
jurisdiction and legal-basis requirements for government-work and public-domain claims;
hypothetical accepted/bundleable, metadata-only, and rejected walkthroughs.

Validation passed:

metadata validator;
all 8 unit tests;
Ruff lint;
Ruff formatting check;
dependency consistency;
git diff --check;
local candidate ignore-rule verification.

No real source document, production pipeline implementation, schema change, database, web application, or safety-critical corpus material was added.

Remaining follow-up limitations

These limitations are intentionally recorded for Issue #9 and later implementation work:

Reviewer identity, review date, evidence references, and review rationale are operational requirements but are not yet dedicated schema fields.
Provenance and license evidence completeness is not yet fully machine-enforced.
Public-domain jurisdiction and legal basis are documented requirements but are not yet schema fields.
License conditions such as no_derivatives, share_alike, non_commercial, attribution, and notice requirements are not yet comprehensively enforced by the validator.
Per-document and per-visual bundle eligibility cannot currently be represented; the present model exposes only source-level bundle eligibility.
Visual document roles, page numbers, figure identifiers, captions, contextual text links, and visual-dependency flags are not yet represented by the schemas.
Deterministic identifier generation is specified operationally but has not yet been implemented as reusable code.
Candidate intake, downloading, extraction, OCR, normalization, translation, chunk generation, visual extraction, multimodal indexing, and runtime visual display remain unimplemented.
The three validation cases are hypothetical; this issue did not make or approve any real source-license decision.

Issue #9 should evaluate and implement the smallest schema and validator changes necessary to enforce this approved workflow while preserving the existing source → document → chunk lineage model.

Completed through PR #31. The merged work establishes Arkive’s source-intake workflow and repository-layout baseline v0.1, including: * candidate quarantine outside committed `data/`; * pending-review, accepted, accepted non-bundleable, metadata-only, rejected, and bundleable handling; * deterministic source and document identifier rules; * stable source-directory and evidence-layout conventions; * objective provenance, license, reviewer, review-date, and bundle-decision requirements; * legal checks that must occur before original source bytes are committed; * explicit source → document → chunk lineage and later-processing gates; * safe metadata-only and rejected-source tombstone handling; * treatment of diagrams, maps, tables, photographs, figures, and other visuals as potentially authoritative source content; * visual-rights review requirements; * license inheritance for derived documents and chunks; * explicit handling of `no_derivatives`, attribution, share-alike, non-commercial, and notice conditions; * jurisdiction and legal-basis requirements for government-work and public-domain claims; * hypothetical accepted/bundleable, metadata-only, and rejected walkthroughs. Validation passed: * metadata validator; * all 8 unit tests; * Ruff lint; * Ruff formatting check; * dependency consistency; * `git diff --check`; * local candidate ignore-rule verification. No real source document, production pipeline implementation, schema change, database, web application, or safety-critical corpus material was added. ## Remaining follow-up limitations These limitations are intentionally recorded for Issue #9 and later implementation work: 1. Reviewer identity, review date, evidence references, and review rationale are operational requirements but are not yet dedicated schema fields. 2. Provenance and license evidence completeness is not yet fully machine-enforced. 3. Public-domain jurisdiction and legal basis are documented requirements but are not yet schema fields. 4. License conditions such as `no_derivatives`, `share_alike`, `non_commercial`, attribution, and notice requirements are not yet comprehensively enforced by the validator. 5. Per-document and per-visual bundle eligibility cannot currently be represented; the present model exposes only source-level bundle eligibility. 6. Visual document roles, page numbers, figure identifiers, captions, contextual text links, and visual-dependency flags are not yet represented by the schemas. 7. Deterministic identifier generation is specified operationally but has not yet been implemented as reusable code. 8. Candidate intake, downloading, extraction, OCR, normalization, translation, chunk generation, visual extraction, multimodal indexing, and runtime visual display remain unimplemented. 9. The three validation cases are hypothetical; this issue did not make or approve any real source-license decision. Issue #9 should evaluate and implement the smallest schema and validator changes necessary to enforce this approved workflow while preserving the existing source → document → chunk lineage model.

McJuniorstein referenced this issue

2026-07-29 19:18:46 -04:00

Align metadata schemas and validation with the intake and processing contracts #9

McJuniorstein referenced this issue

2026-07-29 19:22:26 -04:00

Select and approve the five-document pilot corpus #10

McJuniorstein referenced this issue

2026-07-29 19:22:27 -04:00

Add clean-checkout regeneration and reproducibility tests #17

McJuniorstein referenced this issue

2026-07-29 19:22:27 -04:00

Define artifact-level bundle eligibility and visual document records #32