Arkive/arkive

Fork 0

Define artifact-level bundle eligibility and visual document records #32

New issue

Open

opened 2026-07-29 19:22:27 -04:00 by McJuniorstein · 1 comment

McJuniorstein commented

2026-07-29 19:22:27 -04:00

Member

Goal

Define an artifact-level bundle-eligibility contract and the document-record support needed to represent extracted visual material, so that per-document and per-visual redistribution decisions can be expressed and enforced rather than collapsed into a single source-level flag.

Why

The v0.1 intake workflow merged in #31 records two limitations that the metadata contracts cannot currently express:

bundle eligibility exists only as a single source-level license.bundleable value, with no per-document or per-asset decision;
the document schema provides no roles or linkage fields for extracted figures, diagrams, tables, or page images.

docs/source-intake.md §11.5 resolves this conservatively: because v0.1 cannot safely mark an original bundleable while excluding embedded visuals inside the same file, any source with unresolved embedded visual rights stays entirely non-bundleable. That is the correct rule under the current contract, but it means genuinely useful sources are withheld whenever a single figure has an unclear rights position.

§6.2 already reserves deterministic visual artifact keys (page-<four-digit-page>-figure-<three-digit-sequence> and similar) while explicitly forbidding such records from being committed until schema support exists. This issue is that schema support.

This is deliberately separated from #9. #9 covers the smallest changes needed to enforce the approved intake workflow against the existing source → document → chunk model. The work here is a structural extension of the document contract and a new release-decision concept, which #9's acceptance criteria exclude.

Scope

Define whether bundle eligibility belongs on document records, on a separate release or bundle manifest, or both.
Define how an artifact-level decision interacts with the existing source-level license.bundleable value, including which one governs when they disagree.
Define how inherited source conditions (no_derivatives, share_alike, non_commercial, attribution, notice) constrain artifact-level inclusion.
Add document roles for extracted visual artifacts.
Add the linkage fields §11.2 requires: parent document, page number, figure or table identifier, caption, contextual heading or nearby text, extraction method, and any OCR text or clearly labeled machine-generated description.
Preserve explicit source → document → chunk lineage and the existing half-open chunk offset model.
Update cross-record validation so an artifact-level inclusion cannot claim rights the source evidence does not support.
Update meta/schemas/README.md and the affected sections of docs/source-intake.md, including §11.5's interim restriction once it no longer applies.
Add positive and negative regression tests for every new rule.
Document the metadata compatibility decision in the pull request.

Out of scope

Implementing visual extraction, OCR, captioning, or page rendering.
Multimodal or vector indexing.
Runtime visual display.
Index, vector, translation, speech, or model manifests.
Selecting or processing pilot corpus material.
Adding real source documents or visual assets.
Trust scoring.
Changing the project license.

Dependencies

#9 — Align metadata schemas and validation with the intake and processing contracts

Acceptance criteria

An artifact-level bundle decision can be represented unambiguously, and its precedence relative to the source-level decision is documented.
A source with a clearly redistributable original and one unresolved embedded visual can be represented without either over-claiming rights or discarding the source.
Inherited source conditions cannot be bypassed through an artifact-level inclusion.
Extracted visual artifacts can be recorded with parent document, page, figure identifier, caption, and contextual linkage.
Existing lineage, duplicate-identifier, source-consistency, bundle-eligibility, and offset validations continue to pass.
New invariants have positive and negative regression tests.
The committed worked example remains valid or is intentionally migrated with documented reasoning.
docs/source-intake.md §11.5 is updated to reflect whatever the new contract permits, with no stale interim restriction left in place.
No visual extraction, indexing, or runtime behavior is implemented.

Validation

Run the complete unit-test suite.
Validate every committed source, document, and chunk record.
Exercise negative fixtures for each new validation rule, including an artifact-level inclusion that conflicts with source-level evidence and one that conflicts with an inherited no_derivatives condition.
Confirm no source document or visual asset was added.
Review the rendered Markdown for all updated documentation.
Run git diff --check.

Suggested branch

feature/artifact-level-bundle-eligibility

Notes

Milestone assignment is left open. This work is not required for the Milestone 1 proof loop or the Milestone 2 offline search loop, but §11.5's interim restriction remains in force until it is done, so it may become a practical blocker for bundling real corpus material.

Raised from the review pass on #9 (see that issue's review comment) and from limitations 5 and 6 recorded in the #8 closing comment.

## Goal Define an artifact-level bundle-eligibility contract and the document-record support needed to represent extracted visual material, so that per-document and per-visual redistribution decisions can be expressed and enforced rather than collapsed into a single source-level flag. ## Why The v0.1 intake workflow merged in #31 records two limitations that the metadata contracts cannot currently express: - bundle eligibility exists only as a single source-level `license.bundleable` value, with no per-document or per-asset decision; - the document schema provides no roles or linkage fields for extracted figures, diagrams, tables, or page images. `docs/source-intake.md` §11.5 resolves this conservatively: because v0.1 cannot safely mark an original bundleable while excluding embedded visuals inside the same file, any source with unresolved embedded visual rights stays entirely non-bundleable. That is the correct rule under the current contract, but it means genuinely useful sources are withheld whenever a single figure has an unclear rights position. §6.2 already reserves deterministic visual artifact keys (`page-<four-digit-page>-figure-<three-digit-sequence>` and similar) while explicitly forbidding such records from being committed until schema support exists. This issue is that schema support. This is deliberately separated from #9. #9 covers the smallest changes needed to enforce the approved intake workflow against the existing source → document → chunk model. The work here is a structural extension of the document contract and a new release-decision concept, which #9's acceptance criteria exclude. ## Scope - Define whether bundle eligibility belongs on document records, on a separate release or bundle manifest, or both. - Define how an artifact-level decision interacts with the existing source-level `license.bundleable` value, including which one governs when they disagree. - Define how inherited source conditions (`no_derivatives`, `share_alike`, `non_commercial`, attribution, notice) constrain artifact-level inclusion. - Add document roles for extracted visual artifacts. - Add the linkage fields §11.2 requires: parent document, page number, figure or table identifier, caption, contextual heading or nearby text, extraction method, and any OCR text or clearly labeled machine-generated description. - Preserve explicit source → document → chunk lineage and the existing half-open chunk offset model. - Update cross-record validation so an artifact-level inclusion cannot claim rights the source evidence does not support. - Update `meta/schemas/README.md` and the affected sections of `docs/source-intake.md`, including §11.5's interim restriction once it no longer applies. - Add positive and negative regression tests for every new rule. - Document the metadata compatibility decision in the pull request. ## Out of scope - Implementing visual extraction, OCR, captioning, or page rendering. - Multimodal or vector indexing. - Runtime visual display. - Index, vector, translation, speech, or model manifests. - Selecting or processing pilot corpus material. - Adding real source documents or visual assets. - Trust scoring. - Changing the project license. ## Dependencies - #9 — Align metadata schemas and validation with the intake and processing contracts ## Acceptance criteria - An artifact-level bundle decision can be represented unambiguously, and its precedence relative to the source-level decision is documented. - A source with a clearly redistributable original and one unresolved embedded visual can be represented without either over-claiming rights or discarding the source. - Inherited source conditions cannot be bypassed through an artifact-level inclusion. - Extracted visual artifacts can be recorded with parent document, page, figure identifier, caption, and contextual linkage. - Existing lineage, duplicate-identifier, source-consistency, bundle-eligibility, and offset validations continue to pass. - New invariants have positive and negative regression tests. - The committed worked example remains valid or is intentionally migrated with documented reasoning. - `docs/source-intake.md` §11.5 is updated to reflect whatever the new contract permits, with no stale interim restriction left in place. - No visual extraction, indexing, or runtime behavior is implemented. ## Validation - Run the complete unit-test suite. - Validate every committed source, document, and chunk record. - Exercise negative fixtures for each new validation rule, including an artifact-level inclusion that conflicts with source-level evidence and one that conflicts with an inherited `no_derivatives` condition. - Confirm no source document or visual asset was added. - Review the rendered Markdown for all updated documentation. - Run `git diff --check`. ## Suggested branch `feature/artifact-level-bundle-eligibility` ## Notes Milestone assignment is left open. This work is not required for the Milestone 1 proof loop or the Milestone 2 offline search loop, but §11.5's interim restriction remains in force until it is done, so it may become a practical blocker for bundling real corpus material. Raised from the review pass on #9 (see that issue's review comment) and from limitations 5 and 6 recorded in the #8 closing comment.

McJuniorstein added the

area/metadata

type/design

labels

2026-07-29 19:22:27 -04:00

McJuniorstein commented

2026-07-29 20:49:18 -04:00

Author

Member

Design path for this issue, and a scope reduction that follows from it.

Reframing the problem

The visual-rights framework in docs/source-intake.md §11 is correct, but it can be read as requiring Arkive to establish per-figure rights provenance for arbitrary existing works. That reading makes the problem intractable — illustrations in commercial works are routinely licensed separately from stock agencies, museums, or commissioned artists, and the publisher frequently holds no redistributable right in them at all. Tracing that backward for an arbitrary book is not a solvable research task at project scale.

Arkive does not need to solve that problem. The tractable version is a sourcing and selection problem: prefer material whose visual rights are already documented, and supply the visual directly when they are not. Nothing needs to be traced backward.

This distinction should be recorded in the workflow, because the difference between the two readings is the difference between a feasible project and an infeasible one.

Categories where visual rights are already resolved

Public domain by age. When a work's copyright term has expired, illustrations published in that work expired with it — they were published simultaneously and no separate copyright survives to be cleared. This is the largest clean category and it aligns closely with Arkive's mission: pre-modern and early-modern practical references on agriculture, carpentry, blacksmithing, masonry, food preservation, hand-tool machining, veterinary practice, and herbals. Two cautions: verify the specific edition, since modern reprints may add newly authored illustrations still in term; and confirm the term against the relevant jurisdiction rather than assuming a single rule. A faithful reproduction of a two-dimensional public-domain page does not generate a new copyright in the major jurisdictions Arkive is likely to care about.

Open-licensed works that already cleared their figures. Open textbook projects had to clear or replace third-party figures in order to release under CC BY, and commonly carry per-figure attribution as a result. Where that documentation exists, Arkive inherits it rather than reproducing the effort.

Known-rights visual libraries. Repositories that maintain per-file rights records provide substitutes when a needed illustration from an otherwise usable source cannot be redistributed.

Project-authored visuals. Copyright protects expression, not facts, procedures, or methods. An independently created diagram conveying the same functional information as an unusable illustration is not a derivative of it. Technical and functional drawings also attract comparatively thin protection because much of their content is dictated by function. Project-authored figures can therefore be released under the project's own terms and are unconditionally bundleable.

This last category matters most for design purposes: it means an essential figure with unusable rights does not block the source. It converts a hard legal blocker into an authoring task.

None of the above is legal advice, and the analysis differs by jurisdiction. It should be validated for the jurisdictions Arkive actually operates and distributes in before being relied on for a real bundle decision.

Consequence: three ways to resolve an unusable visual

Where §11.5 currently permits only "keep the whole source non-bundleable," the contract designed here should be able to express three distinct resolutions:

Exclude and cite — ship the text, record the figure as referenced but not included, and preserve a citation precise enough for a reader to consult the original. §11.3 already anticipates this by requiring the system to state that an answer is incomplete without the cited visual.
Substitute — ship a replacement visual from a known-rights source, recorded as a substitution rather than as an extraction from the parent document.
Re-author — ship a project-created figure conveying the same functional information, recorded as project-authored rather than derived from the source.

Options 2 and 3 both introduce an artifact that is associated with a source but not derived from it. That is a lineage relationship the current document model cannot express, since derived_from implies transformation of a parent artifact. This issue should define it.

Suggested scope reduction

This issue does not need to model exhaustive per-figure provenance. Three narrower pieces appear sufficient:

A source-level visual-rights posture recording whether the documented evidence covers embedded visual material, does not cover it, or leaves it unresolved. This is a single reviewed judgement rather than a per-figure inventory, and it is what §11.4 already asks a reviewer to determine.
An artifact-level bundle decision on document records, so a text artifact can be included while a specific visual artifact is excluded — the core limitation §11.5 identifies.
A substitution and authorship relationship distinguishing extracted-from-source visuals, substituted known-rights visuals, and project-authored visuals, so a bundled figure never implies a rights claim over the original it replaces.

Per-figure rights records remain possible later for sources where the evidence genuinely supports that granularity, but requiring them now is what makes the problem feel unbounded. A reviewed posture plus an artifact-level decision plus honest substitution provenance is enough to be legally conservative without demanding exhaustive research.

Relationship to Milestone 1

This work is not required to complete the proof loop. #10 selects five documents, and those five can be chosen specifically for unambiguous visual rights — project-authored, public domain by age, or whole-work open licensed. §11.5's conservative interim rule handles everything else correctly in the meantime.

Deferring this issue until real corpus material exists is preferable to designing the artifact-level contract against hypotheticals, since the substitution and re-authoring cases are much easier to model correctly once there are concrete figures to reason about.

Design path for this issue, and a scope reduction that follows from it. ## Reframing the problem The visual-rights framework in `docs/source-intake.md` §11 is correct, but it can be read as requiring Arkive to establish per-figure rights provenance for arbitrary existing works. That reading makes the problem intractable — illustrations in commercial works are routinely licensed separately from stock agencies, museums, or commissioned artists, and the publisher frequently holds no redistributable right in them at all. Tracing that backward for an arbitrary book is not a solvable research task at project scale. Arkive does not need to solve that problem. The tractable version is a **sourcing and selection** problem: prefer material whose visual rights are already documented, and supply the visual directly when they are not. Nothing needs to be traced backward. This distinction should be recorded in the workflow, because the difference between the two readings is the difference between a feasible project and an infeasible one. ## Categories where visual rights are already resolved **Public domain by age.** When a work's copyright term has expired, illustrations published in that work expired with it — they were published simultaneously and no separate copyright survives to be cleared. This is the largest clean category and it aligns closely with Arkive's mission: pre-modern and early-modern practical references on agriculture, carpentry, blacksmithing, masonry, food preservation, hand-tool machining, veterinary practice, and herbals. Two cautions: verify the specific edition, since modern reprints may add newly authored illustrations still in term; and confirm the term against the relevant jurisdiction rather than assuming a single rule. A faithful reproduction of a two-dimensional public-domain page does not generate a new copyright in the major jurisdictions Arkive is likely to care about. **Open-licensed works that already cleared their figures.** Open textbook projects had to clear or replace third-party figures in order to release under CC BY, and commonly carry per-figure attribution as a result. Where that documentation exists, Arkive inherits it rather than reproducing the effort. **Known-rights visual libraries.** Repositories that maintain per-file rights records provide substitutes when a needed illustration from an otherwise usable source cannot be redistributed. **Project-authored visuals.** Copyright protects expression, not facts, procedures, or methods. An independently created diagram conveying the same functional information as an unusable illustration is not a derivative of it. Technical and functional drawings also attract comparatively thin protection because much of their content is dictated by function. Project-authored figures can therefore be released under the project's own terms and are unconditionally bundleable. This last category matters most for design purposes: it means an essential figure with unusable rights does not block the source. It converts a hard legal blocker into an authoring task. None of the above is legal advice, and the analysis differs by jurisdiction. It should be validated for the jurisdictions Arkive actually operates and distributes in before being relied on for a real bundle decision. ## Consequence: three ways to resolve an unusable visual Where §11.5 currently permits only "keep the whole source non-bundleable," the contract designed here should be able to express three distinct resolutions: 1. **Exclude and cite** — ship the text, record the figure as referenced but not included, and preserve a citation precise enough for a reader to consult the original. §11.3 already anticipates this by requiring the system to state that an answer is incomplete without the cited visual. 2. **Substitute** — ship a replacement visual from a known-rights source, recorded as a substitution rather than as an extraction from the parent document. 3. **Re-author** — ship a project-created figure conveying the same functional information, recorded as project-authored rather than derived from the source. Options 2 and 3 both introduce an artifact that is *associated with* a source but not *derived from* it. That is a lineage relationship the current document model cannot express, since `derived_from` implies transformation of a parent artifact. This issue should define it. ## Suggested scope reduction This issue does not need to model exhaustive per-figure provenance. Three narrower pieces appear sufficient: - **A source-level visual-rights posture** recording whether the documented evidence covers embedded visual material, does not cover it, or leaves it unresolved. This is a single reviewed judgement rather than a per-figure inventory, and it is what §11.4 already asks a reviewer to determine. - **An artifact-level bundle decision** on document records, so a text artifact can be included while a specific visual artifact is excluded — the core limitation §11.5 identifies. - **A substitution and authorship relationship** distinguishing extracted-from-source visuals, substituted known-rights visuals, and project-authored visuals, so a bundled figure never implies a rights claim over the original it replaces. Per-figure rights records remain possible later for sources where the evidence genuinely supports that granularity, but requiring them now is what makes the problem feel unbounded. A reviewed posture plus an artifact-level decision plus honest substitution provenance is enough to be legally conservative without demanding exhaustive research. ## Relationship to Milestone 1 This work is not required to complete the proof loop. #10 selects five documents, and those five can be chosen specifically for unambiguous visual rights — project-authored, public domain by age, or whole-work open licensed. §11.5's conservative interim rule handles everything else correctly in the meantime. Deferring this issue until real corpus material exists is preferable to designing the artifact-level contract against hypotheticals, since the substitution and re-authoring cases are much easier to model correctly once there are concrete figures to reason about.

McJuniorstein referenced this issue

2026-07-29 21:16:32 -04:00

Align metadata schemas and validation with the intake workflow #33