Arkive/arkive
1
0
Fork 0
Code Issues 18 Pull requests 1 Projects Releases Packages Wiki Activity Actions

Adopt user-side acquisition recipes with a separated local-content overlay #34

New issue
Closed
opened 2026-07-30 18:46:31 -04:00 by McJuniorstein · 3 comments
McJuniorstein commented 2026-07-30 18:46:31 -04:00
Member
Copy link

Decision

Arkive will support reviewed user-side acquisition recipes and manual import
for selected sources that cannot be redistributed in Arkive's shipped bundle but
may be lawfully acquired and processed locally by an end user under a documented,
target-specific and jurisdiction-aware basis.

This is an additional local overlay. It does not weaken Arkive's bundle eligibility
rules and does not convert a non-redistributable source into redistributable content.

Binding architecture

  1. Two distribution domains

    • bundled_core: Arkive-created or third-party content that passes existing
      redistribution and bundle review.
    • user_local: user-acquired source bytes, derived documents, chunks, indexes,
      and provenance that remain on that user's system.

  2. Separate stores, manifests, and indexes

    • Runtime search may federate results.
    • Build, release, mirroring, diagnostics, and export paths must preserve the
      boundary and fail closed against accidental contamination.

  3. Declarative reviewed recipes

    • Recipes are versioned data interpreted by trusted Arkive adapters.
    • Recipes are not arbitrary scripts and cannot execute shell, Python, JavaScript,
      or other user-supplied code.

  4. Permission is multidimensional

    • Redistribution, lawful acquisition, local copying/processing, automation,
      terms/contract, authentication, technical access controls, evidence, and
      jurisdiction are reviewed separately.
    • “Personal use,” “purchased,” “freely accessible,” and “non-bundleable” are not
      blanket permission categories.

  5. Allowed initial acquisition modes

    • Manual import after the user obtains a file through the publisher's normal
      workflow.
    • Reviewed HTTPS direct download from an official, publisher, or institutional
      host when automation is acceptable.
    • Authenticated APIs require a later dedicated design.
    • Arkive will not bypass DRM, paywalls, CAPTCHAs, authentication, rate limits, or
      other technical/access controls, and will not direct users to unauthorized copies.

  6. Provenance and recipes remain separate

    • Existing origin.acquisition_path stays descriptive.
    • Executable behavior, review lifecycle, and local acquisition records use
      separate schemas.

  7. Review, expiry, and withdrawal

    • Every published recipe has evidence, reviewed jurisdictions, reviewer, review
      date, recheck/expiry, and withdrawal state.
    • Withdrawal stops future acquisition. Arkive does not claim the ability or right
      to erase copies already created on a user's machine.

  8. User disclosure

    • Arkive explains that local material is not part of the redistributable bundle,
      may be subject to copyright and contract restrictions, may not be safe to share
      or include in backups/exports, and remains the user's responsibility.
    • Disclaimers provide transparency; they do not create legal permission.

Scope of this issue

Produce the architecture decision record and reconcile existing governance language
that currently excludes automated downloading/scraping. The updated policy should
distinguish prohibited general scraping from narrowly reviewed acquisition recipes.

Out of scope

  • Downloader or importer implementation.
  • Processing and indexing code.
  • Selecting or approving production acquisition targets.
  • Adding non-redistributable source bytes to the repository or releases.
  • Hosting, proxying, or mirroring acquired content.
  • Any access-control or DRM circumvention.

Roadmap

This decision creates the following issues in User-Acquired Content Overlay v0.1:

  • #43 — Define jurisdiction and target eligibility rules for user-side acquisition
  • #44 — Define acquisition recipe and local-use rights schemas
  • #45 — Define the bundled-core and user-local storage/index boundary
  • #46 — Implement safe user-side acquisition and manual import
  • #47 — Process user-acquired content as local-only derived artifacts
  • #48 — Generate and validate a separate local-content index
  • #49 — Federate search and citations across core and local stores
  • #50 — Prevent local-content contamination and validate the complete overlay flow

The new milestone may reuse completed work from the current processing and search
milestones. It does not expand or modify Issue #10, and it does not block completion
of the existing five-document pilot corpus.

Dependencies

None. This policy/architecture decision can be completed now.

Acceptance criteria

  • ADR records the decision and rejected alternatives.
  • Existing source-intake and licensing policies are reconciled.
  • Bundled-core and user-local stores/indexes are explicitly separate.
  • origin.acquisition_path remains descriptive rather than executable.
  • Per-target legal, contractual, automation, access-control, and jurisdiction
    review requirements are documented.
  • Declarative-recipe and manual-import boundaries are documented.
  • User notices, expiry, re-review, and withdrawal behavior are documented.
  • The downstream roadmap is linked.
  • No implementation code or third-party source bytes are added by this issue.
## Decision Arkive will support **reviewed user-side acquisition recipes and manual import** for selected sources that cannot be redistributed in Arkive's shipped bundle but may be lawfully acquired and processed locally by an end user under a documented, target-specific and jurisdiction-aware basis. This is an additional local overlay. It does not weaken Arkive's bundle eligibility rules and does not convert a non-redistributable source into redistributable content. ## Binding architecture 1. **Two distribution domains** - `bundled_core`: Arkive-created or third-party content that passes existing redistribution and bundle review. - `user_local`: user-acquired source bytes, derived documents, chunks, indexes, and provenance that remain on that user's system. 2. **Separate stores, manifests, and indexes** - Runtime search may federate results. - Build, release, mirroring, diagnostics, and export paths must preserve the boundary and fail closed against accidental contamination. 3. **Declarative reviewed recipes** - Recipes are versioned data interpreted by trusted Arkive adapters. - Recipes are not arbitrary scripts and cannot execute shell, Python, JavaScript, or other user-supplied code. 4. **Permission is multidimensional** - Redistribution, lawful acquisition, local copying/processing, automation, terms/contract, authentication, technical access controls, evidence, and jurisdiction are reviewed separately. - “Personal use,” “purchased,” “freely accessible,” and “non-bundleable” are not blanket permission categories. 5. **Allowed initial acquisition modes** - Manual import after the user obtains a file through the publisher's normal workflow. - Reviewed HTTPS direct download from an official, publisher, or institutional host when automation is acceptable. - Authenticated APIs require a later dedicated design. - Arkive will not bypass DRM, paywalls, CAPTCHAs, authentication, rate limits, or other technical/access controls, and will not direct users to unauthorized copies. 6. **Provenance and recipes remain separate** - Existing `origin.acquisition_path` stays descriptive. - Executable behavior, review lifecycle, and local acquisition records use separate schemas. 7. **Review, expiry, and withdrawal** - Every published recipe has evidence, reviewed jurisdictions, reviewer, review date, recheck/expiry, and withdrawal state. - Withdrawal stops future acquisition. Arkive does not claim the ability or right to erase copies already created on a user's machine. 8. **User disclosure** - Arkive explains that local material is not part of the redistributable bundle, may be subject to copyright and contract restrictions, may not be safe to share or include in backups/exports, and remains the user's responsibility. - Disclaimers provide transparency; they do not create legal permission. ## Scope of this issue Produce the architecture decision record and reconcile existing governance language that currently excludes automated downloading/scraping. The updated policy should distinguish prohibited general scraping from narrowly reviewed acquisition recipes. ## Out of scope - Downloader or importer implementation. - Processing and indexing code. - Selecting or approving production acquisition targets. - Adding non-redistributable source bytes to the repository or releases. - Hosting, proxying, or mirroring acquired content. - Any access-control or DRM circumvention. ## Roadmap This decision creates the following issues in **User-Acquired Content Overlay v0.1**: - #43 — Define jurisdiction and target eligibility rules for user-side acquisition - #44 — Define acquisition recipe and local-use rights schemas - #45 — Define the bundled-core and user-local storage/index boundary - #46 — Implement safe user-side acquisition and manual import - #47 — Process user-acquired content as local-only derived artifacts - #48 — Generate and validate a separate local-content index - #49 — Federate search and citations across core and local stores - #50 — Prevent local-content contamination and validate the complete overlay flow The new milestone may reuse completed work from the current processing and search milestones. It does not expand or modify Issue #10, and it does not block completion of the existing five-document pilot corpus. ## Dependencies None. This policy/architecture decision can be completed now. ## Acceptance criteria - [ ] ADR records the decision and rejected alternatives. - [ ] Existing source-intake and licensing policies are reconciled. - [ ] Bundled-core and user-local stores/indexes are explicitly separate. - [ ] `origin.acquisition_path` remains descriptive rather than executable. - [ ] Per-target legal, contractual, automation, access-control, and jurisdiction review requirements are documented. - [ ] Declarative-recipe and manual-import boundaries are documented. - [ ] User notices, expiry, re-review, and withdrawal behavior are documented. - [ ] The downstream roadmap is linked. - [ ] No implementation code or third-party source bytes are added by this issue.
JackFrostbyte changed title from Decide whether Arkive ships user-side acquisition recipes for non-redistributable sources to Adopt user-side acquisition recipes with a separated local-content overlay 2026-07-31 01:21:20 -04:00
JackFrostbyte commented 2026-07-31 01:25:56 -04:00
Owner
Copy link

The user-acquired-content roadmap has been created under User-Acquired Content Overlay v0.1:

  • #43 — Define jurisdiction and target eligibility rules for user-side acquisition
  • #44 — Define acquisition recipe and local-use rights schemas
  • #45 — Define the bundled-core and user-local storage/index boundary
  • #46 — Implement safe user-side acquisition and manual import
  • #47 — Process user-acquired content as local-only derived artifacts
  • #48 — Generate and validate a separate local-content index
  • #49 — Federate search and citations across core and local stores
  • #50 — Prevent local-content contamination and validate the complete overlay flow

Dependency direction is intentionally downstream:

  • Issue #34 is the immediate architecture/governance decision.
  • The existing processing and search milestones continue unchanged.
  • The new implementation issues depend on the relevant existing pipeline/search
    outputs instead of blocking or expanding them.
  • Issue #10 is untouched.
<!-- arkive-user-acquired-overlay-roadmap-v1 --> The user-acquired-content roadmap has been created under **User-Acquired Content Overlay v0.1**: - #43 — Define jurisdiction and target eligibility rules for user-side acquisition - #44 — Define acquisition recipe and local-use rights schemas - #45 — Define the bundled-core and user-local storage/index boundary - #46 — Implement safe user-side acquisition and manual import - #47 — Process user-acquired content as local-only derived artifacts - #48 — Generate and validate a separate local-content index - #49 — Federate search and citations across core and local stores - #50 — Prevent local-content contamination and validate the complete overlay flow Dependency direction is intentionally downstream: - Issue #34 is the immediate architecture/governance decision. - The existing processing and search milestones continue unchanged. - The new implementation issues depend on the relevant existing pipeline/search outputs instead of blocking or expanding them. - Issue #10 is untouched.
JackFrostbyte commented 2026-07-31 02:02:45 -04:00
Owner
Copy link

Implementation handoff

The Forgejo milestone, downstream issues, and dependency framework are now complete and have been audited successfully. Issue #34 still requires the architecture decision to be committed to the repository before it can be closed.

Expected repository changes

Please create a documentation-only branch, suggested name:

feature/issue-34-user-acquired-content-overlay

The expected files to review and update are:

  • governance/decision_records/ADR-0002-user-acquired-content-overlay.md
  • docs/source-intake.md
  • meta/policies/source_acceptance_policy.md
  • meta/policies/license_policy.md
  • docs/architecture/overview.md
  • CHANGELOG.md

ADR requirements

The ADR should follow the general structure of ADR-0001 and record:

  • status and date;
  • context and problem;
  • adopted decision;
  • reasoning;
  • consequences and tradeoffs;
  • rejected alternatives;
  • the distinction between bundled_core and user_local;
  • separate stores, manifests, indexes, and release boundaries;
  • declarative reviewed recipes rather than arbitrary executable scripts;
  • manual import and reviewed official direct-download boundaries;
  • separate evaluation of redistribution, lawful acquisition, local processing, automation, terms or contract, authentication, technical access controls, evidence, and jurisdiction;
  • expiry, re-review, withdrawal, and user-disclosure requirements.

Rejected alternatives should include at least:

  • retaining a redistributable-only architecture with no local overlay;
  • treating all non-bundleable sources as automatically eligible for personal use;
  • distributing arbitrary acquisition scripts;
  • mixing local-only files or indexes into the bundled core;
  • duplicating the processing pipeline for local content;
  • implementing download or schema behavior directly in this issue.

Policy reconciliation

The existing policies should clearly state that:

  • general-purpose crawling and scraping remain outside the approved model;
  • narrowly reviewed user-side acquisition recipes may be supported later;
  • online availability, purchase, or personal use does not automatically provide permission;
  • Arkive must not bypass DRM, paywalls, CAPTCHAs, authentication, rate limits, or other access controls;
  • user-acquired source bytes and every derived artifact remain local-only;
  • local content must not enter Git, releases, mirrors, portable core bundles, diagnostics, or support archives;
  • origin.acquisition_path remains descriptive and non-executable;
  • detailed eligibility rules are deferred to #43;
  • recipe and local-use rights schemas are deferred to #44;
  • storage/index implementation is deferred to #45 and later issues.

Scope guard

This issue must remain documentation-only.

Do not add:

  • acquisition or importer code;
  • recipe schemas;
  • processing or indexing implementation;
  • production acquisition targets;
  • third-party source files;
  • credentials or authenticated download handling;
  • DRM or access-control circumvention.

Validation

Run the existing project checks even though the changes are documentation-only:

python -m unittest discover -s tests -v

python -m ruff check .

python -m ruff format --check .

git diff --check

The pull request should target develop, reference Issue #34, summarize how each acceptance criterion is satisfied, and explicitly confirm that no implementation code or third-party source content was added.

Issue #34 should be closed only after the documentation PR is reviewed and merged and all acceptance criteria are checked.

## Implementation handoff The Forgejo milestone, downstream issues, and dependency framework are now complete and have been audited successfully. Issue #34 still requires the architecture decision to be committed to the repository before it can be closed. ### Expected repository changes Please create a documentation-only branch, suggested name: `feature/issue-34-user-acquired-content-overlay` The expected files to review and update are: * `governance/decision_records/ADR-0002-user-acquired-content-overlay.md` * `docs/source-intake.md` * `meta/policies/source_acceptance_policy.md` * `meta/policies/license_policy.md` * `docs/architecture/overview.md` * `CHANGELOG.md` ### ADR requirements The ADR should follow the general structure of `ADR-0001` and record: * status and date; * context and problem; * adopted decision; * reasoning; * consequences and tradeoffs; * rejected alternatives; * the distinction between `bundled_core` and `user_local`; * separate stores, manifests, indexes, and release boundaries; * declarative reviewed recipes rather than arbitrary executable scripts; * manual import and reviewed official direct-download boundaries; * separate evaluation of redistribution, lawful acquisition, local processing, automation, terms or contract, authentication, technical access controls, evidence, and jurisdiction; * expiry, re-review, withdrawal, and user-disclosure requirements. Rejected alternatives should include at least: * retaining a redistributable-only architecture with no local overlay; * treating all non-bundleable sources as automatically eligible for personal use; * distributing arbitrary acquisition scripts; * mixing local-only files or indexes into the bundled core; * duplicating the processing pipeline for local content; * implementing download or schema behavior directly in this issue. ### Policy reconciliation The existing policies should clearly state that: * general-purpose crawling and scraping remain outside the approved model; * narrowly reviewed user-side acquisition recipes may be supported later; * online availability, purchase, or personal use does not automatically provide permission; * Arkive must not bypass DRM, paywalls, CAPTCHAs, authentication, rate limits, or other access controls; * user-acquired source bytes and every derived artifact remain local-only; * local content must not enter Git, releases, mirrors, portable core bundles, diagnostics, or support archives; * `origin.acquisition_path` remains descriptive and non-executable; * detailed eligibility rules are deferred to #43; * recipe and local-use rights schemas are deferred to #44; * storage/index implementation is deferred to #45 and later issues. ### Scope guard This issue must remain documentation-only. Do not add: * acquisition or importer code; * recipe schemas; * processing or indexing implementation; * production acquisition targets; * third-party source files; * credentials or authenticated download handling; * DRM or access-control circumvention. ### Validation Run the existing project checks even though the changes are documentation-only: `python -m unittest discover -s tests -v` `python -m ruff check .` `python -m ruff format --check .` `git diff --check` The pull request should target `develop`, reference Issue #34, summarize how each acceptance criterion is satisfied, and explicitly confirm that no implementation code or third-party source content was added. Issue #34 should be closed only after the documentation PR is reviewed and merged and all acceptance criteria are checked.
JackFrostbyte commented 2026-08-02 00:08:40 -04:00
Owner
Copy link

Issue #34 has been completed by merged PR #52.

Implemented:

  • recorded ADR-0002 adopting reviewed user-side acquisition with a strictly separated user-local content overlay;
  • established separate bundled_core and user_local stores, manifests, and indexes that fail closed against accidental bundle contamination;
  • documented the independent legal, contractual, automation, access-control, evidence, and jurisdiction review dimensions;
  • limited the initial acquisition modes to manual import and reviewed HTTPS direct download;
  • required declarative, non-executable acquisition recipes;
  • preserved origin.acquisition_path as descriptive provenance;
  • documented review evidence, expiry, re-review, withdrawal, and user-disclosure requirements;
  • recorded seven rejected alternatives and their reasoning;
  • reconciled the relevant source-intake, contribution, licensing, source-acceptance, and agent guidance;
  • recorded the downstream roadmap through Issues #43–#50.

Verified:

  • the metadata validator passes;
  • all 21 unit tests pass;
  • Ruff checks pass;
  • Markdown structure and relative links were reviewed;
  • the complete diff was inspected.

No downloader, importer, processing, indexing, schema, production acquisition-target, or third-party source-content work was included.

Follow-up implementation can proceed through Issues #43–#50 according to their established dependencies.

Issue #34 has been completed by merged PR #52. Implemented: * recorded ADR-0002 adopting reviewed user-side acquisition with a strictly separated user-local content overlay; * established separate `bundled_core` and `user_local` stores, manifests, and indexes that fail closed against accidental bundle contamination; * documented the independent legal, contractual, automation, access-control, evidence, and jurisdiction review dimensions; * limited the initial acquisition modes to manual import and reviewed HTTPS direct download; * required declarative, non-executable acquisition recipes; * preserved `origin.acquisition_path` as descriptive provenance; * documented review evidence, expiry, re-review, withdrawal, and user-disclosure requirements; * recorded seven rejected alternatives and their reasoning; * reconciled the relevant source-intake, contribution, licensing, source-acceptance, and agent guidance; * recorded the downstream roadmap through Issues #43–#50. Verified: * the metadata validator passes; * all 21 unit tests pass; * Ruff checks pass; * Markdown structure and relative links were reviewed; * the complete diff was inspected. No downloader, importer, processing, indexing, schema, production acquisition-target, or third-party source-content work was included. Follow-up implementation can proceed through Issues #43–#50 according to their established dependencies.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feature/issue-17-reproducibility-tests
develop
No results found.
Labels
Clear labels
  
area/deployment
Portable packaging and constrained-environment validation

  
area/governance
Project rules, policies, contributor workflow

  
area/metadata
Schemas, manifests, validation contracts

  
area/pipeline
Import, extraction, normalization, chunking

  
area/search
Indexing, retrieval, citation resolution

  
type/data
Source selection, metadata, fixtures, and corpus work

  
type/design
Architecture, policy, contracts, and ADR work

  
type/implementation
Production code or tooling

  
type/testing
Reproducibility, integration, evaluation, and benchmark work

No labels
area/deployment
area/governance
area/metadata
area/pipeline
area/search
type/data
type/design
type/implementation
type/testing
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
McJuniorstein
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks
#43 Define jurisdiction and target eligibility rules for user-side acquisition
Arkive/arkive
#44 Define acquisition recipe and local-use rights schemas
Arkive/arkive
#45 Define the bundled-core and user-local storage/index boundary
Arkive/arkive
Reference
Arkive/arkive#34
No description provided.