Arkive/arkive
1
0
Fork 0
Code Issues 18 Pull requests 1 Projects Releases Packages Wiki Activity Actions

Record ADR-0002 for user-side acquisition and reconcile scraping policy #52

Merged
JackFrostbyte merged 1 commit from feature/issue-34-user-acquisition-adr into develop 2026-08-02 00:06:59 -04:00
Conversation 0 Commits 1 Files changed 7 +180 -3
McJuniorstein commented 2026-08-01 20:02:49 -04:00
Member
Copy link

Summary

  • Records governance/decision_records/ADR-0002-user-side-acquisition-overlay.md, adopting reviewed user-side acquisition with a strictly separated user-local content overlay, following ADR-0001's structure and date format.
  • Documents the two distribution domains (bundled_core and user_local) with separate stores, manifests, and indexes; permission reviewed as independent dimensions; manual import and reviewed HTTPS download as the only initial acquisition modes; declarative recipes that cannot execute code; origin.acquisition_path remaining descriptive; and the review, expiry, and withdrawal lifecycle.
  • Records seven rejected alternatives with the reason each fails.
  • Reconciles the existing prohibition on scraping and automated downloading across AGENTS.md, CONTRIBUTING.md, docs/source-intake.md, meta/policies/license_policy.md, and meta/policies/source_acceptance_policy.md, stating the distinction once and consistently.
  • Resulting project behaviour: the direction decided in #34 is now recorded with its reasoning and rejected alternatives, and no document in the repository contradicts it. Bundle eligibility rules are unchanged.

Verification

  • git diff --check
  • python pipeline/validate/validate_records.py — PASS, 6 sources, 6 documents, 2 chunks
  • python -m unittest discover -s tests -v — 21 tests, all passing
  • python -m ruff check src pipeline tests
  • python -m ruff format --check src pipeline tests
  • Rendered Markdown reviewed — heading structure inspected, 17 relative links checked across changed files, 0 broken
  • Complete diff inspected

Scope confirmation

  • No unrelated production behaviour changed
  • No metadata contract changed — no schema, validator, or record file is touched
  • No unreviewed source or generated data was added
  • No implementation code and no third-party source bytes added; this change is documentation only
  • No acquisition target selected or approved

Acceptance criteria

  • ADR records the decision and rejected alternatives — seven alternatives with reasoning
  • Existing source-intake and licensing policies are reconciled — five files updated
  • Bundled-core and user-local stores/indexes are explicitly separate — stated as an architectural boundary that fails closed, with the reasoning for why it cannot be deferred to implementation
  • origin.acquisition_path remains descriptive rather than executable
  • Per-target legal, contractual, automation, access-control, and jurisdiction review requirements are documented
  • Declarative-recipe and manual-import boundaries are documented
  • User notices, expiry, re-review, and withdrawal behaviour are documented
  • The downstream roadmap is linked — #43 through #50 under User-Acquired Content Overlay v0.1
  • No implementation code or third-party source bytes are added by this issue

Notes for review

Rejected alternatives were not enumerated in the issue body, so they were derived from the decision's own constraints. The one most worth a second opinion is the rejection of a single merged store with a per-record redistributable flag. It is rejected because it fails open: any defect in flag handling, export, backup, or diagnostics silently contaminates a shipped bundle, whereas a structural boundary fails closed. If a merged store is later preferred for practical reasons, that reasoning is the thing to argue against.

The reconciliation wording is deliberately identical in substance across all five files. The distinguishing property of permitted acquisition is stated as per-target review, not the presence of a network request, so that no future reader can conclude that automation itself is the prohibited thing. If that phrasing should differ anywhere, it is better changed in all five places at once.

license_policy.md gains a note at the metadata-only section stating that acquisition eligibility is a separate determination that grants Arkive nothing, and that the record in this repository stays metadata-only. This seemed the most likely place for a future reader to mistakenly infer that overlay eligibility relaxes the repository restriction.

Jurisdictional evaluation is recorded as a prerequisite to implementation, not as something this record resolves. The ADR states plainly that it is not legal advice and that its conclusions need validating for the jurisdictions Arkive operates and distributes in.

Linked issue

Closes #34

## Summary - Records `governance/decision_records/ADR-0002-user-side-acquisition-overlay.md`, adopting reviewed user-side acquisition with a strictly separated user-local content overlay, following ADR-0001's structure and date format. - Documents the two distribution domains (`bundled_core` and `user_local`) with separate stores, manifests, and indexes; permission reviewed as independent dimensions; manual import and reviewed HTTPS download as the only initial acquisition modes; declarative recipes that cannot execute code; `origin.acquisition_path` remaining descriptive; and the review, expiry, and withdrawal lifecycle. - Records seven rejected alternatives with the reason each fails. - Reconciles the existing prohibition on scraping and automated downloading across `AGENTS.md`, `CONTRIBUTING.md`, `docs/source-intake.md`, `meta/policies/license_policy.md`, and `meta/policies/source_acceptance_policy.md`, stating the distinction once and consistently. - Resulting project behaviour: the direction decided in #34 is now recorded with its reasoning and rejected alternatives, and no document in the repository contradicts it. Bundle eligibility rules are unchanged. ## Verification - [x] `git diff --check` - [x] `python pipeline/validate/validate_records.py` — PASS, 6 sources, 6 documents, 2 chunks - [x] `python -m unittest discover -s tests -v` — 21 tests, all passing - [x] `python -m ruff check src pipeline tests` - [x] `python -m ruff format --check src pipeline tests` - [x] Rendered Markdown reviewed — heading structure inspected, 17 relative links checked across changed files, 0 broken - [x] Complete diff inspected ## Scope confirmation - [x] No unrelated production behaviour changed - [x] No metadata contract changed — no schema, validator, or record file is touched - [x] No unreviewed source or generated data was added - [x] No implementation code and no third-party source bytes added; this change is documentation only - [x] No acquisition target selected or approved ## Acceptance criteria - [x] ADR records the decision and rejected alternatives — seven alternatives with reasoning - [x] Existing source-intake and licensing policies are reconciled — five files updated - [x] Bundled-core and user-local stores/indexes are explicitly separate — stated as an architectural boundary that fails closed, with the reasoning for why it cannot be deferred to implementation - [x] `origin.acquisition_path` remains descriptive rather than executable - [x] Per-target legal, contractual, automation, access-control, and jurisdiction review requirements are documented - [x] Declarative-recipe and manual-import boundaries are documented - [x] User notices, expiry, re-review, and withdrawal behaviour are documented - [x] The downstream roadmap is linked — #43 through #50 under **User-Acquired Content Overlay v0.1** - [x] No implementation code or third-party source bytes are added by this issue ## Notes for review **Rejected alternatives were not enumerated in the issue body**, so they were derived from the decision's own constraints. The one most worth a second opinion is the rejection of a *single merged store with a per-record `redistributable` flag*. It is rejected because it fails open: any defect in flag handling, export, backup, or diagnostics silently contaminates a shipped bundle, whereas a structural boundary fails closed. If a merged store is later preferred for practical reasons, that reasoning is the thing to argue against. **The reconciliation wording is deliberately identical in substance across all five files.** The distinguishing property of permitted acquisition is stated as *per-target review*, not the presence of a network request, so that no future reader can conclude that automation itself is the prohibited thing. If that phrasing should differ anywhere, it is better changed in all five places at once. **`license_policy.md` gains a note at the metadata-only section** stating that acquisition eligibility is a separate determination that grants Arkive nothing, and that the record in this repository stays metadata-only. This seemed the most likely place for a future reader to mistakenly infer that overlay eligibility relaxes the repository restriction. **Jurisdictional evaluation is recorded as a prerequisite to implementation**, not as something this record resolves. The ADR states plainly that it is not legal advice and that its conclusions need validating for the jurisdictions Arkive operates and distributes in. ## Linked issue Closes #34 
Adopt reviewed user-side acquisition recipes with a strictly separated
user-local content overlay, so that sources Arkive may not redistribute
can still reach an offline user who may lawfully obtain them.

The record documents:
- two distribution domains, bundled_core and user_local, with separate
  stores, manifests, and indexes that fail closed against contamination
- permission reviewed as independent dimensions, so that lawful
  acquisition, local processing, automation, contract terms, access
  controls, evidence, and jurisdiction are never conflated
- manual import and reviewed HTTPS download as the only initial modes,
  with no circumvention of DRM, paywalls, authentication, or other
  access controls under any circumstances
- recipes as declarative data interpreted by trusted adapters, never
  executable code
- origin.acquisition_path remaining descriptive rather than executable
- review evidence, expiry, re-review, and withdrawal, with no claim to
  erase copies already on a user's machine
- seven rejected alternatives and why each fails

Reconcile the existing prohibition on scraping and automated downloading
across AGENTS.md, CONTRIBUTING.md, docs/source-intake.md, and both
policies. The prohibition targets indiscriminate collection without
provenance or review; the distinguishing property of permitted
acquisition is per-target review, not the presence of a network request.
Nothing here lowers the bar for Arkive's redistributable bundle.

Documentation only. No implementation code and no third-party source
bytes are added.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
JackFrostbyte deleted branch feature/issue-34-user-acquisition-adr 2026-08-02 00:07:00 -04:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
  
area/deployment
Portable packaging and constrained-environment validation

  
area/governance
Project rules, policies, contributor workflow

  
area/metadata
Schemas, manifests, validation contracts

  
area/pipeline
Import, extraction, normalization, chunking

  
area/search
Indexing, retrieval, citation resolution

  
type/data
Source selection, metadata, fixtures, and corpus work

  
type/design
Architecture, policy, contracts, and ADR work

  
type/implementation
Production code or tooling

  
type/testing
Reproducibility, integration, evaluation, and benchmark work

No labels
area/deployment
area/governance
area/metadata
area/pipeline
area/search
type/data
type/design
type/implementation
type/testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Arkive/arkive!52
No description provided.