Arkive/arkive
1
0
Fork 0
Code Issues 18 Pull requests 1 Projects Releases Packages Wiki Activity Actions

Define acquisition recipe and local-use rights schemas #44

New issue
Closed
opened 2026-07-31 01:25:50 -04:00 by JackFrostbyte · 1 comment
JackFrostbyte commented 2026-07-31 01:25:50 -04:00
Owner
Copy link

Goal

Define machine-readable contracts for reviewed user-side acquisition without
turning descriptive source provenance into executable code.

Design requirements

Keep origin.acquisition_path descriptive

origin.acquisition_path remains human-readable provenance. It must not contain
shell commands, executable code, credentials, cookies, tokens, or an implicit
permission decision.

Add a distinct acquisition recipe contract

Define a versioned recipe schema that can represent:

  • recipe ID and version;
  • source ID and reviewed target;
  • acquisition mode: manual_import, official_direct_download, or a future
    explicitly reviewed official API adapter;
  • approved hosts and redirect hosts;
  • expected media/content types and size limits;
  • optional stable checksums or publisher signatures;
  • required user acknowledgement;
  • review evidence and jurisdictions;
  • last-reviewed, expiry/recheck, and withdrawn fields;
  • adapter type and adapter-specific parameters from a closed allowlist;
  • post-download integrity and provenance requirements.

Recipes must be declarative data consumed by trusted Arkive adapters, not arbitrary
Python, shell, JavaScript, templates, or commands.

Add a local-use rights profile

Represent independently:

  • redistribution status;
  • local-copy/local-processing status;
  • automation status;
  • terms/contract review status;
  • authentication requirement;
  • technical access-control status;
  • conditions and evidence;
  • reviewed jurisdictions;
  • reviewer and review lifecycle.

Record local provenance

Define a local acquisition record containing the recipe/version, acquisition time,
final URL, checksums, byte size, adapter/tool version, and user acknowledgement.
Do not require storage of secrets, account identifiers, order receipts, or other
sensitive data.

Inheritance

All documents, chunks, and indexes derived from a user-local acquisition must
inherit a machine-enforced local-only distribution scope.

Deliverables

  • Versioned JSON Schemas.
  • Valid and invalid fixtures.
  • Schema documentation and examples.
  • Migration/compatibility notes for the current source schema.
  • ADR note explaining why provenance, permission, recipe, and runtime records are
    separate concepts.

Acceptance criteria

  • Existing bundled-source records remain valid or have a documented migration.
  • Unknown permission states fail closed.
  • Arbitrary code and secrets cannot be represented by a valid recipe.
  • Recipe withdrawal and expiry are representable.
  • Derived local artifacts inherit local_only.
  • Validation tests cover contradictory states and missing evidence.

Out of scope

  • Network access.
  • Download execution.
  • Search implementation.
## Goal Define machine-readable contracts for reviewed user-side acquisition without turning descriptive source provenance into executable code. ## Design requirements ### Keep `origin.acquisition_path` descriptive `origin.acquisition_path` remains human-readable provenance. It must not contain shell commands, executable code, credentials, cookies, tokens, or an implicit permission decision. ### Add a distinct acquisition recipe contract Define a versioned recipe schema that can represent: - recipe ID and version; - source ID and reviewed target; - acquisition mode: `manual_import`, `official_direct_download`, or a future explicitly reviewed official API adapter; - approved hosts and redirect hosts; - expected media/content types and size limits; - optional stable checksums or publisher signatures; - required user acknowledgement; - review evidence and jurisdictions; - last-reviewed, expiry/recheck, and withdrawn fields; - adapter type and adapter-specific parameters from a closed allowlist; - post-download integrity and provenance requirements. Recipes must be declarative data consumed by trusted Arkive adapters, not arbitrary Python, shell, JavaScript, templates, or commands. ### Add a local-use rights profile Represent independently: - redistribution status; - local-copy/local-processing status; - automation status; - terms/contract review status; - authentication requirement; - technical access-control status; - conditions and evidence; - reviewed jurisdictions; - reviewer and review lifecycle. ### Record local provenance Define a local acquisition record containing the recipe/version, acquisition time, final URL, checksums, byte size, adapter/tool version, and user acknowledgement. Do not require storage of secrets, account identifiers, order receipts, or other sensitive data. ### Inheritance All documents, chunks, and indexes derived from a user-local acquisition must inherit a machine-enforced local-only distribution scope. ## Deliverables - Versioned JSON Schemas. - Valid and invalid fixtures. - Schema documentation and examples. - Migration/compatibility notes for the current source schema. - ADR note explaining why provenance, permission, recipe, and runtime records are separate concepts. ## Acceptance criteria - [ ] Existing bundled-source records remain valid or have a documented migration. - [ ] Unknown permission states fail closed. - [ ] Arbitrary code and secrets cannot be represented by a valid recipe. - [ ] Recipe withdrawal and expiry are representable. - [ ] Derived local artifacts inherit `local_only`. - [ ] Validation tests cover contradictory states and missing evidence. ## Out of scope - Network access. - Download execution. - Search implementation.
JackFrostbyte commented 2026-08-09 01:16:28 -04:00
Author
Owner
Copy link

Issue #44 is complete.

PR #56 implemented the machine-readable contracts for reviewed user-side acquisition:

  • versioned local-use rights profiles;
  • versioned declarative acquisition recipes;
  • local acquisition provenance records;
  • machine-enforced local-only distribution inheritance;
  • fail-closed eligibility and cross-record validation;
  • evidence, jurisdiction, withdrawal, and review lifecycle representation;
  • closed adapter and URL contracts that cannot carry arbitrary code or credentials;
  • compatibility with existing bundled-source records;
  • regression coverage for contradictory and incomplete states.

Final review also tightened local provenance so the adapter/tool version is mandatory and a local acquisition must match the exact referenced recipe version, source, acquisition mode, and adapter type.

Final verification passed with 128/128 repository tests plus repository-wide Ruff check/format validation.

Closing as completed by PR #56.

Issue #44 is complete. PR #56 implemented the machine-readable contracts for reviewed user-side acquisition: * versioned local-use rights profiles; * versioned declarative acquisition recipes; * local acquisition provenance records; * machine-enforced local-only distribution inheritance; * fail-closed eligibility and cross-record validation; * evidence, jurisdiction, withdrawal, and review lifecycle representation; * closed adapter and URL contracts that cannot carry arbitrary code or credentials; * compatibility with existing bundled-source records; * regression coverage for contradictory and incomplete states. Final review also tightened local provenance so the adapter/tool version is mandatory and a local acquisition must match the exact referenced recipe version, source, acquisition mode, and adapter type. Final verification passed with 128/128 repository tests plus repository-wide Ruff check/format validation. Closing as completed by PR #56.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feature/issue-17-reproducibility-tests
develop
No results found.
Labels
Clear labels
  
area/deployment
Portable packaging and constrained-environment validation

  
area/governance
Project rules, policies, contributor workflow

  
area/metadata
Schemas, manifests, validation contracts

  
area/pipeline
Import, extraction, normalization, chunking

  
area/search
Indexing, retrieval, citation resolution

  
type/data
Source selection, metadata, fixtures, and corpus work

  
type/design
Architecture, policy, contracts, and ADR work

  
type/implementation
Production code or tooling

  
type/testing
Reproducibility, integration, evaluation, and benchmark work

No labels
area/deployment
area/governance
area/metadata
area/pipeline
area/search
type/data
type/design
type/implementation
type/testing
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
McJuniorstein
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks
#46 Implement safe user-side acquisition and manual import
Arkive/arkive
#47 Process user-acquired content as local-only derived artifacts
Arkive/arkive
Depends on
#34 Adopt user-side acquisition recipes with a separated local-content overlay
Arkive/arkive
#43 Define jurisdiction and target eligibility rules for user-side acquisition
Arkive/arkive
Reference
Arkive/arkive#44
No description provided.