Arkive/arkive
1
0
Fork 0
Code Issues 14 Pull requests Projects Releases Packages Wiki Activity Actions

Implement safe user-side acquisition and manual import #46

New issue
Closed
opened 2026-07-31 01:25:50 -04:00 by JackFrostbyte · 1 comment
JackFrostbyte commented 2026-07-31 01:25:50 -04:00
Owner
Copy link

Goal

Implement the minimum trusted acquisition layer that consumes reviewed declarative
recipes and writes only to the user-local overlay.

Initial supported modes

  1. manual_import

    • the user obtains the file through the publisher's normal workflow;
    • Arkive validates and imports the user-selected file locally.

  2. official_direct_download

    • HTTPS only;
    • reviewed, allowlisted official/publisher/institutional hosts;
    • bounded redirects to approved hosts;
    • no login, paywall, CAPTCHA, session-cookie harvesting, or access-control bypass.

Official authenticated APIs may be added later only through a dedicated reviewed
adapter and credential-handling design.

Required safeguards

  • Closed adapter allowlist; no arbitrary command execution.
  • HTTPS and hostname validation.
  • Redirect limits and redirect-host allowlist.
  • Timeouts, maximum compressed/uncompressed sizes, and content-type checks.
  • Checksum/signature verification when available.
  • Archive path-traversal, symlink, decompression-bomb, and nested-archive defenses.
  • Atomic writes and quarantine before validation.
  • No secrets in recipe files, logs, manifests, exceptions, or telemetry.
  • Dry-run/explain mode showing what Arkive would contact and store.
  • Refusal of expired, withdrawn, unsupported-jurisdiction, or contradictory recipes.
  • Clear user acknowledgement and local-only notice before acquisition.

Deliverables

  • Recipe validator and trusted adapters.
  • CLI for recipe inspection, dry run, direct acquisition, and manual import.
  • Local acquisition provenance record.
  • Unit and integration tests using local fixtures/test servers only.
  • Security documentation and threat model.

Acceptance criteria

  • No valid recipe can execute arbitrary code.
  • Network tests do not depend on live third-party sites.
  • Partial or invalid downloads never become approved local inputs.
  • Credentials and authenticated browser sessions are not scraped or persisted.
  • Withdrawn/expired recipes fail closed.
  • All successful outputs are created only in the user-local store.

Out of scope

  • Circumvention of any technical or contractual access control.
  • Browser automation for paid/login-bound content.
  • Public proxying, mirroring, or hosting of acquired files.

Roadmap clarification: individual incremental acquisition

  • On a repeated request for the same reviewed recipe/source, inspect the existing user-local artifact, pinned recipe snapshot, identifiers, hash, size, type and provenance before deciding whether it can be reused. Do not infer current eligibility solely from an old acquisition record.
  • Return a structured, actionable existing/changed/invalid/conflicting result without silently overwriting valid files, promoting partial files or weakening any new-acquisition policy checks. A newly published version or checksum change requires reviewed recipe evidence, never automatic trust.
  • Include the narrowly scoped validator correction and regression tests for the documented recipe-snapshot.json filename versus the validator's currently recognized recipe.json, without redesigning the prerequisite schemas. Stop for review if a larger prerequisite change is necessary.
  • Keep #46 focused on one source at a time; batch retries are specified in #72, maintainer source checking in #71, and the 10-source end-to-end pilot in #73. No processing or indexing is part of #46.

Additional acceptance criteria

  • Repeated acquisition can securely recognize and validate an already present artifact with its pinned recipe/provenance and report its disposition.
  • Conflicting or invalid existing artifacts fail safely without silent replacement.
  • Recipe-snapshot validation is consistent with the documented storage contract and has focused regression coverage.
## Goal Implement the minimum trusted acquisition layer that consumes reviewed declarative recipes and writes only to the user-local overlay. ## Initial supported modes 1. `manual_import` - the user obtains the file through the publisher's normal workflow; - Arkive validates and imports the user-selected file locally. 2. `official_direct_download` - HTTPS only; - reviewed, allowlisted official/publisher/institutional hosts; - bounded redirects to approved hosts; - no login, paywall, CAPTCHA, session-cookie harvesting, or access-control bypass. Official authenticated APIs may be added later only through a dedicated reviewed adapter and credential-handling design. ## Required safeguards - Closed adapter allowlist; no arbitrary command execution. - HTTPS and hostname validation. - Redirect limits and redirect-host allowlist. - Timeouts, maximum compressed/uncompressed sizes, and content-type checks. - Checksum/signature verification when available. - Archive path-traversal, symlink, decompression-bomb, and nested-archive defenses. - Atomic writes and quarantine before validation. - No secrets in recipe files, logs, manifests, exceptions, or telemetry. - Dry-run/explain mode showing what Arkive would contact and store. - Refusal of expired, withdrawn, unsupported-jurisdiction, or contradictory recipes. - Clear user acknowledgement and local-only notice before acquisition. ## Deliverables - Recipe validator and trusted adapters. - CLI for recipe inspection, dry run, direct acquisition, and manual import. - Local acquisition provenance record. - Unit and integration tests using local fixtures/test servers only. - Security documentation and threat model. ## Acceptance criteria - [ ] No valid recipe can execute arbitrary code. - [ ] Network tests do not depend on live third-party sites. - [ ] Partial or invalid downloads never become approved local inputs. - [ ] Credentials and authenticated browser sessions are not scraped or persisted. - [ ] Withdrawn/expired recipes fail closed. - [ ] All successful outputs are created only in the user-local store. ## Out of scope - Circumvention of any technical or contractual access control. - Browser automation for paid/login-bound content. - Public proxying, mirroring, or hosting of acquired files. --- <!-- arkive-overlay-roadmap-2026-09-27:issue-46:start --> ### Roadmap clarification: individual incremental acquisition - On a repeated request for the same reviewed recipe/source, inspect the existing user-local artifact, pinned recipe snapshot, identifiers, hash, size, type and provenance before deciding whether it can be reused. Do not infer current eligibility solely from an old acquisition record. - Return a structured, actionable existing/changed/invalid/conflicting result without silently overwriting valid files, promoting partial files or weakening any new-acquisition policy checks. A newly published version or checksum change requires reviewed recipe evidence, never automatic trust. - Include the narrowly scoped validator correction and regression tests for the documented `recipe-snapshot.json` filename versus the validator's currently recognized `recipe.json`, without redesigning the prerequisite schemas. Stop for review if a larger prerequisite change is necessary. - Keep #46 focused on **one source at a time**; batch retries are specified in #72, maintainer source checking in #71, and the 10-source end-to-end pilot in #73. No processing or indexing is part of #46. **Additional acceptance criteria** - [ ] Repeated acquisition can securely recognize and validate an already present artifact with its pinned recipe/provenance and report its disposition. - [ ] Conflicting or invalid existing artifacts fail safely without silent replacement. - [ ] Recipe-snapshot validation is consistent with the documented storage contract and has focused regression coverage. <!-- arkive-overlay-roadmap-2026-09-27:issue-46:end -->
JackFrostbyte commented 2026-09-28 02:44:47 -04:00
Author
Owner
Copy link

Completed — Issue #46

Implemented and merged through PR #74.

Arkive now has its initial secure, single-source acquisition foundation, supporting reviewed manual imports and official HTTPS downloads.

The implementation includes eligibility enforcement, trusted recipe loading, secure transport, quarantine, integrity verification, atomic storage, provenance, validated reuse and CLI operations.

Final validation: 616 passing tests, including 288 acquisition tests with external networking blocked. All quality checks passed.

The production acquisition catalogue intentionally remains empty until external sources receive formal approval.

Further development continues through the dedicated maintainer health-monitoring issue (#71), resilient batch acquisition (#72), and final ten-source pilot (#73).

Closing Issue #46 as completed.

### Completed — Issue #46 Implemented and merged through PR #74. Arkive now has its initial secure, single-source acquisition foundation, supporting reviewed manual imports and official HTTPS downloads. The implementation includes eligibility enforcement, trusted recipe loading, secure transport, quarantine, integrity verification, atomic storage, provenance, validated reuse and CLI operations. **Final validation:** 616 passing tests, including 288 acquisition tests with external networking blocked. All quality checks passed. The production acquisition catalogue intentionally remains empty until external sources receive formal approval. Further development continues through the dedicated maintainer health-monitoring issue (#71), resilient batch acquisition (#72), and final ten-source pilot (#73). Closing Issue #46 as completed.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
develop
No results found.
Labels
Clear labels
  
area/deployment
Portable packaging and constrained-environment validation

  
area/governance
Project rules, policies, contributor workflow

  
area/metadata
Schemas, manifests, validation contracts

  
area/pipeline
Import, extraction, normalization, chunking

  
area/search
Indexing, retrieval, citation resolution

  
type/data
Source selection, metadata, fixtures, and corpus work

  
type/design
Architecture, policy, contracts, and ADR work

  
type/implementation
Production code or tooling

  
type/testing
Reproducibility, integration, evaluation, and benchmark work

No labels
area/deployment
area/governance
area/metadata
area/pipeline
area/search
type/data
type/design
type/implementation
type/testing
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
JackFrostbyte
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks
#47 Process user-acquired content as local-only derived artifacts
Arkive/arkive
#50 Prevent local-content contamination and validate the complete overlay flow
Arkive/arkive
#71 Add maintainer-only acquisition catalogue health checks
Arkive/arkive
#72 Implement resilient batch acquisition and incremental retry
Arkive/arkive
#73 Run a ten-source user-overlay end-to-end pilot for v0.1
Arkive/arkive
Depends on
#43 Define jurisdiction and target eligibility rules for user-side acquisition
Arkive/arkive
#44 Define acquisition recipe and local-use rights schemas
Arkive/arkive
#45 Define the bundled-core and user-local storage/index boundary
Arkive/arkive
Reference
Arkive/arkive#46
No description provided.