Arkive/arkive
1
0
Fork 0
Code Issues 18 Pull requests 1 Projects Releases Packages Wiki Activity Actions

[Duplicate] Implement safe user-side acquisition and manual import #38

New issue
Closed
opened 2026-07-31 01:21:14 -04:00 by JackFrostbyte · 1 comment
JackFrostbyte commented 2026-07-31 01:21:14 -04:00
Owner
Copy link

Goal

Implement the minimum trusted acquisition layer that consumes reviewed declarative
recipes and writes only to the user-local overlay.

Initial supported modes

  1. manual_import

    • the user obtains the file through the publisher's normal workflow;
    • Arkive validates and imports the user-selected file locally.

  2. official_direct_download

    • HTTPS only;
    • reviewed, allowlisted official/publisher/institutional hosts;
    • bounded redirects to approved hosts;
    • no login, paywall, CAPTCHA, session-cookie harvesting, or access-control bypass.

Official authenticated APIs may be added later only through a dedicated reviewed
adapter and credential-handling design.

Required safeguards

  • Closed adapter allowlist; no arbitrary command execution.
  • HTTPS and hostname validation.
  • Redirect limits and redirect-host allowlist.
  • Timeouts, maximum compressed/uncompressed sizes, and content-type checks.
  • Checksum/signature verification when available.
  • Archive path-traversal, symlink, decompression-bomb, and nested-archive defenses.
  • Atomic writes and quarantine before validation.
  • No secrets in recipe files, logs, manifests, exceptions, or telemetry.
  • Dry-run/explain mode showing what Arkive would contact and store.
  • Refusal of expired, withdrawn, unsupported-jurisdiction, or contradictory recipes.
  • Clear user acknowledgement and local-only notice before acquisition.

Deliverables

  • Recipe validator and trusted adapters.
  • CLI for recipe inspection, dry run, direct acquisition, and manual import.
  • Local acquisition provenance record.
  • Unit and integration tests using local fixtures/test servers only.
  • Security documentation and threat model.

Acceptance criteria

  • No valid recipe can execute arbitrary code.
  • Network tests do not depend on live third-party sites.
  • Partial or invalid downloads never become approved local inputs.
  • Credentials and authenticated browser sessions are not scraped or persisted.
  • Withdrawn/expired recipes fail closed.
  • All successful outputs are created only in the user-local store.

Out of scope

  • Circumvention of any technical or contractual access control.
  • Browser automation for paid/login-bound content.
  • Public proxying, mirroring, or hosting of acquired files.
## Goal Implement the minimum trusted acquisition layer that consumes reviewed declarative recipes and writes only to the user-local overlay. ## Initial supported modes 1. `manual_import` - the user obtains the file through the publisher's normal workflow; - Arkive validates and imports the user-selected file locally. 2. `official_direct_download` - HTTPS only; - reviewed, allowlisted official/publisher/institutional hosts; - bounded redirects to approved hosts; - no login, paywall, CAPTCHA, session-cookie harvesting, or access-control bypass. Official authenticated APIs may be added later only through a dedicated reviewed adapter and credential-handling design. ## Required safeguards - Closed adapter allowlist; no arbitrary command execution. - HTTPS and hostname validation. - Redirect limits and redirect-host allowlist. - Timeouts, maximum compressed/uncompressed sizes, and content-type checks. - Checksum/signature verification when available. - Archive path-traversal, symlink, decompression-bomb, and nested-archive defenses. - Atomic writes and quarantine before validation. - No secrets in recipe files, logs, manifests, exceptions, or telemetry. - Dry-run/explain mode showing what Arkive would contact and store. - Refusal of expired, withdrawn, unsupported-jurisdiction, or contradictory recipes. - Clear user acknowledgement and local-only notice before acquisition. ## Deliverables - Recipe validator and trusted adapters. - CLI for recipe inspection, dry run, direct acquisition, and manual import. - Local acquisition provenance record. - Unit and integration tests using local fixtures/test servers only. - Security documentation and threat model. ## Acceptance criteria - [ ] No valid recipe can execute arbitrary code. - [ ] Network tests do not depend on live third-party sites. - [ ] Partial or invalid downloads never become approved local inputs. - [ ] Credentials and authenticated browser sessions are not scraped or persisted. - [ ] Withdrawn/expired recipes fail closed. - [ ] All successful outputs are created only in the user-local store. ## Out of scope - Circumvention of any technical or contractual access control. - Browser automation for paid/login-bound content. - Public proxying, mirroring, or hosting of acquired files.
JackFrostbyte commented 2026-07-31 01:35:51 -04:00
Author
Owner
Copy link

Closing this issue as a duplicate of #46.

Issues #38 and #46 were created with the same roadmap
content during recovery from an interrupted automation run. The later set
#43–#50 is canonical because Issue #34's finalized roadmap body, roadmap comment,
and dependency graph reference that set.

No implementation work was completed in this duplicate issue.

<!-- arkive-roadmap-duplicate-cleanup-v1 --> Closing this issue as a duplicate of #46. Issues #38 and #46 were created with the same roadmap content during recovery from an interrupted automation run. The later set #43–#50 is canonical because Issue #34's finalized roadmap body, roadmap comment, and dependency graph reference that set. No implementation work was completed in this duplicate issue.
JackFrostbyte changed title from Implement safe user-side acquisition and manual import to [Duplicate] Implement safe user-side acquisition and manual import 2026-07-31 01:35:51 -04:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feature/issue-17-reproducibility-tests
develop
No results found.
Labels
Clear labels
  
area/deployment
Portable packaging and constrained-environment validation

  
area/governance
Project rules, policies, contributor workflow

  
area/metadata
Schemas, manifests, validation contracts

  
area/pipeline
Import, extraction, normalization, chunking

  
area/search
Indexing, retrieval, citation resolution

  
type/data
Source selection, metadata, fixtures, and corpus work

  
type/design
Architecture, policy, contracts, and ADR work

  
type/implementation
Production code or tooling

  
type/testing
Reproducibility, integration, evaluation, and benchmark work

No labels
area/deployment
area/governance
area/metadata
area/pipeline
area/search
type/data
type/design
type/implementation
type/testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Arkive/arkive#38
No description provided.