Arkive/arkive
1
0
Fork 0
Code Issues 18 Pull requests 1 Projects Releases Packages Wiki Activity Actions

Define the bundled-core and user-local storage/index boundary #45

New issue
Closed
opened 2026-07-31 01:25:50 -04:00 by JackFrostbyte · 1 comment
JackFrostbyte commented 2026-07-31 01:25:50 -04:00
Owner
Copy link

Goal

Specify a hard architectural boundary between Arkive's redistributable core and
content acquired locally by an end user.

Required model

Arkive has two distribution domains:

  1. Bundled core

    • committed, released, mirrored, and redistributable according to existing policy;
    • reproducible core manifests and indexes.

  2. User-local overlay

    • raw acquisitions, derived documents, chunks, indexes, and provenance that stay
      on the user's machine;
    • excluded from source control, release archives, mirrors, caches, diagnostics,
      and support bundles by default.

The runtime may query both domains, but storage, manifests, index generation, and
citation resolution remain independently identifiable.

Decisions to document

  • Canonical directories and configurable paths.
  • Ownership and permissions.
  • Manifest boundaries and stable identifiers.
  • How the shared processing pipeline receives an explicit distribution context.
  • How deletion, reprocessing, recipe withdrawal, and stale indexes are handled.
  • Backup/export warnings and safe defaults.
  • How container volumes and portable installations preserve the separation.
  • How release tooling fails closed if local content appears in a bundle input path.

Acceptance criteria

  • No command can silently promote a local artifact into the bundled core.
  • Core builds do not enumerate or hash user-local content.
  • User-local data paths are ignored by version control and excluded from releases.
  • Runtime results can always identify their originating domain.
  • Container and non-container deployments follow the same boundary.
  • The design reuses one processing pipeline rather than creating two divergent
    implementations.

Out of scope

  • Acquisition implementation.
  • Index generation.
  • Federated ranking.
## Goal Specify a hard architectural boundary between Arkive's redistributable core and content acquired locally by an end user. ## Required model Arkive has two distribution domains: 1. **Bundled core** - committed, released, mirrored, and redistributable according to existing policy; - reproducible core manifests and indexes. 2. **User-local overlay** - raw acquisitions, derived documents, chunks, indexes, and provenance that stay on the user's machine; - excluded from source control, release archives, mirrors, caches, diagnostics, and support bundles by default. The runtime may query both domains, but storage, manifests, index generation, and citation resolution remain independently identifiable. ## Decisions to document - Canonical directories and configurable paths. - Ownership and permissions. - Manifest boundaries and stable identifiers. - How the shared processing pipeline receives an explicit distribution context. - How deletion, reprocessing, recipe withdrawal, and stale indexes are handled. - Backup/export warnings and safe defaults. - How container volumes and portable installations preserve the separation. - How release tooling fails closed if local content appears in a bundle input path. ## Acceptance criteria - [ ] No command can silently promote a local artifact into the bundled core. - [ ] Core builds do not enumerate or hash user-local content. - [ ] User-local data paths are ignored by version control and excluded from releases. - [ ] Runtime results can always identify their originating domain. - [ ] Container and non-container deployments follow the same boundary. - [ ] The design reuses one processing pipeline rather than creating two divergent implementations. ## Out of scope - Acquisition implementation. - Index generation. - Federated ranking.
JackFrostbyte commented 2026-08-11 02:18:49 -04:00
Author
Owner
Copy link

Completed by PR #59.

Issue #45 established the hard architectural separation between Arkive's redistributable bundled core and the user-local content overlay.

The merged design now defines:

  • canonical and configurable storage locations;
  • user-local ownership and privacy requirements for POSIX and Windows;
  • independent core and local manifests/indexes;
  • domain-qualified identifiers and citations;
  • one shared processing pipeline receiving an explicit distribution context;
  • deletion, reprocessing, recipe-withdrawal, and stale-index behavior;
  • safe export, diagnostic, container, and portable-deployment boundaries;
  • fail-closed release/validation behavior when local content contaminates the core namespace.

The validator enforcement added with this work rejects user_local records under the core store and conservatively rejects symlinked core metadata paths. Review uncovered several edge cases around lexical versus resolved paths, symlinked directories, an early-success validator path, and explicit CLI targets; all were corrected before merge and covered by regression tests.

Final verification was performed after bringing the branch up to current develop:

  • 178 tests passing
  • committed corpus validator PASS
  • Ruff check PASS
  • Ruff format check PASS
  • git diff --check clean

The acceptance criteria for #45 are therefore satisfied.

Acquisition, local processing, local index generation, and the complete end-to-end contamination proof remain intentionally assigned to the downstream overlay issues.

Closing as completed.

Completed by PR #59. Issue #45 established the hard architectural separation between Arkive's redistributable bundled core and the user-local content overlay. The merged design now defines: * canonical and configurable storage locations; * user-local ownership and privacy requirements for POSIX and Windows; * independent core and local manifests/indexes; * domain-qualified identifiers and citations; * one shared processing pipeline receiving an explicit distribution context; * deletion, reprocessing, recipe-withdrawal, and stale-index behavior; * safe export, diagnostic, container, and portable-deployment boundaries; * fail-closed release/validation behavior when local content contaminates the core namespace. The validator enforcement added with this work rejects `user_local` records under the core store and conservatively rejects symlinked core metadata paths. Review uncovered several edge cases around lexical versus resolved paths, symlinked directories, an early-success validator path, and explicit CLI targets; all were corrected before merge and covered by regression tests. Final verification was performed after bringing the branch up to current `develop`: * 178 tests passing * committed corpus validator PASS * Ruff check PASS * Ruff format check PASS * `git diff --check` clean The acceptance criteria for #45 are therefore satisfied. Acquisition, local processing, local index generation, and the complete end-to-end contamination proof remain intentionally assigned to the downstream overlay issues. Closing as completed.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feature/issue-17-reproducibility-tests
develop
No results found.
Labels
Clear labels
  
area/deployment
Portable packaging and constrained-environment validation

  
area/governance
Project rules, policies, contributor workflow

  
area/metadata
Schemas, manifests, validation contracts

  
area/pipeline
Import, extraction, normalization, chunking

  
area/search
Indexing, retrieval, citation resolution

  
type/data
Source selection, metadata, fixtures, and corpus work

  
type/design
Architecture, policy, contracts, and ADR work

  
type/implementation
Production code or tooling

  
type/testing
Reproducibility, integration, evaluation, and benchmark work

No labels
area/deployment
area/governance
area/metadata
area/pipeline
area/search
type/data
type/design
type/implementation
type/testing
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
McJuniorstein
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks
#46 Implement safe user-side acquisition and manual import
Arkive/arkive
#47 Process user-acquired content as local-only derived artifacts
Arkive/arkive
#48 Generate and validate a separate local-content index
Arkive/arkive
#50 Prevent local-content contamination and validate the complete overlay flow
Arkive/arkive
#60 Consolidate the test harness and add coverage measurement
Arkive/arkive
Depends on
#34 Adopt user-side acquisition recipes with a separated local-content overlay
Arkive/arkive
Reference
Arkive/arkive#45
No description provided.