Define acquisition eligibility policy v0.1 #54

Merged

JackFrostbyte merged 3 commits from feature/issue-43-acquisition-eligibility into develop 2026-08-03 02:05:22 -04:00

Conversation 2 Commits 3 Files changed 3 +306 -1

McJuniorstein commented 2026-08-02 11:26:58 -04:00

Member

Copy link

Summary

• Adds docs/acquisition-eligibility.md, turning ADR-0002's architecture decision into an operational policy for deciding whether a source may be offered as a user-side acquisition target.
• Defines eligibility as a per-target determination valid only for the reviewed target, jurisdictions, review date, and recorded evidence; eight independently evaluated review dimensions each resolving to one of four result values; four availability states as derived outcomes; distinct manual-import and automated-acquisition bars; six absolute hard boundaries; fail-closed handling; named-jurisdiction rules; an evidence checklist; expiry, re-review, and emergency withdrawal; and the questions requiring qualified legal review.
• Adds a pointer from the licence policy's metadata-only section to the review this determination requires.
• Resulting behaviour: the review process is defined and reviewable. Nothing is machine-enforced yet, and the document says so — representation is Issue #44, the storage boundary is Issue #45, implementation is Issue #46.

Deliverables

Required by #43	Where
Policy document under governance/ or docs/	docs/acquisition-eligibility.md
Review checklist and evidence requirements	Section 8
Jurisdiction-handling rules, including fail-closed for unsupported	Section 7
Re-review and emergency-withdrawal procedure	Section 9
Statement of what requires qualified legal review	Section 11

Acceptance criteria

• Eligibility is decided per target, not by source class alone — Section 1 states this as the core principle and gives the reasoning; Section 3 requires per-target evidence for every dimension.
• Copyright, contract/terms, automation, and access controls are separate checks — Sections 3.1, 3.5, 3.4, and 3.6 respectively, with Section 3's preamble stating that a permission in one dimension implies nothing about any other, and hard boundary 4 forbidding either review from substituting for the other.
• Unknown or expired evidence fails closed — Section 6 blocks user_acquirable on any required undetermined result, on an unjustified or misapplied not applicable, and on a mode change since review, and states there is no default-permit path; Section 9.1 reverts expired determinations to catalog_only automatically rather than pending re-review.
• Manual import and automated acquisition have distinct eligibility rules — Section 4 gives each its own required, exempt, and non-blocking dimensions; manual import does not require automation permission, and automated acquisition adds host authority, retrieval-pattern, published-access-rule, and no-credential requirements.
• Withdrawal stops future acquisition without claiming Arkive can delete users' existing local copies — Section 9.3, stated explicitly and in bold, with the note that any documentation implying otherwise is wrong and must be corrected.
• The policy does not claim that a disclaimer makes an otherwise unauthorized action lawful — Section 10 and hard boundary 5: a determination whose validity depends on a disclaimer is invalid, and no determination may cite one as part of its basis.

Verification

• git diff --check
• python pipeline/validate/validate_records.py — PASS, 6 sources, 6 documents, 2 chunks
• python -m unittest discover -s tests -v — 21 tests passing
• python -m ruff check src pipeline tests
• python -m ruff format --check src pipeline tests
• Rendered Markdown reviewed — heading structure inspected, 9 relative links checked, 0 broken
• Complete diff inspected

Scope confirmation

• No unrelated production behaviour changed
• No metadata contract changed — no schema, validator, or record file touched
• No unreviewed source or generated data added
• No fetcher, recipe, or acquisition code added
• No production acquisition target selected or approved
• No legal advice offered to end users

Correction applied after review

The review correctly identified an internal inconsistency. Section 3 permitted only three result values and blocked eligibility on any undetermined result, while Section 4 exempted some dimensions for manual import and Section 8 allowed not applicable. A manual-only target had no consistent value for its exempt dimensions, which would have made the Issue #44 schema ambiguous.

Applied in 14d7de4:

1. not applicable is now an explicit fourth result value, permitted only where Section 4 exempts the dimension for the selected mode and only with a recorded rationale. It is defined as a positive determination and as mode-bound, so a missing answer is undetermined and a change of acquisition mode invalidates every not applicable result.
2. Only an undetermined result in a dimension required for the selected acquisition mode blocks user_acquirable. Section 6 additionally fails closed on an unjustified not applicable, on not applicable recorded for a required dimension, and on a mode change since review.
3. Sections 4.1 and 4.2 now state required, exempt, and non-blocking dimensions per mode, and 3.4 and 3.5 carry explicit manual-import treatment.
4. Section 8's checklist and Section 13's summary are aligned, and a blank answer is read as undetermined.

One related gap the review did not name was fixed alongside it, since it is the same ambiguity: what a not permitted result does. It now makes that mode unavailable rather than the target ineligible, so a target that cannot be automated may still qualify for manual import, and a target with no qualifying mode is catalog_only. Section 3.1 remains the stated exception.

The branch has been updated from develop following PR #53, and the complete validation baseline was rerun clean afterwards.

Points for reviewer attention

Manual import does not require automation permission or, in most cases, service-terms review. This is the most substantive judgement in the document. The reasoning in Section 4.1 is that Arkive retrieves nothing in that mode, so terms governing retrieval do not bind Arkive's conduct — while terms governing the user's use of the file still apply and are surfaced to them. If that reasoning is wrong, Section 4.1 is where it is wrong.

The four states are derived outcomes rather than a new hand-set status field. They are explicitly not to be collapsed with review_status or license.status, consistent with how the intake workflow keeps its dimensions orthogonal. Machine representation is deferred to #44 rather than guessed at here.

Jurisdiction is user-declared configuration, and IP geolocation is explicitly rejected as a legal control (Section 7, rule 4). This constraint was not in the issue text. It is included because the alternative tends to get built by default, and geolocation is unreliable for travellers and VPN users, trivially wrong at borders, and privacy-invasive. If Arkive wants geolocation as a convenience default rather than a control, that should be stated deliberately.

Section 8 states that an automated suggestion identifies a candidate and is never evidence. This closes the gap where a plausible-looking model-generated licence claim could be recorded as though it were an authoritative source. It does not prevent using automated tools to find candidates; it prevents their output from satisfying an evidence requirement.

Section 11 is deliberately a list of open questions rather than answers. The policy is written to be conservative in their absence, and says so. It should not be read as having resolved them.

Linked issue

Closes #43

## Summary - Adds `docs/acquisition-eligibility.md`, turning ADR-0002's architecture decision into an operational policy for deciding whether a source may be offered as a user-side acquisition target. - Defines eligibility as a per-target determination valid only for the reviewed target, jurisdictions, review date, and recorded evidence; eight independently evaluated review dimensions each resolving to one of four result values; four availability states as derived outcomes; distinct manual-import and automated-acquisition bars; six absolute hard boundaries; fail-closed handling; named-jurisdiction rules; an evidence checklist; expiry, re-review, and emergency withdrawal; and the questions requiring qualified legal review. - Adds a pointer from the licence policy's metadata-only section to the review this determination requires. - Resulting behaviour: the review process is defined and reviewable. Nothing is machine-enforced yet, and the document says so — representation is Issue #44, the storage boundary is Issue #45, implementation is Issue #46. ## Deliverables | Required by #43 | Where | |---|---| | Policy document under `governance/` or `docs/` | `docs/acquisition-eligibility.md` | | Review checklist and evidence requirements | Section 8 | | Jurisdiction-handling rules, including fail-closed for unsupported | Section 7 | | Re-review and emergency-withdrawal procedure | Section 9 | | Statement of what requires qualified legal review | Section 11 | ## Acceptance criteria - [x] **Eligibility is decided per target, not by source class alone** — Section 1 states this as the core principle and gives the reasoning; Section 3 requires per-target evidence for every dimension. - [x] **Copyright, contract/terms, automation, and access controls are separate checks** — Sections 3.1, 3.5, 3.4, and 3.6 respectively, with Section 3's preamble stating that a permission in one dimension implies nothing about any other, and hard boundary 4 forbidding either review from substituting for the other. - [x] **Unknown or expired evidence fails closed** — Section 6 blocks `user_acquirable` on any **required** undetermined result, on an unjustified or misapplied `not applicable`, and on a mode change since review, and states there is no default-permit path; Section 9.1 reverts expired determinations to `catalog_only` automatically rather than pending re-review. - [x] **Manual import and automated acquisition have distinct eligibility rules** — Section 4 gives each its own required, exempt, and non-blocking dimensions; manual import does not require automation permission, and automated acquisition adds host authority, retrieval-pattern, published-access-rule, and no-credential requirements. - [x] **Withdrawal stops future acquisition without claiming Arkive can delete users' existing local copies** — Section 9.3, stated explicitly and in bold, with the note that any documentation implying otherwise is wrong and must be corrected. - [x] **The policy does not claim that a disclaimer makes an otherwise unauthorized action lawful** — Section 10 and hard boundary 5: a determination whose validity depends on a disclaimer is invalid, and no determination may cite one as part of its basis. ## Verification - [x] `git diff --check` - [x] `python pipeline/validate/validate_records.py` — PASS, 6 sources, 6 documents, 2 chunks - [x] `python -m unittest discover -s tests -v` — 21 tests passing - [x] `python -m ruff check src pipeline tests` - [x] `python -m ruff format --check src pipeline tests` - [x] Rendered Markdown reviewed — heading structure inspected, 9 relative links checked, 0 broken - [x] Complete diff inspected ## Scope confirmation - [x] No unrelated production behaviour changed - [x] No metadata contract changed — no schema, validator, or record file touched - [x] No unreviewed source or generated data added - [x] No fetcher, recipe, or acquisition code added - [x] No production acquisition target selected or approved - [x] No legal advice offered to end users ## Correction applied after review The review correctly identified an internal inconsistency. Section 3 permitted only three result values and blocked eligibility on *any* undetermined result, while Section 4 exempted some dimensions for manual import and Section 8 allowed `not applicable`. A manual-only target had no consistent value for its exempt dimensions, which would have made the Issue #44 schema ambiguous. Applied in `14d7de4`: 1. `not applicable` is now an explicit fourth result value, permitted only where Section 4 exempts the dimension for the selected mode and only with a recorded rationale. It is defined as a positive determination and as mode-bound, so a missing answer is `undetermined` and a change of acquisition mode invalidates every `not applicable` result. 2. Only an `undetermined` result in a dimension **required for the selected acquisition mode** blocks `user_acquirable`. Section 6 additionally fails closed on an unjustified `not applicable`, on `not applicable` recorded for a required dimension, and on a mode change since review. 3. Sections 4.1 and 4.2 now state required, exempt, and non-blocking dimensions per mode, and 3.4 and 3.5 carry explicit manual-import treatment. 4. Section 8's checklist and Section 13's summary are aligned, and a blank answer is read as `undetermined`. One related gap the review did not name was fixed alongside it, since it is the same ambiguity: what a **`not permitted`** result does. It now makes that *mode* unavailable rather than the target ineligible, so a target that cannot be automated may still qualify for manual import, and a target with no qualifying mode is `catalog_only`. Section 3.1 remains the stated exception. The branch has been updated from `develop` following PR #53, and the complete validation baseline was rerun clean afterwards. ## Points for reviewer attention **Manual import does not require automation permission or, in most cases, service-terms review.** This is the most substantive judgement in the document. The reasoning in Section 4.1 is that Arkive retrieves nothing in that mode, so terms governing retrieval do not bind Arkive's conduct — while terms governing the user's *use* of the file still apply and are surfaced to them. If that reasoning is wrong, Section 4.1 is where it is wrong. **The four states are derived outcomes rather than a new hand-set status field.** They are explicitly not to be collapsed with `review_status` or `license.status`, consistent with how the intake workflow keeps its dimensions orthogonal. Machine representation is deferred to #44 rather than guessed at here. **Jurisdiction is user-declared configuration, and IP geolocation is explicitly rejected as a legal control** (Section 7, rule 4). This constraint was not in the issue text. It is included because the alternative tends to get built by default, and geolocation is unreliable for travellers and VPN users, trivially wrong at borders, and privacy-invasive. If Arkive wants geolocation as a convenience default rather than a control, that should be stated deliberately. **Section 8 states that an automated suggestion identifies a candidate and is never evidence.** This closes the gap where a plausible-looking model-generated licence claim could be recorded as though it were an authoritative source. It does not prevent using automated tools to find candidates; it prevents their output from satisfying an evidence requirement. **Section 11 is deliberately a list of open questions rather than answers.** The policy is written to be conservative in their absence, and says so. It should not be read as having resolved them. ## Linked issue Closes #43

McJuniorstein added 1 commit 2026-08-02 11:26:59 -04:00

Define acquisition eligibility policy v0.1 ... 33e31dc626

Turn ADR-0002's architecture decision into an operational policy for
deciding whether a source may be offered as a user-side acquisition
target, and what evidence that decision requires.

The policy establishes:
- eligibility decided per target, never by source class, and valid only
  for the reviewed target, jurisdictions, date, and evidence
- eight review dimensions evaluated independently, where any
  undetermined result blocks acquisition eligibility
- four availability states as derived outcomes rather than a new
  hand-set status field, with user_acquirable never inherited from a
  failed bundle review
- distinct bars for manual import and automated acquisition, with paid
  and account-bound sources defaulting to manual
- six hard boundaries that are absolute rather than weighed, covering
  circumvention, credentials, unauthorized mirrors, the separation of
  terms from copyright review, disclaimers, and blanket permission
  categories
- fail-closed handling for unknown, incomplete, expired, or
  contradicted evidence, reverting a target to catalog_only
- named-jurisdiction rules where unsupported and unknown jurisdictions
  both fail closed, and IP geolocation is rejected as a legal control
- an evidence checklist stating that an automated suggestion identifies
  a candidate and is never evidence
- expiry with a twelve-month default, early re-review triggers, and
  emergency withdrawal that stops future acquisition while claiming
  nothing about copies already on a user's machine
- eight questions requiring qualified legal review before any
  production recipe catalogue is published, including Crown copyright
  and Copyright Act s.41

Documentation only. No schema, validator, recipe, or acquisition code is
added, and no acquisition target is selected or approved.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

JackFrostbyte commented 2026-08-03 00:47:32 -04:00

Owner

Copy link

Review — one correction required before merge

This is a strong implementation of Issue #43 and the overall policy structure is correct. It covers the required eligibility dimensions, source states, hard boundaries, jurisdiction handling, evidence requirements, re-review, emergency withdrawal, legal-review questions, and the distinction between manual import and automated acquisition.

I found one internal consistency issue that should be corrected before merge:

Section 3 says every review dimension resolves only to permitted, not permitted, or undetermined, and that any undetermined result prevents user_acquirable.

However:

• Section 4 says automation permission and, in many manual-import cases, service-terms review are not required.
• Section 8 permits checklist items to be marked not applicable.
• Section 6 again states that any undetermined dimension blocks acquisition eligibility.

This leaves manual-only eligibility without a consistent result value and would make the downstream Issue #44 schema ambiguous.

Please update the policy so that:

1. not applicable is an explicitly defined result, allowed only where the selected acquisition mode does not require that dimension and accompanied by a recorded rationale; and
2. only an undetermined result in a dimension required for the selected acquisition mode blocks user_acquirable.

The PR description and commit summary should likewise say “any required undetermined result” rather than “any undetermined result.”

The branch is also one commit behind develop following PR #53. Please update it from develop and rerun the complete validation baseline after making the correction.

Once those changes are made and validation remains clean, I expect this PR to be ready for approval.

## Review — one correction required before merge This is a strong implementation of Issue #43 and the overall policy structure is correct. It covers the required eligibility dimensions, source states, hard boundaries, jurisdiction handling, evidence requirements, re-review, emergency withdrawal, legal-review questions, and the distinction between manual import and automated acquisition. I found one internal consistency issue that should be corrected before merge: Section 3 says every review dimension resolves only to `permitted`, `not permitted`, or `undetermined`, and that any `undetermined` result prevents `user_acquirable`. However: * Section 4 says automation permission and, in many manual-import cases, service-terms review are not required. * Section 8 permits checklist items to be marked `not applicable`. * Section 6 again states that any undetermined dimension blocks acquisition eligibility. This leaves manual-only eligibility without a consistent result value and would make the downstream Issue #44 schema ambiguous. Please update the policy so that: 1. `not applicable` is an explicitly defined result, allowed only where the selected acquisition mode does not require that dimension and accompanied by a recorded rationale; and 2. only an `undetermined` result in a dimension required for the selected acquisition mode blocks `user_acquirable`. The PR description and commit summary should likewise say “any required undetermined result” rather than “any undetermined result.” The branch is also one commit behind `develop` following PR #53. Please update it from `develop` and rerun the complete validation baseline after making the correction. Once those changes are made and validation remains clean, I expect this PR to be ready for approval.

McJuniorstein referenced this pull request from a commit 2026-08-03 01:47:32 -04:00

Define not-applicable results and scope fail-closed to required dimensions

McJuniorstein added 2 commits 2026-08-03 01:47:32 -04:00

Merge remote-tracking branch 'origin/develop' into feature/issue-43-acquisition-eligibility 3949866927

Define not-applicable results and scope fail-closed to required dimensions ... 14d7de47fe

Review of PR #54 identified an internal inconsistency: section 3 allowed
only permitted, not permitted, and undetermined, and blocked eligibility
on any undetermined result, while section 4 exempted some dimensions for
manual import and section 8 allowed a checklist item to be marked not
applicable. A manual-only target therefore had no consistent result value
for its exempt dimensions, which would leave the Issue #44 schema
ambiguous.

Corrections:
- define `not applicable` as an explicit fourth result value, permitted
  only where section 4 exempts the dimension for the selected mode and
  only with a recorded rationale
- state that `not applicable` is a positive determination and is
  mode-bound, so a missing answer is `undetermined` and a change of
  acquisition mode invalidates every `not applicable` result
- block `user_acquirable` only on an undetermined result in a dimension
  required for the selected acquisition mode
- fail closed on an unjustified `not applicable`, on `not applicable`
  recorded for a required dimension, and on a mode change since review
- state per-mode required, exempt, and non-blocking dimensions in
  sections 4.1 and 4.2, and give 3.4 and 3.5 explicit manual-import
  treatment
- align the section 8 checklist and the section 13 summary, and read a
  blank answer as undetermined

Also define what a `not permitted` result does, which the same ambiguity
left open: it makes that mode unavailable rather than the target
ineligible, so a target that cannot be automated may still qualify for
manual import, and a target with no qualifying mode is catalog_only.

Merges current develop into the branch following PR #53.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

McJuniorstein commented 2026-08-03 01:48:22 -04:00

Author

Member

Copy link

Correction applied in 14d7de4. The finding was right: a manual-only target had automation permission sitting at undetermined, and section 6's blanket rule would have blocked it, contradicting section 4.1. There was no result value meaning "not required in this mode".

1. not applicable is now an explicit fourth result value. Section 3 carries a table of permitted, not permitted, not applicable, and undetermined, with not applicable allowed only where section 4 exempts the dimension for the selected mode and only with a recorded rationale. Two constraints are stated: it is a positive determination rather than an omission, so a missing answer is undetermined; and it is mode-bound, so changing the acquisition mode invalidates every not applicable result and forces re-evaluation.

2. Only a required undetermined blocks. Section 3 and section 6 both now say "any dimension required for the selected acquisition mode". Section 6 additionally fails closed on an unjustified not applicable, on not applicable recorded for a dimension section 4 requires, and on a mode change since review.

3. Sections 4.1 and 4.2 now list required, exempt, and non-blocking dimensions explicitly, and 3.4 and 3.5 carry their manual-import treatment inline. 3.5 is exempt only where the terms govern retrieval alone; where they also govern the user's possession, processing, or onward use, it stays required, since those obligations survive the download.

4. Section 8's checklist and section 13's summary are aligned, and a blank answer is read as undetermined.

Wording updated in the PR description and in the new commit message to "any required undetermined result".

One related gap fixed alongside it, since it is the same ambiguity and would have reached #44 the same way: the policy never said what a not permitted result does. It now makes that mode unavailable rather than the target ineligible — a target that cannot be automated may still qualify for manual import, and a target with no qualifying mode is catalog_only. Section 3.1 remains the stated exception, where not permitted is the normal case and blocks nothing.

Branch updated from develop following PR #53, merged rather than rebased so the pushed history stays intact. The complete baseline was rerun afterwards and is clean: git diff --check, validator PASS on 6 sources, 21 unit tests passing, ruff check, and ruff format --check. I also swept every occurrence of undetermined in the document to confirm none remains unqualified.

Ready for another look.

Correction applied in `14d7de4`. The finding was right: a manual-only target had automation permission sitting at `undetermined`, and section 6's blanket rule would have blocked it, contradicting section 4.1. There was no result value meaning "not required in this mode". **1. `not applicable` is now an explicit fourth result value.** Section 3 carries a table of `permitted`, `not permitted`, `not applicable`, and `undetermined`, with `not applicable` allowed only where section 4 exempts the dimension for the selected mode and only with a recorded rationale. Two constraints are stated: it is a positive determination rather than an omission, so a missing answer is `undetermined`; and it is mode-bound, so changing the acquisition mode invalidates every `not applicable` result and forces re-evaluation. **2. Only a required `undetermined` blocks.** Section 3 and section 6 both now say "any dimension required for the selected acquisition mode". Section 6 additionally fails closed on an unjustified `not applicable`, on `not applicable` recorded for a dimension section 4 requires, and on a mode change since review. **3. Sections 4.1 and 4.2 now list required, exempt, and non-blocking dimensions explicitly**, and 3.4 and 3.5 carry their manual-import treatment inline. 3.5 is exempt only where the terms govern retrieval alone; where they also govern the user's possession, processing, or onward use, it stays required, since those obligations survive the download. **4. Section 8's checklist and section 13's summary are aligned**, and a blank answer is read as `undetermined`. **Wording updated** in the PR description and in the new commit message to "any required undetermined result". **One related gap fixed alongside it,** since it is the same ambiguity and would have reached #44 the same way: the policy never said what a **`not permitted`** result does. It now makes *that mode* unavailable rather than the target ineligible — a target that cannot be automated may still qualify for manual import, and a target with no qualifying mode is `catalog_only`. Section 3.1 remains the stated exception, where `not permitted` is the normal case and blocks nothing. **Branch updated from `develop`** following PR #53, merged rather than rebased so the pushed history stays intact. The complete baseline was rerun afterwards and is clean: `git diff --check`, validator PASS on 6 sources, 21 unit tests passing, `ruff check`, and `ruff format --check`. I also swept every occurrence of `undetermined` in the document to confirm none remains unqualified. Ready for another look.

JackFrostbyte merged commit c37bbf73f8 into develop 2026-08-03 02:05:22 -04:00

JackFrostbyte referenced this pull request from a commit 2026-08-03 02:05:22 -04:00

Merge pull request 'Define acquisition eligibility policy v0.1' (#54) from feature/issue-43-acquisition-eligibility into develop

JackFrostbyte deleted branch feature/issue-43-acquisition-eligibility 2026-08-03 02:05:22 -04:00

JackFrostbyte referenced this pull request 2026-08-03 02:05:40 -04:00

Define jurisdiction and target eligibility rules for user-side acquisition #43

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

  

area/deployment

Portable packaging and constrained-environment validation

  

area/governance

Project rules, policies, contributor workflow

  

area/metadata

Schemas, manifests, validation contracts

  

area/pipeline

Import, extraction, normalization, chunking

  

area/search

Indexing, retrieval, citation resolution

  

type/data

Source selection, metadata, fixtures, and corpus work

  

type/design

Architecture, policy, contracts, and ADR work

  

type/implementation

Production code or tooling

  

type/testing

Reproducibility, integration, evaluation, and benchmark work

No labels

area/deployment

area/governance

area/metadata

area/pipeline

area/search

type/data

type/design

type/implementation

type/testing

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Arkive/arkive!54

No description provided.