Research source acquisition

Research Source Acquisition

Acquire exactly one approved source as untrusted bytes. Keep it in quarantine until separate security, quality, and scholarly gates promote it. Version 1.1.2 preserves verification compatibility with 1.1.0 and 1.1.1 acquisition manifests while emitting 1.1.2 for every new manifest and verification receipt.

Read the contract

Read references/rights-record-contract.md before creating a rights record or invoking the downloader. Use references/rights-record-v1.schema.json for authoring assistance; rely on the CLI for authoritative evidence checks.

Preserve these invariants

• Require a rights record with D1 or higher, download: true, an asset_url exactly equal to --url, and approved_redirect_urls as an explicit exact URL list.
• Bind every asset record to one absolute candidate-registry path, registry SHA-256, candidate ID, item page, structured downloadable URL, exact three rights layers, and a terms run for that same candidate.
• Permit only HTTPS and exact repeated --allow-host values. Never add wildcards or broaden the list merely because a redirect was blocked.
• Reject URL userinfo, credential-like query parameters, local hostnames, and every DNS result that is not globally routable.
• Default to zero redirects. When a reviewed redirect is necessary, require its exact URL in the rights record, re-resolve it, and pin the connection to a validated IP while retaining the hostname for TLS certificate validation.
• Disable environment proxies by using the direct HTTPS transport.
• Require determinate body framing on the final 200. A body delimited only by connection close cannot be distinguished from a body the peer cut short, so a response with neither Content-Length nor a chunked transfer coding is blocked (indeterminate-body-framing, exit 3) rather than published as complete. Declaring both is ambiguous-body-framing (exit 3); any transfer coding other than identity or chunked is unsupported-transfer-coding (exit 5). This is an acquisition-time gate only: manifests written before it existed still verify byte-identically.
• Stream into a newly created quarantine run, enforce header and body limits, hash while writing, and call fsync.
• Never execute, import, decompress, render, OCR, or otherwise interpret the payload as instructions. Metadata JSON is parsed only after a fixed-budget, no-AST syntax scan and bounded numeric conversion. Downloaded JSON format recognition uses a separate fixed-budget syntax scanner and never calls json.loads or constructs a complete payload value tree. Both paths refuse duplicate keys, non-finite values, BOMs, invalid UTF-8, excessive nesting, oversized numbers, and their respective resource limits.
• Compare rights-declared MIME, HTTP Content-Type, and detected magic/actual format. Keep mismatches as read-only .blocked evidence.
• Never reuse a run ID or overwrite a rights snapshot, payload, manifest, or sidecar.
• Require a canonical non-symlink quarantine root owned by the effective user and not writable by group or others. Create the run as exact mode 0700 under the same owner, then pin it and every ancestor by directory descriptor for the full transaction. Perform all run-local creation, append, read, stat, link, unlink, and publication descriptor-relatively. Refuse ancestry or run-entry identity drift, require single-link regular files and no-replace publication, and retain a two-event append-only run journal.
• Treat status: quarantine as acquisition success, not security clearance or permission to publish.

Bootstrap rights evidence

When no local terms snapshot exists, use the contract's non-circular evidence-bootstrap flow:

1. Pin the reviewed candidate registry by absolute path and actual local SHA-256.
2. Approve one HTML or JSON URL occurring in a structured rights field for the selected candidate.
3. Run preflight, acquire, and verify for the evidence URL.
4. Create the final asset rights record from that verified run and payload hash, repeating the same registry path/hash and candidate ID.

Do not hand-enter an unverifiable terms digest. The final asset record must point to a successful local evidence run; the CLI re-verifies it before any asset request.

Preflight

Run preflight before every acquisition:

python3 scripts/source_acquisition.py preflight \
  --url "https://approved.example/path/asset.jpg" \
  --rights-record /absolute/path/rights-record.json \
  --allow-host approved.example \
  --timeout-seconds 15 \
  --max-redirects 0 \
  --max-header-bytes 65536 \
  --max-body-bytes 26214400

Preflight parses the rights record, verifies its evidence chain, checks the exact URL and allow-list, resolves DNS, and validates limits. It does not fetch the response body.

Acquire

Use the same inputs and limits that passed preflight:

python3 scripts/source_acquisition.py acquire \
  --url "https://approved.example/path/asset.jpg" \
  --rights-record /absolute/path/rights-record.json \
  --allow-host approved.example \
  --quarantine-root /absolute/path/quarantine \
  --run-id acq-approved-asset-001 \
  --timeout-seconds 15 \
  --max-redirects 0 \
  --max-header-bytes 65536 \
  --max-body-bytes 26214400

Repeat --allow-host only for each independently approved redirect host, and also list every exact redirect target in approved_redirect_urls. The command emits its manifest and returns the manifest exit code. Run it through research-run-ledger; preserve a nonzero run directory and events.jsonl as audit evidence, and never feed a .blocked payload downstream.

Verify

Recompute the complete offline contract: manifest sidecar and exact fields, event journal, request and redirect chain, candidate and terms evidence, rights snapshot, provider digest, payload hash/size/magic type, fixed filenames, single-link files, and exact run-directory contents:

python3 scripts/source_acquisition.py verify \
  --run-directory /absolute/path/quarantine/acq-approved-asset-001

Run verification before registering the file in the provenance ledger and again before any later promotion. Stop on exit code 6. A recomputed SHA sidecar without the required acquisition and evidence semantics does not pass. Verification of an application/json payload reuses the same bounded, no-AST format scanner as acquisition; it is format recognition, not semantic validation or downstream content admission.

When the run ledger or a later promotion gate needs a stable machine-readable handoff, add an absolute path to a new receipt file in an already existing, canonical directory:

python3 scripts/source_acquisition.py verify \
  --run-directory /absolute/path/quarantine/acq-approved-asset-001 \
  --receipt /absolute/path/run-evidence/source-verification.json

For a separately preregistered cross-process handoff, also pass the exact unsigned decimal identities captured by the parent process:

python3 scripts/source_acquisition.py verify \
  --run-directory /absolute/path/quarantine/acq-approved-asset-001 \
  --run-directory-identity 16777234:123456 \
  --receipt /absolute/path/run-evidence/source-verification.json \
  --receipt-parent-identity 16777234:654321

Both identity guards use strict DEV:INO form. A mismatch is rejected before offline verification reads the run payload or creates the receipt. The verifier consumes the manifest, sidecar, event journal, rights snapshot, payload, and directory listing through the pinned run descriptor for the full offline verification. A transient pathname replacement cannot redirect those reads to replacement bytes; persistent ancestry or run-entry drift blocks before receipt publication.

The receipt must be outside the acquisition run and every descendant of that run. The verifier pins the run and receipt-parent directory identities, opens each parent component descriptor-relatively without following links, creates the receipt exclusively, applies exact mode 0400, and syncs both the file and its parent directory. It refuses relative, non-canonical, symlink-traversing, identity-changing, in-run, or existing output paths without overwriting them, and emits the exact same JSON bytes to stdout for research-run-ledger capture. It also preserves a receipt for an integrity mismatch, so receipt existence is not admission: the ledger and every downstream gate must still require process exit 0, ok: true, and status: verified. Never retry by deleting or overwriting an existing receipt; allocate a new append-only run or correction artifact.

Handoff

For an admitted payload:

1. Register the exact payload and manifest in the provenance ledger.
2. Run format-specific malware, resource, pixel/page/frame, metadata, and decompression-bomb checks in a sandbox.
3. Keep original, deterministic derivative, and hypothetical reconstruction identities separate.
4. Promote only through an explicit later gate. Never copy quarantine directly into a public, model, OCR, or application directory.

See the contract for the AIC single-sample sequence, output fields, and exit codes.