Scrapling ops

scrapling-ops — ingestion skill (Phase 6, generic Scrapling helper)

10-step protocol. Steps map 1:1 to workflow_runner invocations + the spec §16.5 8-step MCP discipline + the §14.5 canonical tier ladder. Raw HTML drift recovery is mandatory: every successful tier response is dropped into inbox/scrapling/ before the staging projection runs, so a staging bug never costs us the upstream payload.

This skill is the generic Scrapling helper. It does NOT enforce per-scenario sub-schemas (templates/scrapling/[scenario].schema.json) — those belong to Phase 7+ scenario consumer skills (S1 competitor snapshot, S2 external mentions, etc.) per ADR-025. The templates/scrapling/.gitkeep placeholder must remain untouched until those skills land.

Inputs (frontmatter contract)

Name	Type	Default	Notes
project_slug	string	—	Required. Resolves projects/{slug}/_state/staging/.
urls	array	—	Required. URL list. ≤10 → single MCP, >10 → bulk MCP.
scenario	string	generic	Stored on every staging row; downstream filter key.
max_urls	integer	50	DURUR threshold; do not silently truncate (DURUR #6).

Outputs (artifacts produced)

• projects/{slug}/inbox/scrapling/{date}-{tier}-{slug}.json — raw per-tier MCP response envelopes (drift recovery; one file per tier that produced ≥1 successful response).
• projects/{slug}/_state/staging/scrapling_{date}_{slug}.json — SQLite-shaped staging table (rows array, columns array). NO Excel write in Phase 6 — staging-only per ADR-025.
• projects/{slug}/outputs/reports/{date}-scrapling-ops.md — human-readable summary (per-tier success counts, DURUR list).
• projects/{slug}/_state/events.jsonl — event_kind=provenance entries (source.kind=scrapling_mcp, target_excel_sheet=null).

Tier Escalation (canonical, schema-locked §14.5)

The state machine is fixed by schemas/scrapling-output-mapping.schema .json (tierEscalation definition: ["get", "fetch", "stealthy_fetch"], minItems=3, maxItems=3, const-equal). Each URL flows through tiers in order; first success short-circuits. All-fail records the URL in the staging table with tier_used=null and status='durur' — the orchestrator decides whether to halt the whole run (DURUR #1) based on aggregate failure rate.

Tier 0  get             — basic HTTP GET, no JS, no anti-bot.
                          Failure modes: 403, 429, captcha_detected, timeout.
Tier 1  fetch           — browser-like headers, optional JS render.
                          Failure modes: persistent 403, captcha,
                                         Cloudflare challenge.
Tier 2  stealthy_fetch  — anti-bot bypass (Cloudflare-aware,
                          fingerprint randomization).
                          Failure modes: still blocked → DURUR
                          (do not invent tier 3).

Bulk URL lists (>10) use mcp__ScraplingServer__bulk_fetch and mcp__ScraplingServer__bulk_stealthy_fetch. Tier 0 (get) always runs per-URL — it is the cheapest tier and per-URL form keeps logic identical to the small-list path. The escalation is per-URL: a partial-success batch can have URLs at different tiers in the same staging table.

10-Step Body Protocol

Each step name must match the steps[*].name passed to workflow_runner.create_run. Names are stable identifiers across runs.

Step 1 — create_run

Open a workflow run shell. The state file lives at projects/{slug}/_state/workflows/{run_id}.json (ADR-021, ADR-019).

from scripts.state import workflow_runner
handle = workflow_runner.create_run(
    skill="scrapling-ops",
    project_slug=project_slug,
    steps=[
        {"name": "validate_inputs"},
        {"name": "open_session"},
        {"name": "tier_escalate"},
        {"name": "persist_inbox"},
        {"name": "request_approval"},
        {"name": "write_staging"},
        {"name": "render_report"},
        {"name": "close_session"},
    ],
)

Step 2 — validate_inputs

Validate len(urls) <= max_urls and project workspace exists. URL count over budget → DURUR #6 immediately, no MCP call wasted.

from scripts.ingestion import scrapling_ops
if len(urls) > max_urls:
    raise scrapling_ops.UrlBudgetExceededError(
        f"url count {len(urls)} exceeds max_urls={max_urls}"
    )

Step 3 — open_session (optional)

If the MCP server exposes session lifecycle (open_session, close_session), open one and bind it to the tier callables. Session imbalance — open without matching close, or close without prior open — is DURUR #5.

Step 4 — tier_escalate (canonical §14.5 ladder)

Drive the state machine via the helper:

from scripts.ingestion import scrapling_ops
mcp_clients = {
    "get":            mcp__ScraplingServer__get,            # or fetch fallback
    "fetch":          mcp__ScraplingServer__fetch,
    "stealthy_fetch": mcp__ScraplingServer__stealthy_fetch,
}
bulk_clients = {
    "fetch":          mcp__ScraplingServer__bulk_fetch,
    "stealthy_fetch": mcp__ScraplingServer__bulk_stealthy_fetch,
}
results = scrapling_ops.bulk_tier_escalate(
    urls, mcp_clients, bulk_clients=bulk_clients,
)

If aggregate failure rate exceeds 50% the orchestrator halts → DURUR #2.

Step 5 — persist_inbox (raw drift recovery, §16.5 step 3)

For every tier that produced ≥1 success, persist its raw response envelope to:

projects/{slug}/inbox/scrapling/{date}-{tier}-{slug}.json

Inbox path creation failure → DURUR #3.

Step 6 — request_approval (skill EXIT awaiting_approval)

workflow_runner.request_approval(
    handle.run_id, project_slug=project_slug,
    approver="user",
    subject=f"{len(urls)} URL fetch tamamlandı, staging'e yazalım mı?",
    step_index=4,
)

Skill exits here. The user replies in a fresh session; resume below.

Step 7 — write_staging (NO Excel — Phase 6 staging-only)

Build SQLite-shaped staging rows and write to disk. Schema drift — column count or order mismatch with STAGING_COLUMNS — is DURUR #4.

output_path = (
    workspace_root / "projects" / project_slug
    / "_state" / "staging"
    / f"scrapling_{date}_{project_slug}.json"
)
rows = scrapling_ops.build_staging_rows(results, scenario=scenario)
scrapling_ops.write_staging_json(rows, output_path)

Note: transaction.append is NOT called — there is no Excel target in Phase 6. The staging file is the durable artifact.

Step 8 — render_report

render_template.py templates/reports/scrapling-ops.template.md data.json → outputs/reports/{date}-scrapling-ops.md. Variables (must match the template's $tokens exactly): $project_slug, $date, $scenario, $url_count, $max_urls, $tier_distribution, $staging_path, $run_id.

Step 9 — Provenance event (source.kind=scrapling_mcp)

from scripts.state import events_writer
events_writer.append_provenance(
    project_id=project_slug,
    source={
        "kind": "scrapling_mcp",
        "mcp_server": "ScraplingServer",
        "mcp_tool": "ScraplingServer__bulk_fetch",
        "response_bytes": total_bytes,
    },
    operation="ingest",
    target_excel_sheet=None,   # staging-only — no Excel projection
    target_table=f"scrapling_staging_{scenario}",
    rows_written=len(rows),
)

target_excel_sheet=None is intentional and supported by schemas/events.schema.json (oneOf: logicalSheet | null) — Phase 6 staging operations carry no Excel sheet.

Step 10 — complete (+ close_session)

workflow_runner.complete(handle.run_id, project_slug=project_slug, outputs={
    "row_count": len(rows),
    "staging_path": str(output_path),
    "report_path": str(report_path),
    "tier_counts": tier_counts,
})

If a session was opened in Step 3, close it here; imbalance → DURUR #5.

Staging Table (SQLite-shaped JSON)

Columns (scripts.ingestion.scrapling_ops.STAGING_COLUMNS):

Column	Type	Notes
id	integer	1-based contiguous; row order = input URL order.
url	string	Original URL as supplied.
tier_used	string | null	get / fetch / stealthy_fetch; null when all 3 failed.
status	string	ok (tier_used set) or durur (all tiers failed).
content_hash	string | null	sha256:<hex64> of UTF-8 content; null on failure.
content_bytes	integer	Byte length of content; 0 on failure.
fetched_at	string	ISO 8601 UTC, microsecond, 'Z'-suffixed.
error_message	string | null	; -joined per-tier errors on durur; null on ok.
scenario	string	Carried from input; downstream filter key.

The container envelope is {"schema_version": "1.0", "table": "scrapling_staging", "columns": [...], "rows": [...], "row_count": N}.

Idempotency note: re-running the same URL list yields the same row shape (id sequence, columns) but content_hash may drift if the target page has changed between runs. This is intentional — content drift is a signal, not a bug.

DURUR conditions (6+)

Stop and flag the manager — do not patch, do not fall back.

1. Tier escalation all-3 fail for a URL the orchestrator considers load-bearing (e.g. all URLs in a single-URL run, or aggregate durur_count / total > 0.5 in a batch).
2. Bulk fetch >50% URL error rate at tier 1 or 2 (per-batch).
3. Inbox path creation fail — inbox/scrapling/ not writable (workspace path missing, permission denied).
4. Staging JSON schema drift — STAGING_COLUMNS mismatch (column added/removed without ADR), staging row missing required key, or id sequence not contiguous-1-based.
5. Session open/close imbalance — open_session succeeded but close_session skipped/failed, or close_session called without prior open (resource leak).
6. URL count > max_urls — refuse rather than silently truncate (raises UrlBudgetExceededError upstream of any MCP call).

Cross-references

• Schemas: schemas/scrapling-output-mapping.schema.json (§14.5 tierEscalation const-locked sequence; scenarios.* deferred to Phase 7+ per ADR-025), schemas/events.schema.json (source.kind=scrapling_mcp, target_excel_sheet=null), schemas/skill-frontmatter.schema.json.
• Cross-modules: scripts/state/workflow_runner.py, scripts/state/events_writer.py, scripts/reporting/render_template.py. Note: scripts/excel/transaction.py is NOT consumed — Phase 6 has no Excel target.
• Helper: scripts/ingestion/scrapling_ops.py (tier state machine, bulk variants, staging writer).
• Tests: tests/skills/test_scrapling_ops.py (≥6 cases incl. all-fail DURUR, bulk threshold, max_urls budget).
• Templates: templates/scrapling/.gitkeep placeholder kept untouched per ADR-025 — per-scenario sub-schemas land with Phase 7+ skills.
• ADR: ADR-025 (Phase 7+ scenario sub-schema defer). The max_urls cap is DURUR #6 — no accepted ADR governs the URL-count budget (ADR-026 is the unrelated DECISIONS byte-cap; Q-015 → ADR-025 per OPEN_QUESTIONS.md).