Robots policy audit

robots-policy-audit — discovery skill (GAP-T3, FREE)

robots.txt / noindex lifecycle governance per rules/tech-seo-governance.md R-131 (governed robots.txt policy), R-132 (noindex deployment path / R-58 deployability), R-133 (noindex/disallow mutual exclusion). Today the robots_txt sheet is an SF-issue dump; R-58 (rules/content-html-discipline.md) maps content lifecycle → meta robots but is undeployable (the content pipeline emits body fragments with no <head> access → ON_HOLD content is silently indexable). This skill VALIDATES the deployed reality and emits the deployment path R-58 lacks.

Recommendation-only. The live robots.txt GET is a read of a public file (no consent gate). Every outward change — robots.txt edits, per-page noindex, X-Robots-Tag headers — is operator-deployed. The engine emits findings (RP- rows), a proposed robots.txt artifact, and platform deployment instructions; it NEVER writes to client infrastructure. The Excel write is behind the standard request_approval gate. Scope fence: content-remediation owns single-URL retire (R-90/R-91); this skill validates outcomes, never re-issues a retire.

Outputs

• master.xlsx#robots_txt — RP-NNN findings (5-col; severityEnum) via scripts/util/sheet_merge.merge_prefixed_rows(..., id_prefix="RP-") — idempotent, preserves sf_projection R-, facet-nav-audit FN-, hreflang HF- rows.
• outputs/reports/{date}-robots-policy-audit.md — lint + conflict + lifecycle drift tables + proposed robots.txt + deployment instructions.
• outputs/robots/{date}-robots.proposed.txt — the proposed robots.txt as a plain artifact (plain Write, no template; new outputs/robots/ subdir — workspace data, no engine schema impact). Recommendation only — operator deploys.
• events.jsonl — append_provenance(source.kind="sf_csv", operation="project_excel", target_excel_sheet="robots_txt").

Body protocol

1. create_run.
2. fetch + read — fetch_live (default true): Scrapling GET https://{domain}/robots.txt. AMBER fallback (R9 pattern): on unreachable / non-200 the run AMBER-warns and continues file-only with an empty robots.txt (the transform emits a MEDIUM "missing robots.txt", never crashes) — this is AMBER, NOT a DURUR. Read directives_all.csv + internal_all.csv from the export; read master.xlsx#new_content_plan (col C url_slug, col K lifecycle_status); optionally read the facet-nav-audit proposed block.
3. transform (pure) —

   from scripts.discovery import robots_policy_transform as rpt
   out = rpt.transform(
       robots_txt_text, directives_rows, internal_rows, lifecycle_rows,
       platform, domain, facet_block=facet_block, url_patterns=url_patterns,
   )
   

   Lints robots.txt (noindex-line=HIGH, missing-Sitemap=MEDIUM, site-wide Disallow=CRITICAL, unknown=LOW), scans R-133 conflicts + R-58 lifecycle drift, protects important pages, and builds proposed_robots_txt + a platform deployment_instructions matrix (unverified platforms marked verified: false).
4. request_approval — skill EXIT awaiting_approval.
5. write_excel — on approve, sheet_merge.merge_prefixed_rows(..., id_prefix="RP-", writer="robots-policy-audit").
6. write proposed artifact — plain Write of out["proposed_robots_txt"] to outputs/robots/{date}-robots.proposed.txt.
7. render_report — render_template.py templates/reports/robots-policy-audit.template.md.
8. provenance event + complete (string-typed outputs).

DURUR conditions

Stop and flag the manager (live-fetch failure is AMBER, NOT here).

1. directives_all.csv missing (RobotsDirectivesMissingError).
2. lifecycle sheet (new_content_plan) unreadable (RobotsLifecycleUnreadableError).
3. Proposed file would disallow / (RobotsProposedSiteWideBlockError) — never propose a site-wide block.
4. directives_all.csv Address column absent (RobotsDirectivesSchemaDriftError).
5. master.xlsx#robots_txt row schema mismatch on write (RowSchemaError).
6. PSEO_WORKSPACE_ROOT unset and no workspace_root arg.

AMBER (NOT DURUR): live robots.txt fetch unreachable / non-200 → warn, continue file-only (mirror tech-audit R9 never-hard-fail).

Cross-references

• Rules: rules/tech-seo-governance.md (R-131 / R-132 / R-133); rules/content-html-discipline.md (R-58).
• Transform: scripts/discovery/robots_policy_transform.py (transform, PLATFORM_DEPLOYMENT_MATRIX, RobotsPolicyError hierarchy).
• Write path: scripts/util/sheet_merge.py (merge_prefixed_rows, prefix RP-).
• Upstream feed: facet-nav-audit proposed robots block (optional facet_block).
• Schemas: schemas/master-excel.schema.json#robots_txt, #new_content_plan, schemas/events.schema.json.
• Template: templates/reports/robots-policy-audit.template.md.
• Command: commands/pseo-robots-policy.md.

Discipline checklist

• Recommendation-only — live GET is a public-file READ; all deploys operator-owned (R-131/R-132).
• Free — budget.uses_paid_mcp=false; no DFS.
• AMBER never RED on live-fetch failure (R9 pattern); transform tolerates empty robots.txt.
• Schema-first — frontmatter validates; RP- rows validate against master-excel.schema.json#robots_txt.
• Plugin-agnostik — no slug literals; pure transform; unverified platform channels marked verified: false (no fabricated panel paths).
• Scope fence — validates lifecycle outcomes; never re-issues content-remediation's single-URL retire.
• Idempotent write — merge_prefixed_rows re-lands the RP- namespace; foreign rows preserved.