Shipkit prompt audit

shipkit-prompt-audit - LLM Prompt Architecture Audit

Purpose: Find structural problems in how your app talks to LLMs — monolithic prompts, missing fallbacks, sequential bottlenecks, unvalidated outputs, unsafe inputs.

What this is NOT: Not a prompt text quality checker. Not "make this prompt better." This audits the engineering architecture around prompts — how they're decomposed, chained, validated, and recovered from failure.

---

When to Invoke

• /shipkit-prompt-audit — audit all LLM integrations
• /shipkit-prompt-audit src/ai/ — focus on specific directory
• "Audit my prompts", "Check prompt architecture", "LLM pipeline review"
• "Are my AI calls structured well?"
• "Check my prompts for anti-patterns"

Workflow position:

• After implementing AI features, before shipping
• When AI features feel slow or unreliable
• During architecture review of LLM-heavy applications
• When scaling from prototype to production AI

---

Prerequisites

Required:

• Project has LLM integrations (API calls to OpenAI, Anthropic, Gemini, etc.)

Recommended:

• .shipkit/stack.json — Knows which AI SDKs are in use
• .shipkit/architecture.json — Knows pipeline design intent

If no LLM integrations found: Report cleanly and exit. Don't fabricate findings.

---

Process

Completion Tracking

After discovering integration points (Step 1), create tasks:

• TaskCreate: "Map pipeline topology"
• TaskCreate: "Audit all 10 dimensions (PA-DEC through PA-REF)"
• TaskCreate: "Write prompt-audit.json (schema-validated)"
• TaskCreate: "Present summary to user"

TaskUpdate each task to in_progress when starting it, completed when done.

If using parallel subagents for dimension auditing, verify ALL agent results are collected before synthesizing. Do NOT write the report until all 10 dimensions have findings (even if finding is "N/A").

Step 1: Discover LLM Integration Points

Scan the codebase for all LLM-related code using detection patterns.

Index-Accelerated Discovery — Read .shipkit/codebase-index.json first:

1. Read: .shipkit/codebase-index.json
2. If index exists:
   • Check concepts for any AI/LLM-related concept mappings
   • Use configFiles to identify AI SDK config files (e.g., AI provider configs)
   • Use directories to identify likely locations (e.g., src/lib, src/services, src/api)
   • Pass this context to the Explore agent to narrow its search scope
3. If index doesn't exist → full scan as below

USE SUBAGENT FOR DISCOVERY — Launch Explore subagent for efficient parallel scanning:

Agent tool with subagent_type: "Explore"
Prompt: "Find all LLM integration points in this codebase.
[If index exists, include: 'The codebase index shows: directories=[dirs], config files=[configs], concepts=[concepts]. Focus search on these locations first before scanning broadly.']

Search for these patterns (see references/detection-patterns.md for full list):

API CLIENTS:
- OpenAI: openai.chat.completions, new OpenAI(
- Anthropic: anthropic.messages, new Anthropic(
- Gemini: googleGenerativeAI, model.generateContent
- LangChain: ChatOpenAI, LLMChain, RunnableSequence
- Vercel AI: generateText, streamText, generateObject

PROMPT TEMPLATES:
- Template literals with system/user roles
- Prompt files (.prompt, .txt used as prompts)
- Structured message arrays [{role, content}]

SCHEMA VALIDATION:
- Zod schemas near AI calls
- JSON.parse on AI responses
- Structured output definitions

Return for each integration point:
- File path and line number
- Provider/SDK used
- Whether it's a single call or part of a chain
- Input source and output destination"

Why subagent: Discovery requires many parallel grep/glob operations across the codebase. Explore agent handles this efficiently.

Fallback (if subagent unavailable) — Manual scanning:

1. Grep for API client instantiation:
   pattern: "new OpenAI\(|new Anthropic\(|GoogleGenerativeAI|ChatOpenAI"

2. Grep for API calls:
   pattern: "chat\.completions\.create|messages\.create|generateContent|generateText|streamText"

3. Grep for prompt templates:
   pattern: "system.*role|role.*system|You are a|As an AI"

4. Glob for prompt files:
   pattern: "**/*.prompt", "**/*.prompt.md", "**/prompts/**"

If zero integration points found: Report "No LLM integrations detected" and exit cleanly.

Output of Step 1: List of all LLM integration points with file:line references.

---

Step 2: Map Pipeline Topology

For each discovered integration point, trace the data flow:

For each LLM call, determine:

Question	How to Check	Classification
Is this a single call or part of a chain?	Read surrounding code for sequential AI calls	SINGLE / CHAIN
Does output feed into another prompt?	Grep for variable reuse in next AI call	FEEDS_NEXT / TERMINAL
Are there parallel calls?	Check for Promise.all, concurrent AI calls	PARALLEL / SEQUENTIAL
Is the output validated?	Check for schema validation, JSON.parse, type checks	VALIDATED / UNVALIDATED
Is there error handling?	Check for try/catch, fallback logic	HANDLED / UNHANDLED

Build a topology map and serialize it for graph visualization:

For each multi-stage pipeline, build a pipelines[] entry with stages (nodes) and edges (data flow):

Pipeline: [name/purpose]
  Stage 1: [provider] [model] — [purpose]
    Input: [source]
    Output: [validated? schema?]
    Fallback: [yes/no]
    ↓
  Stage 2: [provider] [model] — [purpose]
    Input: [from Stage 1 output]
    Output: [validated? schema?]
    Fallback: [yes/no]

Capture as graph-ready data:

• Each stage becomes a node: { id, promptId, provider, purpose, location }
• Each data dependency becomes an edge: { id, source, target, dataFlow, validated }
• Mark validated: true if data is schema-checked between stages, false otherwise
• Tag pipeline type: "chain" (sequential), "parallel" (concurrent stages), "mixed" (both)
• Link issues[] to relevant issue IDs from the audit findings

This topology data powers pipeline flow graph visualization.

For multi-stage pipelines (3+ stages):

• Identify which stages could run in parallel
• Note sequential dependencies that can't be parallelized
• Flag stages that share no data dependencies but run sequentially

---

Step 3: Audit 10 Dimensions

Review each integration point and pipeline against these dimensions.

See references/audit-checklist.md for detailed checks per dimension.

FOR MULTIPLE INTEGRATION POINTS (3+), USE PARALLEL SUBAGENTS:

Launch these Agent subagents IN PARALLEL (single message, multiple tool calls):

1. ARCHITECTURE AGENT (subagent_type: "Explore")
   Prompt: "Audit prompt decomposition and parallelization for these LLM integrations: [list files:lines]
   PA-DEC: Check for God Prompts (too many concerns), identify decomposition opportunities.
   PA-PAR: Check for sequential calls that could run parallel, unnecessary data dependencies.
   PA-CHN: Check chain integrity - does each stage get what it needs, are intermediates type-checked?
   Report Pass/Fail/Warning with file:line evidence and anti-pattern matches."

2. VALIDATION & SAFETY AGENT (subagent_type: "Explore")
   Prompt: "Audit schema validation and input safety for these LLM integrations: [list files:lines]
   PA-SCH: Check if outputs are validated with schema/zod, structured output modes used, JSON.parse wrapped.
   PA-INP: Check for input sanitization, prompt injection defenses, system/user content separation.
   PA-REF: Check if prompts reference existing code/data, examples are valid, no deprecated APIs/models.
   Report Pass/Fail/Warning with file:line evidence and anti-pattern matches."

3. RELIABILITY AGENT (subagent_type: "Explore")
   Prompt: "Audit reliability patterns for these LLM integrations: [list files:lines]
   PA-FBK: Check for fallback paths on failure, retry logic with backoff, rate limit handling.
   PA-CAC: Check if deterministic prompts are cached, expensive stages memoized, cache invalidation exists.
   PA-CTX: Check for unnecessary context, repeated context across stages, token limit risks.
   PA-CON: Check if output format instructions are explicit and testable.
   Report Pass/Fail/Warning with file:line evidence and anti-pattern matches."

Why parallel subagents:

• 10 dimensions across multiple integration points = many checks
• Each agent focuses on related dimensions
• Findings grouped logically (architecture vs safety vs reliability)

When to use parallel agents:

• 3+ LLM integration points discovered
• Multi-stage pipelines detected
• Deep audit requested

When to scan manually:

• 1-2 simple integration points
• Single-stage prompts only

Dimension 1: Decomposition (PA-DEC)

• Is the prompt doing too many things at once? (God Prompt)
• Could it be broken into focused sub-tasks?
• Are different concerns mixed in one prompt?

Dimension 2: Parallelization (PA-PAR)

• Are independent stages running sequentially?
• Could Promise.all/concurrent execution speed things up?
• Are there unnecessary data dependencies between stages?

Dimension 3: Schema Tightness (PA-SCH)

• Is the LLM output validated against a schema?
• Are structured output modes used where available?
• Is JSON.parse wrapped in try/catch?

Dimension 4: Chain Integrity (PA-CHN)

• Does each stage receive what it needs from the previous?
• Are intermediate results type-checked?
• Could a stage fail silently and corrupt downstream?
• Do early knowledge-gathering stages ask the right questions for downstream needs?

Dimension 5: Cache Boundaries (PA-CAC)

• Are deterministic prompts (same input = same output) cached?
• Are expensive stages memoized?
• Is there cache invalidation logic?

Dimension 6: Fallback Paths (PA-FBK)

• What happens when an LLM call fails?
• Is there retry logic with backoff?
• Is there a degraded experience (fallback to simpler model/static content)?
• Are rate limits handled?

Dimension 7: Context Efficiency (PA-CTX)

• Are prompts including unnecessary context?
• Is the same context repeated across chain stages?
• Could RAG/retrieval reduce prompt size?
• Are token limits being approached?

Dimension 8: Constraint Precision (PA-CON)

• Are output format instructions explicit?
• Are constraints testable (not vague)?
• Are few-shot examples used where format matters?
• Does the task verb match the intended cognitive mode? (generate vs evaluate vs research)

Dimension 9: Input Safety (PA-INP)

• Is user input sanitized before injection into prompts?
• Are prompt injection defenses in place?
• Are system prompts separated from user content?

Dimension 10: Reference Integrity (PA-REF)

• Do prompts reference code/data that exists?
• Are hardcoded examples still valid?
• Do prompts reference deprecated APIs or models?

---

Step 3.5: Verification Protocol

Critical: Execute tools before classifying findings.

Every finding MUST be backed by actual tool output. Follow the same verification protocol as /shipkit-review-shipping:

Step	Action
1. IDENTIFY	What tool call proves this claim?
2. RUN	Execute the Glob/Grep/Read
3. READ	Examine full output
4. CLASSIFY	Determine severity from evidence
5. REPORT	State finding WITH evidence

Anti-pattern checks (see references/anti-patterns.md):

• Check each integration point against known anti-patterns
• Cross-reference with pipeline topology
• Classify matches by severity

---

Step 4: Generate Report

Create file using Write tool: .shipkit/prompt-audit.json

The output MUST conform to the schema below. This is a strict contract — other skills depend on this structure.

---

JSON Schema

Full schema: See references/output-schema.md Example output: See references/example.json

Quick Reference

{
  "$schema": "shipkit-artifact",
  "type": "prompt-audit",
  "version": "1.0",
  "lastUpdated": "YYYY-MM-DD",
  "source": "shipkit-prompt-audit",
  "summary": {
    "totalPromptsAudited": number,
    "totalIssuesFound": number,
    "bySeverity": { "critical", "shouldFix", "minor" },
    "integrationPoints": number,
    "providers": ["OpenAI", "Anthropic", ...],
    "pipelines": { "singleStage", "multiStage" }
  },
  "prompts": [{
    "id", "location", "provider", "purpose", "pipelineType", "score",
    "issues": [{ "id", "dimension", "severity", "title", "evidence", "impact", "fix", "antiPattern?" }]
  }],
  "pipelines": [{
    "id", "name", "type",
    "stages": [{ "id", "promptId", "provider", "purpose", "location" }],
    "edges": [{ "id", "source", "target", "dataFlow", "validated" }],
    "issues?": ["issue-id-refs"]
  }],
  "patterns": {
    "antiPatterns": [{ "name", "instances", "files", "description" }],
    "positivePatterns": [{ "name", "instances", "description" }]
  },
  "recommendations": [{ "priority", "title", "description", "affectedFiles", "effort" }]
}

Key Fields

Field	Description
prompts[].pipelineType	"single" | "chain" | "parallel"
prompts[].issues[].severity	"critical" | "shouldFix" | "minor"
pipelines[].type	"chain" | "parallel" | "mixed"
pipelines[].edges[].validated	Whether data between stages is schema-checked
recommendations[].effort	"low" | "medium" | "high"

---

Step 5: Present Results

Output to user:

Prompt Architecture Audit complete

Scanned: X integration points across Y files
Providers: [OpenAI, Anthropic, ...]
Pipelines: X single-stage, Y multi-stage

Critical: X findings
Should Fix: X findings
Minor: X suggestions

Top issues:
1. [Brief description with file reference]
2. [Brief description with file reference]
3. [Brief description with file reference]

Full report: .shipkit/prompt-audit.json

Want me to fix any of these issues?

---

Priority Definitions

Priority	Meaning	Example
Critical	Will cause failures, security issues, or data corruption	Unvalidated LLM output used in database query; no fallback on payment-critical AI call
Should Fix	Quality/reliability issues, technical debt	God Prompt that should be decomposed; sequential calls that could parallelize
Minor	Suggestions, optimizations	Caching opportunity; slightly verbose context

---

Shipkit Artifact Convention

This skill follows the Shipkit JSON artifact convention. See references/output-schema.md for full documentation.

---

What This Skill Does NOT Do

• Rewrite prompts — reports issues, doesn't fix them
• Judge prompt text quality — "make this prompt better" is not in scope
• Run prompts — static analysis only, no execution
• Test output quality — that's evaluation tooling (Promptfoo, etc.)
• Persist beyond report — single audit snapshot

---

Context Files This Skill Reads

File	Purpose
Source code	The actual LLM integration code
.shipkit/stack.json	Know which AI SDKs/providers are used
.shipkit/architecture.json	Understand intended pipeline design
package.json / requirements.txt	Detect AI SDK dependencies

---

Context Files This Skill Writes

Write Strategy: OVERWRITE

• .shipkit/prompt-audit.json — Audit report (JSON artifact, overwritten on each run)

---

When This Skill Integrates with Others

Routes FROM

Source	When
User (direct)	User asks "audit prompts", "check AI pipeline"
/shipkit-review-shipping	Flags AI-related code for deeper review
/shipkit-preflight	Pre-launch check of AI features

After This Skill

• User fixes findings (natural capability)
• Re-run /shipkit-prompt-audit to verify fixes
• /shipkit-engineering-definition — update engineering blueprint with pipeline decisions

Differs From

Skill	Focus
/shipkit-review-shipping	Code quality across 12 dimensions (breadth)
/shipkit-prompt-audit	LLM pipeline architecture (depth on AI code)
/shipkit-preflight	Production readiness (broader than AI)
/shipkit-scale-ready	Scale & operations (infrastructure focus)

---

<!-- SECTION:after-completion -->

After Completion

Audit delivered. User decides next steps:

1. Fix critical issues — Ask Claude to help restructure pipelines
2. Review should-fix items — Decide which to address now
3. Re-run audit — Verify fixes resolved the issues
4. Log decisions — Use /shipkit-engineering-definition for pipeline design choices

Natural capabilities (no skill needed): Implementing fixes, refactoring prompts, adding validation.

No follow-up skill automatically triggered.

<!-- /SECTION:after-completion -->

---

<!-- SECTION:success-criteria -->

Success Criteria

• LLM integration points discovered via codebase scan
• Pipeline topology mapped (single vs chain, parallel vs sequential)
• All 10 dimensions audited with tool-verified evidence
• Anti-patterns checked against known patterns
• Findings prioritized (Critical / Should Fix / Minor)
• Each finding has specific file:line reference
• Report saved to .shipkit/prompt-audit.json
• Output conforms to JSON schema above
• Summary field is accurate
• Report only — no unsolicited fixes

<!-- /SECTION:success-criteria -->

---

References

• references/detection-patterns.md — Grep/Glob patterns for discovering LLM integrations
• references/audit-checklist.md — Detailed checks per dimension with IDs and severity
• references/anti-patterns.md — Common prompt architecture anti-patterns