Deploy verify

Post-Deploy Verification

Automated post-deploy health check across all services. Run after every deploy to catch regressions before users do. Works with any project — auto-detects configuration.

Critical Rules

Run within 15 minutes of deploy. The sooner you check, the smaller the blast radius.

Parallel where possible. Sentry, Supabase, and Langfuse checks are independent — run them in parallel to save time.

The verdict must be binary. SHIP, ROLLBACK, or MONITOR (with clear criteria for when MONITOR escalates to ROLLBACK).

Evidence over opinion. Every check produces a PASS/FAIL with specific data. Never say "looks fine."

Always use the browser-anti-stall protocol when using Playwright browser MCP tools.

---

Phase 0: Auto-Detect Deployment Context

0a. Detect Deployment Platform

Grep(pattern: "vercel\\.json|netlify\\.toml|fly\\.toml|render\\.yaml|railway\\.json|Dockerfile|docker-compose|appspec\\.yml", output_mode: "files_with_matches")
Grep(pattern: "VERCEL_URL|RENDER_EXTERNAL_URL|FLY_APP_NAME|RAILWAY_STATIC_URL", glob: ".env*")

0b. Find Production URL

Grep(pattern: "NEXT_PUBLIC_APP_URL|NEXT_PUBLIC_BASE_URL|VITE_APP_URL|PUBLIC_URL|PRODUCTION_URL|APP_URL", glob: ".env*")
Grep(pattern: "https://.*\\.vercel\\.app|https://.*\\.netlify\\.app|https://.*\\.fly\\.dev|https://.*\\.railway\\.app", glob: ".env*")

Also check framework config files:

Grep(pattern: "url|domain|hostname", glob: "{vercel.json,netlify.toml,next.config.*}")

0c. Detect Sentry Release Tracking

Grep(pattern: "SENTRY_ORG|SENTRY_PROJECT|SENTRY_DSN|SENTRY_AUTH_TOKEN|@sentry/nextjs|@sentry/react|@sentry/node|sentry-cli", glob: "{.env*,package.json,.sentryclirc,sentry.*.config.*}")

0d. Detect Supabase Project

Grep(pattern: "SUPABASE_URL|NEXT_PUBLIC_SUPABASE_URL|SUPABASE_SERVICE_ROLE_KEY|SUPABASE_ANON_KEY", glob: ".env*")

To get the project ID for MCP calls:

supabase:list_projects
{}

Match the detected URL to a project to get project_id.

0e. Detect Langfuse Integration

Grep(pattern: "LANGFUSE_PUBLIC_KEY|LANGFUSE_SECRET_KEY|LANGFUSE_HOST|@langfuse", glob: "{.env*,package.json}")

0f. Find Critical User Flows

Detect the app's route structure to identify critical paths for smoke testing:

Glob("**/app/**/page.{tsx,jsx,ts,js}")
Glob("**/pages/**/*.{tsx,jsx,ts,js}")
Glob("**/src/routes/**/*.{tsx,jsx,svelte}")

Identify the critical paths (prioritized):

1. Landing/home page
2. Authentication (login/signup)
3. Main feature (the primary value proposition)
4. One CRUD operation (create or update something)
5. Any AI/LLM feature (if present)

Record all detected values:

• PRODUCTION_URL
• SENTRY_ORG, SENTRY_PROJECT
• SUPABASE_PROJECT_ID
• LANGFUSE_AVAILABLE (boolean)
• CRITICAL_PATHS (list of URLs to smoke test)

---

Phase 1: Sentry Error Check

1a. Find Organization and Project

sentry:find_organizations
{}

Use the detected org slug and project slug from Phase 0c.

1b. Check Recent Releases

sentry:find_releases
{
 "organizationSlug": "<ORG_SLUG>",
 "projectSlug": "<PROJECT_SLUG>"
}

Verify:

• Latest release matches the deploy (commit hash or version tag)
• Release was created within the expected deploy window

1c. Search for New Issues Since Deploy

sentry:search_issues
{
 "organizationSlug": "<ORG_SLUG>",
 "projectSlug": "<PROJECT_SLUG>",
 "query": "is:unresolved firstSeen:>1h",
 "sortBy": "freq"
}

Also check for regressions (previously resolved issues that re-opened):

sentry:search_issues
{
 "organizationSlug": "<ORG_SLUG>",
 "projectSlug": "<PROJECT_SLUG>",
 "query": "is:regressed",
 "sortBy": "freq"
}

1d. Seer Analysis on P0 Issues

For any new issue with high event count or critical severity, run Sentry's AI root-cause analysis:

sentry:analyze_issue_with_seer
{
 "organizationSlug": "<ORG_SLUG>",
 "issueId": "<ISSUE_ID>"
}

This provides:

• Root cause explanation with code-level detail
• Specific file locations and line numbers
• Concrete code fix suggestions

Decision criteria:

• 0 new issues → PASS
• New issues but low frequency (<5 events) and non-critical → MONITOR
• New high-frequency issues or regressions → FAIL (potential rollback trigger)
• Seer identifies a one-line fix → APPLY FIX instead of rollback

---

Phase 2: Supabase Health Check

Skip this phase if no Supabase integration detected.

2a. Migration Health

supabase:list_migrations
{
 "project_id": "<PROJECT_ID>"
}

Verify:

• Latest migration matches what the deploy expected
• No failed or pending migrations

2b. API Logs (last 30 minutes)

supabase:get_logs
{
 "project_id": "<PROJECT_ID>",
 "service": "api"
}

Check for:

• 5xx errors (server errors)
• Unusual 4xx spike (broken client code)
• Slow queries (>1s response time)

2c. Edge Function Logs

supabase:get_logs
{
 "project_id": "<PROJECT_ID>",
 "service": "edge-function"
}

Check for:

• Function boot errors
• Runtime exceptions
• Timeout errors

2d. Auth Logs

supabase:get_logs
{
 "project_id": "<PROJECT_ID>",
 "service": "auth"
}

Check for:

• Auth failures unrelated to wrong credentials
• Token refresh errors
• RLS policy violations

2e. Security and Performance Advisors

supabase:get_advisors
{
 "project_id": "<PROJECT_ID>",
 "type": "security"
}

supabase:get_advisors
{
 "project_id": "<PROJECT_ID>",
 "type": "performance"
}

Flag any new warnings that weren't present before the deploy.

2f. Critical Data Integrity

Run a quick data integrity check on the most important tables:

supabase:execute_sql
{
 "project_id": "<PROJECT_ID>",
 "query": "SELECT schemaname, relname, n_dead_tup, last_autovacuum FROM pg_stat_user_tables WHERE n_dead_tup > 10000 ORDER BY n_dead_tup DESC LIMIT 5"
}

Decision criteria:

• All logs clean, migrations applied, no advisor warnings → PASS
• Minor warnings in logs but no 5xx → MONITOR
• 5xx errors in API/Edge logs or failed migrations → FAIL

---

Phase 3: Langfuse Health Check

Skip this phase if no Langfuse integration detected.

3a. Check Recent Traces

npx langfuse-cli api traces list --limit 10

Verify:

• New traces are arriving (timestamps within last 15 minutes)
• Trace names match expected AI features
• No error status on traces

3b. Check Latency and Errors

From the trace list, examine:

• Average latency compared to baseline (is it significantly higher post-deploy?)
• Any traces with error status or missing generations
• Token usage anomalies (sudden spike could indicate prompt regression)

3c. Verify Prompt Versions

npx langfuse-cli api prompts list

Confirm that active prompt versions match what the deploy should be using (check the production label).

Decision criteria:

• Traces flowing, normal latency, correct prompt versions → PASS
• No new traces but deploy just happened → MONITOR (wait 5 more minutes)
• Error traces or latency spike → FAIL

---

Phase 4: Playwright Smoke Test

4a. Navigate to Production

playwright:browser_navigate
{
 "url": "<PRODUCTION_URL>"
}

Important: Apply the browser-anti-stall protocol:

• Set 15-second timeout expectations
• Skip browser_wait_for on navigation
• Use browser_snapshot to detect ready state

4b. Test Critical Paths

For each critical path identified in Phase 0f:

1. Navigate to the page
2. Snapshot to verify it loaded (not a blank page or error screen)
3. Check console for errors:

playwright:browser_console_messages
{}

4. Check network for failed requests:

playwright:browser_network_requests
{}

4c. Test Authentication (if applicable)

If the app has auth:

1. Navigate to login page
2. Verify login form renders
3. (If test credentials available) Log in and verify redirect to authenticated area

4d. Test Main Feature

Navigate to the primary feature and perform one basic interaction:

• If it's a form: fill and submit
• If it's a list: verify items render
• If it's a dashboard: verify data loads (not empty state when data should exist)
• If it's an AI feature: trigger one generation and verify response appears

4e. Take Evidence Screenshot

playwright:browser_take_screenshot
{}

Decision criteria:

• All critical paths load, no console errors, no 5xx network requests → PASS
• Minor visual glitches but functional → MONITOR
• Pages fail to load, 5xx errors, or auth broken → FAIL

---

Phase 5: Ship-or-Rollback Verdict

Aggregate all phase results into a final verdict.

Decision Matrix

Scenario	Verdict
All phases PASS	SHIP — Deploy is healthy
All phases PASS except 1-2 MONITOR items	MONITOR — Watch for 1 hour, re-check
Any phase FAIL with a quick fix available	HOTFIX — Apply fix and re-verify
Any critical phase FAIL (auth broken, data loss, 5xx on main feature)	ROLLBACK — Revert immediately
Seer identifies a simple root cause for the only FAIL	HOTFIX — Apply Seer's fix, re-verify

Report

═══════════════════════════════════════════════════════
 POST-DEPLOY VERIFICATION REPORT
 Project: <PROJECT_NAME>
 Deploy Time: <TIMESTAMP>
 Production URL: <URL>
 Checked At: <CHECK_TIMESTAMP>
═══════════════════════════════════════════════════════

## HEALTH CHECK RESULTS

| Check | Status | Details |
|------------------------------|---------|--------------------------------------|
| Sentry: new errors | ✅/❌ | [0 new / N new — list top issues] |
| Sentry: regressions | ✅/❌ | [0 regressed / N regressed] |
| Sentry: release tracked | ✅/❌ | [release ID and timestamp] |
| Supabase: migrations | ✅/❌ | [all applied / N pending/failed] |
| Supabase: API logs | ✅/❌ | [clean / N errors in last 30m] |
| Supabase: Edge Function logs | ✅/❌ | [clean / N errors] |
| Supabase: Auth logs | ✅/❌ | [clean / N errors] |
| Supabase: advisors | ✅/❌ | [no new warnings / N new warnings] |
| Langfuse: traces flowing | ✅/❌ | [N traces in last 15m / no traces] |
| Langfuse: latency | ✅/❌ | [normal / Xms above baseline] |
| Langfuse: prompt versions | ✅/❌ | [match expected / mismatch on N] |
| Smoke: home page | ✅/❌ | [loads in Xs / error] |
| Smoke: auth flow | ✅/❌ | [works / broken] |
| Smoke: main feature | ✅/❌ | [works / broken] |
| Smoke: console errors | ✅/❌ | [0 errors / N errors] |
| Smoke: network 5xx | ✅/❌ | [0 / N failed requests] |

## SEER ANALYSIS (if any P0 issues)

Issue: <ISSUE_ID>
Root Cause: <SEER_EXPLANATION>
Suggested Fix: <CODE_FIX>
Files Affected: <FILE_LIST>

## VERDICT

┌─────────────────────────────────────────────────┐
│ │
│ [SHIP / MONITOR / HOTFIX / ROLLBACK] │
│ │
│ Reason: <one-line explanation> │
│ │
│ Next action: <what to do now> │
│ │
└─────────────────────────────────────────────────┘

## MONITOR ESCALATION (if verdict is MONITOR)

Re-check in: [30 minutes / 1 hour]
Escalate to ROLLBACK if: [specific condition, e.g., error count exceeds 50]
Watch: [specific metrics to monitor]

═══════════════════════════════════════════════════════