Going incident to fix

Incident to Fix

What this skill uses

Harness MCP tools: harness:harness_diagnose, harness:harness_execute, harness:harness_list, harness:harness_search Resource types: pipeline Traceable MCP tools: traceable-ai:execute_graphql_query, traceable-ai:get_api_insights_for_api_id, traceable-ai:get_graphql_query_template_details, traceable-ai:get_sample_spans_for_api_id, traceable-ai:get_threat_activity_timerange

Orchestrates the complete incident response lifecycle: from Traceable threat detection and investigation, through root cause analysis, to Harness pipeline hotfix deployment and Traceable verification. This skill chains security investigation with rapid remediation.

Instructions

Prerequisites

Collect: nature of incident (alert, anomaly, active attack), affected service/API, Harness org and project, repository and branch for the fix, Traceable environment. For active attacks, prioritize speed -- gather details as you progress.

---

Phase 1: Traceable Threat Detection and Investigation

Follow the /investigating-threats skill instructions for this phase.

First call traceable-ai:get_threat_activity_timerange (REQUIRED before any threatActivity queries). Then query threat activity for the affected service using traceable-ai:get_graphql_query_template_details (query_name: "threatActivity") and traceable-ai:execute_graphql_query. Identify: threat actor IPs/geolocation, attack patterns, affected endpoints, and request volume/timeline.

Follow the /responding-to-incidents skill instructions to scope the blast radius: how many APIs affected, lateral movement, data exposure, related anomalies. Use traceable-ai:get_api_insights_for_api_id on each affected API for risk decomposition.

Gather forensic evidence with traceable-ai:get_sample_spans_for_api_id -- collect sample traces showing exact payloads, response codes, and headers. Document all findings for the incident timeline.

---

Phase 2: Root Cause Analysis

Follow the /engineering-deep-dive skill instructions for technical analysis.

Step 2.1: Identify the vulnerability

Based on the threat investigation from Phase 1, determine the root cause:

• Is it a code-level vulnerability (injection, broken auth, SSRF)?
• Is it a misconfiguration (missing auth, overly permissive CORS)?
• Is it an infrastructure issue (exposed internal API, missing WAF rule)?

Use the Traceable /prioritizing-findings skill to check if this vulnerability was already known and ranked.

Step 2.2: Determine the fix

Based on the root cause:

• Code fix needed: Identify the specific file, function, and line. Determine which branch and repository to patch.
• Configuration fix: Identify the configuration change needed (API gateway rules, auth policy, etc.).
• Infrastructure fix: Identify the infrastructure change (network policy, WAF rule, etc.).

Step 2.3: Decision point

Present the root cause analysis to the user:

• Root cause description
• Affected component(s)
• Proposed fix approach
• Risk of the fix (blast radius of the code change)

Get user confirmation before proceeding to the hotfix deployment.

---

Phase 3: Harness Pipeline Hotfix Deployment

Follow the /running-pipelines skill instructions for execution and /debugging-pipelines for failure diagnosis.

Use harness:harness_search (query: "<service-name> deploy", resource_type: "pipeline") to find the relevant pipeline. Prefer pipelines tagged as hotfix or emergency. If multiple results, ask the user.

Execute with harness:harness_execute (resource_type: "pipeline", action: "run"), passing the hotfix branch and tag as inputs. Monitor aggressively -- use harness:harness_diagnose (resource_type: "pipeline", url=<execution URL>) on any failure and report immediately.

Decision point: Pipeline succeeded -- proceed to Phase 4. Pipeline failed -- suggest alternatives: fix and re-run, apply manual mitigation (WAF rule, feature flag via /managing-feature-flags, rate limit), or roll back to a known-good version.

---

Phase 4: Traceable Verification

Follow the /remediating-vulnerabilities skill instructions for this phase.

Step 4.1: Verify the fix is deployed

Wait a few minutes for the new deployment to receive traffic, then:

• Re-run the threat investigation query from Phase 1
• Check if the attack pattern is still succeeding against the patched endpoint
• Verify the vulnerable endpoint now returns appropriate error responses for malicious payloads

Step 4.2: Confirm posture improvement

Follow the /assessing-risk skill instructions. Compare the risk score before and after the fix:

• Has the overall risk score decreased?
• Is the specific vulnerability no longer present?
• Are there any new issues introduced by the fix?

Step 4.3: Check for residual exposure

Follow the /analyzing-auth-coverage skill instructions if the incident involved authentication gaps. Verify:

• The fixed endpoint now requires proper authentication
• No similar gaps exist in related endpoints
• Session handling is correct

---

Phase 5: Incident Report

Generate a comprehensive incident report:

## Security Incident Report

**Incident ID:** INC-<timestamp>
**Date:** <date>
**Severity:** CRITICAL / HIGH / MEDIUM
**Status:** RESOLVED / MITIGATED / MONITORING

### Timeline
| Time | Event |
|------|-------|
| <T0> | Threat detected / User reported incident |
| <T1> | Investigation started |
| <T2> | Root cause identified |
| <T3> | Hotfix deployed |
| <T4> | Fix verified |

### Threat Summary
- **Attack Type:** <e.g., SQL Injection, Credential Stuffing>
- **Threat Actors:** <IPs, geolocation, actor profile>
- **Affected APIs:** <list of endpoints>
- **Data Exposure:** <what data was potentially accessed>

### Root Cause
<description of the vulnerability and why it existed>

### Fix Applied
- **Pipeline:** <pipeline-id>
- **Execution:** <execution-id>
- **Change:** <brief description of the fix>
- **Branch:** <hotfix-branch>

### Verification
- **Attack Pattern Blocked:** YES / NO
- **Risk Score Change:** <before> -> <after>
- **Auth Coverage:** <before> -> <after>

### MTTR (Mean Time to Remediate)
<time from detection to verified fix>

### Recommendations
1. <prevent similar incidents>
2. <improve detection>
3. <strengthen defenses>

Examples

Example 1: Active API attack

User: "We're seeing a spike in 500 errors on the checkout API, security team thinks it's an attack"

What the AI does:

1. Calls traceable-ai:get_threat_activity_timerange, then queries threat activity for the checkout service
2. Finds SQL injection attempts from 3 IPs targeting /api/checkout/payment
3. Gets sample spans showing malicious payloads in the card_number field
4. Identifies the root cause: unsanitized input in the payment processing handler
5. Searches Harness for the checkout-service pipeline
6. Executes the pipeline with the hotfix branch containing input validation
7. After deployment, verifies the injection attempts now return 400 errors
8. Generates incident report with 47-minute MTTR

Example 2: Vulnerability needs hotfix

User: "Traceable found a critical auth bypass on our user API, we need to fix it now"

What the AI does:

1. Investigates the auth bypass using Traceable deep-dive on the user API
2. Finds that /api/users/{id}/profile accepts requests without a valid JWT
3. Root cause: middleware ordering issue where the auth check runs after the handler
4. Presents the analysis to the user, confirms the fix approach
5. Deploys the hotfix pipeline with the corrected middleware ordering
6. Runs analyzing-auth-coverage to confirm the endpoint now requires authentication
7. Reports: risk score improved from 82 to 34, auth coverage from 91% to 99%

Example 3: Pipeline failure during hotfix

User: "We have an active threat and the hotfix pipeline just failed"

What the AI does:

1. Runs harness:harness_diagnose on the failed execution
2. Identifies: Docker build failed due to a dependency resolution error
3. Reports the issue and suggests two paths:
   • Fix the Dockerfile and re-run (10-minute estimate)
   • Apply a WAF rule to block the attack pattern immediately while fixing the build
4. User chooses to do both. AI assists with the WAF recommendation and re-runs the pipeline
5. Both mitigations succeed. Generates the incident report noting the two-phase response

Related skills

Prerequisites (skills typically run before this one):

• /investigating-threats

Composes with (skills commonly invoked together):

• /engineering-deep-dive
• /responding-to-incidents
• /remediating-vulnerabilities
• /creating-pipelines
• /running-pipelines

Follow-ups (skills commonly chained after this one):

• /managing-pull-requests
• /qwiet-analyzing
• /briefing-ciso

Performance Notes

• During active incidents, speed matters. Start the threat investigation immediately rather than gathering all prerequisites upfront.
• Phase 1 (investigation) and Phase 3 (deployment) are the most time-sensitive. Keep communication concise during these phases.
• Traceable threat queries may return large result sets. Use time range filters and service filters to narrow results.
• Pipeline execution for hotfixes should use the fastest available deployment strategy. Recommend canary or rolling updates over blue-green for speed.
• Always verify the fix in Phase 4. A deployed but unverified fix is not a resolved incident.

Troubleshooting

Threat activity query returns no results

• Expand the time range -- the attack may have stopped or be intermittent
• Check the service name matches exactly in Traceable
• Verify the Traceable agent is reporting data from the affected environment
• Try querying by specific API endpoint URL instead of service name

Cannot find the hotfix pipeline in Harness

• Use harness:harness_search with broader keywords (service name only)
• Use harness:harness_list with resource_type: "pipeline" to see all pipelines in the project
• The user may need to create a hotfix pipeline -- follow /creating-pipelines skill

Hotfix pipeline fails on approval stage

• For incidents, approvals may need to be expedited
• Use harness:harness_execute with action: "approve" if the user has authority
• Suggest the user contact the approver directly for urgent incidents

Verification shows the fix did not work

• Double-check the deployed artifact matches the fix branch
• The vulnerability may have multiple entry points -- check related endpoints
• The Traceable agent may need a few minutes to reflect the new behavior
• Re-run the deep-dive analysis to check if the fix addressed the correct root cause

User reports a new attack during the fix process

• Prioritize containment: suggest immediate mitigations (WAF rules, rate limiting, feature flags)
• Continue the hotfix deployment in parallel with containment measures
• Update the incident report to reflect the evolving situation