Pipeline triage workflow
Ordered agent-style triage for a single failed Jenkins build - gather metadata, logs, classification, summaryFrom its SKILL.md
npx -y skills add coreos/ai-helpers --skill pipeline-triage-workflowAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 7 stars7 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
5.4 KB, as published. Nobody here has run it
Pipeline Triage Workflow (agentic)
Run one failed build through a fixed sequence of stages. Each stage has a single job and structured output for the next. Do not stop between stages for "what next?"
Inputs (required): JOB (Jenkins job name), BUILD (integer build number).
Related skills: pipeline-failures (deep patterns, kola analysis), pipeline-jira (COS ticket formatting).
Jenkins Job Hierarchy (know before you start)
| Job | Has downstream? | Analysis approach |
|---|---|---|
build | Yes → triggers build-arch per-arch | Check console for which arch failed, then analyze that build-arch child job |
build-arch | No (leaf job) | Analyze directly - kola tests run here |
build-node-image | No (independent pipeline) | Analyze directly - does NOT trigger build-arch |
Critical: build-node-image is a separate pipeline from build/build-arch. If investigating a build-node-image failure:
- Analyze its console log directly
- Do NOT search for "downstream"
build-archjobs - there are none - A
build-archjob running at the same time is coincidental, not related
Stream validation: Always verify streams match. A failure in build-node-image for stream 4.21-9.6 cannot be caused by build-arch for stream rhel-10.2.
Stage 1 — Gather (build metadata)
Agent role: Collect facts about the failing build.
Run:
coreos-tools jenkins builds info <JOB> <BUILD>
coreos-tools jenkins jobs info <JOB>
Output (write this block before Stage 2):
### Gather
- **Job:** …
- **Build:** …
- **Status / result:** …
- **Stream / parameters (if any):** …
- **URL:** …
Stage 2 — Logs (console evidence)
Agent role: Pull console log and extract high-signal lines.
Run:
coreos-tools jenkins builds log <JOB> <BUILD> | jq -r '.console_log[]' > /tmp/build.log
If the log is huge, capture tail:
coreos-tools jenkins builds log <JOB> <BUILD> | jq -r '.console_log[]' | tail -n 200
Output:
### Logs (excerpt)
- **Last ~N lines or key errors:** …
- **Patterns seen** (error / timeout / registry / test): …
Use grep patterns from pipeline-failures when analyzing saved log text (error:, FATAL:, timeout, unauthorized, FAILED, etc.).
Optional: If kola tests failed, add kola summary:
coreos-tools jenkins builds kola-failures <JOB> <BUILD>
Stage 3 — Classify
Agent role: Map the failure to a single primary category and note confidence.
Categories (pick one primary): infrastructure | flake | test_regression | package_change | registry_auth | tooling | unknown
Rules of thumb:
- Transient network / GitLab / "try rerun" language → flake or infrastructure
unauthorizedpulling images → registry_auth- Kola
rerun_failed: true(if you have kola output) → test_regression / package_change perpipeline-failures - Only flakes on rerun → do not treat as sole root cause; look for compose errors in log
Output:
### Classify
- **Primary:** …
- **Confidence:** low | medium | high
- **Why (1–3 bullets):** …
Stage 4 — Summarize (triage conclusion)
Agent role: Produce the handoff package for humans (and later Jira).
Output:
### Triage summary
- **Summary:** One paragraph describing the failure, root cause, and impact.
- **Evidence:** build URL, log pointers
- **Suggested next steps:** (e.g. rerun / open COS subtask / escalate to RHEL / snooze test)
- **Related downstream jobs to check:** e.g. `build-arch` if `build` failed (see `pipeline-failures`)
Stage 5 — Upstream Links (optional but recommended)
Agent role: Find relevant upstream references for the failure.
When to search:
- Test failures → find test source code
- Package changes → find Brew build, changelog
- cosa changes → find coreos-assembler commits
- Known issues → search for existing GitHub/GitLab issues
Load skills:
rhcos-repositories— repo locations and test pathsrhcos-brew— package build infopipeline-failures— hasgh searchcommands
Commands:
# Find test source (use rhcos-repositories to determine which repo)
gh search code "<test-name>" --repo coreos/coreos-assembler --repo openshift/os \
--json repository,path,url
# Search for related issues
gh search issues "<error-pattern>" --repo coreos/coreos-assembler --repo openshift/os \
--json repository,title,url,state
# Get Brew build info (if package change)
brew buildinfo <package-nvr>
Output:
### Upstream Links
- **Test source:** <url to test file>
- **Related issues:** <issue URLs or "None found">
- **Related PRs:** <PR URLs or "None found">
- **Package build:** <Brew URL if applicable>
Execution rules
- Complete Stages 1–5 in order in one run when the user provides
JOBandBUILD. - Ask for JOB and BUILD only if missing; do not ask "what next?" between stages.
- On CLI errors (auth, network), stop and report; do not invent build data.
- Use
coreos-tools jenkinscommands (Go CLI in the container).