Platform docs and culture

Platform Docs and Culture

Overview

DevOps is not CI/CD pipelines — it's collaboration, automation, shared responsibility, observability, and continuous improvement. The tools are easy; the operating model is the work. The goal of a platform/DevOps function is to make other teams fast and safe, not to be the team that does deployments for them.

Documentation is code's neighbor. ADRs, runbooks, and architecture docs live in the repo, versioned, reviewed, and updated in the same PRs that change behavior. A wiki page separate from the code rots; a doc next to the code ships.

Measure the process, not just the systems. DORA's four metrics tell you whether your DevOps practice is working: deploy frequency, lead time, change failure rate, time to restore.

When to Use

• Writing or reviewing Architecture Decision Records (ADRs)
• Writing runbooks for operational procedures
• Setting up DORA metrics measurement
• Designing self-service platform capabilities
• Improving documentation culture and knowledge sharing
• Evaluating whether to build a platform/IDP (Internal Developer Portal)

The Golden Path Philosophy

BAD:  DevOps team as a ticket queue
      "To deploy, open a ticket and wait 3 days"
      → bottleneck, learned helplessness, shadow IT

GOOD: Platform team as product team
      Paved, self-service paths: templates, pipelines, guardrails
      Teams deploy themselves inside the guardrails
      Platform's job: make the safe path the easy path

The platform team's customers are the developers. Success is measured in their speed and safety, not in the platform team's busyness.

Architecture Decision Records (ADRs)

Decisions that are expensive to reverse get an ADR. The format is deliberately small — one page, written while deciding, not after:

# ADR 0007: Use Argo CD for Kubernetes deployments

- Status: Accepted
- Date: 2026-07-20
- Deciders: platform team

## Context
Deployments currently run via `kubectl apply` from CI pipelines. No audit
trail of who deployed what; cluster credentials must live in CI; drift
between Git and cluster is discovered during incidents.

## Decision
Adopt Argo CD with a pull-based GitOps model: a config repo per team holds
desired state; Argo CD reconciles; CI updates image digests via PRs to the
config repo.

## Consequences
Positive: full audit trail in Git; no cluster credentials in CI; rollback
is `git revert`; drift self-heals.
Negative: new Tier-0 component to operate; team must learn the model;
UI-driven "quick sync" tempts bypassing Git (mitigate: disable in prod).
Alternatives rejected: Flux (comparable; Argo CD's UI wins for the team's
maturity), CI-push with kubectl (status quo — rejected, it's the problem).

Rules:

• ADRs are numbered, immutable once accepted (superseded by a new ADR, not edited), and stored in docs/adr/ in the repo.
• Write the ADR during the decision — the alternatives section is only honest if written before you commit.
• Not every decision needs one. Irreversible + expensive + contentious = ADR. Reversible + cheap = just do it.

Runbooks That Actually Work

A runbook is written for the 3 AM version of your teammate — tired, stressed, possibly unfamiliar with the service. Test: can a new team member execute it without asking questions?

# Runbook: Checkout error rate high

**Alert:** CheckoutErrorBudgetBurn | **Severity:** page

## 1. Confirm real user impact (2 min)
- Dashboard: https://grafana/d/checkout — error rate panel
- External check: https://status.example.com

## 2. Check for recent changes (2 min)
- Argo CD history: was there a deploy in the last hour? → If yes, ROLL BACK FIRST (step 4)
- Feature flags: any flag flipped recently?

## 3. Isolate the failure (5 min)
- Dependency health: DB connection pool saturation panel; Redis panel
- Logs: `loki: {service="checkout"} |= "error" | json` — group by error type

## 4. Mitigate
- Rollback: revert the deploy PR in deploy-config repo; Argo syncs in ~2 min
- If DB saturation: raise pool limit via flag `DB_POOL_SIZE`, restart rolling
- Escalate to @team-lead if not mitigated in 30 min

## 5. Aftermath
- Capture timeline notes → postmortem (see incident-response skill)

Runbook rules:

1. Linked from the alert itself (see observability-and-alerting)
2. Numbered steps with expected durations
3. The mitigation step is explicit and tested
4. Escalation path named
5. Updated after every incident that used it

DORA Metrics: Measuring the Practice

Metric	What it measures	Elite	How to improve it
Deployment frequency	Batch size, automation, confidence	On demand (multiple/day)	Smaller PRs, better pipelines, feature flags
Lead time for changes	Commit → production speed	< 1 hour	Remove queues, faster CI, fewer hand-offs
Change failure rate	Quality of what ships	0–15%	Better gates, progressive delivery, smaller changes
Time to restore	Incident response	< 1 hour	Rollback automation, observability, runbooks

Rules:

• Measure from pipeline/Git data — automated, not surveys.
• Track trends, not point scores; the goal is trajectory.
• Never use DORA metrics to compare individuals or punish teams. Metrics that become targets get gamed (Goodhart's law): change failure rate improves magically when failures stop being reported.

Knowledge Sharing Practices

• Code review as teaching: reviewers explain why, not just what.
• Pair on the scary stuff: production changes, incident response, new tooling.
• Rotation of responsibilities: on-call, release captain, Build Cop — knowledge silos dissolve when everyone rotates through the roles.
• Internal talks/demos: 30 minutes, monthly, someone shows something they built or broke.
• Bus factor audits: for each critical system, "if X is unavailable, who can operate it?" If the answer is nobody, that's a P1 risk, scheduled for fixing like any other.

When to Build a Portal (Backstage et al.)

An Internal Developer Portal (Backstage or similar) earns its complexity when:

• You have enough services that "who owns this, where are its docs, how do I deploy it" is a recurring question (~dozens of services)
• Golden-path templates exist and need a front door
• A service catalog would eliminate recurring toil

Before that scale: a well-organized repo of templates + good READMEs is the portal. A Backstage instance is a product you must maintain — don't adopt it to solve a 5-service documentation problem.

Common Rationalizations

Rationalization	Reality
"We're too busy to write docs"	You're busy because there are no docs — every question interrupts someone. Documentation is how the busy team buys back time.
"The code is self-documenting"	Code documents what. It never documents why — why this provider, why this pattern, what alternatives were rejected. That's what ADRs are for.
"DevOps will handle the deploy"	Then you're a ticket queue with a pager. Build the self-service path; the team's job is leverage, not labor.
"DORA metrics will tell us who's underperforming"	They measure the system's performance. A team with bad DORA numbers is usually a team fighting bad tooling, unclear requirements, or too much WIP. Fix the system.
"We need Backstage to be a real platform team"	You need paved paths people actually use. A portal without golden paths behind it is an empty lobby.
"Runbooks are for juniors"	Runbooks are for everyone at 3 AM. Experts write runbooks precisely because they know where the traps are.

Red Flags

• Deployments require filing a ticket to another team
• Critical systems with exactly one person who can operate them
• Documentation in a wiki last edited two years ago, contradicting the code
• Decisions made in chat, rationale lost forever
• DORA metrics used in performance reviews
• Alerts with no runbooks (symptom of the same disease)
• "Platform team" that spends 90% of its time doing tasks for other teams instead of eliminating them
• New hire takes weeks to make their first production change

Verification

• ADRs exist for all significant decisions, current, in the repo, with alternatives documented
• Every paging alert links a runbook; runbooks tested by someone who didn't write them
• DORA metrics collected automatically and reviewed monthly as trends
• Self-service paths exist for the top 3 recurring platform requests (new service, new env, new pipeline)
• Bus factor ≥ 2 for every critical system
• Documentation lives in repos, reviewed in PRs, updated with behavior changes
• On-call and release responsibilities rotate across the team
• A new team member shipped to production in their first two weeks (the ultimate platform-health test)