Performance testing

是什么

这是一份性能测试规范，覆盖 k6 与 Locust（两款主流压测工具）脚本编写、SLO（服务等级目标）阈值判定、压测瓶颈诊断、CI（持续集成）流水线集成，让团队在大促或流量翻倍前提前压出系统极限，而不是在线上被流量打挂。

怎么用

1. 新接口上线前按本文档的压测三段式（基线、阶梯加压、长稳压测）跑一遍，确定容量底线。
2. 压测脚本里把 SLO 阈值写成判定逻辑，超出 P95 延迟或错误率自动失败，让流水线把把关。
3. 压测时配合可观测性大盘抓 CPU、内存、慢 SQL、连接池四类瓶颈，按文档诊断流程逐项定位。
4. 重大版本发布前必跑压测，结果归档到性能趋势库，对比上一版本看是否出现回退。
5. 压测环境数据规模必须和生产同量级，否则结论失真，按文档建议做数据脱敏后复制。

架构图

flowchart LR
    A[压测脚本] --> B[基线压测]
    B --> C[阶梯加压]
    C --> D[瓶颈定位]
    D --> E{SLO 达标?}
    E -->|否| F[回归优化]

Performance Testing

Load and performance testing validates that your system meets latency and throughput requirements under realistic and extreme traffic conditions.

When to Activate

• Load testing an API before a product launch
• Setting up k6 or Locust for a project
• Writing Go benchmark functions for critical code paths
• Defining SLO-based pass/fail thresholds for load tests
• Identifying bottlenecks under load (pool exhaustion, N+1, GC pressure)
• Adding performance regression detection to a CI/CD pipeline

Test Type Decision Table

Type	Description	Load shape	Goal	When to run
Load	Simulate expected traffic	Ramp to normal, hold	Verify baseline meets SLO	Pre-launch, nightly
Stress	Push beyond capacity	Ramp past normal	Find breaking point	Before scaling decisions
Soak	Sustained load over time	Constant for 1–4 hours	Detect memory leaks, pool exhaustion	Weekly
Spike	Sudden burst	0 → peak instantly	Test autoscaling, queue buffering	Before planned events
Volume	Large datasets, normal load	Normal rps, huge data	Find data-size bottlenecks	When data volume increases

k6

Script Structure

import http from 'k6/http';
import { check, sleep } from 'k6';
import { Rate, Trend } from 'k6/metrics';

const errorRate = new Rate('errors');
const paymentDuration = new Trend('payment_duration');

export const options = {
  stages: [
    { duration: '2m', target: 50 },   // ramp up
    { duration: '5m', target: 50 },   // hold
    { duration: '2m', target: 100 },  // ramp up further
    { duration: '5m', target: 100 },  // hold
    { duration: '2m', target: 0 },    // ramp down
  ],
  thresholds: {
    // SLO-based pass/fail: test fails if these are breached
    'http_req_duration': ['p(95)<500', 'p(99)<1000'],
    'http_req_failed':   ['rate<0.01'],
    'errors':            ['rate<0.05'],
  },
};

export default function () {
  const res = http.post(
    'https://api.example.com/payments',
    JSON.stringify({ amount: 100, currency: 'USD' }),
    {
      headers: {
        'Content-Type': 'application/json',
        Authorization: `Bearer ${__ENV.API_TOKEN}`,
      },
    }
  );

  const ok = check(res, {
    'status is 201':          (r) => r.status === 201,
    'response time < 500ms':  (r) => r.timings.duration < 500,
  });

  errorRate.add(!ok);
  paymentDuration.add(res.timings.duration);
  sleep(1);  // think time between requests
}

Scenarios (Mixed Workloads)

export const options = {
  scenarios: {
    browse: {
      executor: 'constant-vus',
      vus: 100,
      duration: '10m',
      exec: 'browseProducts',
    },
    checkout: {
      executor: 'ramping-arrival-rate',
      startRate: 10,
      timeUnit: '1s',
      stages: [{ duration: '5m', target: 50 }],
      preAllocatedVUs: 60,
      exec: 'checkout',
    },
  },
};

export function browseProducts() { /* ... */ }
export function checkout() { /* ... */ }

Running k6

k6 run script.js
k6 run --vus 100 --duration 10m script.js

# Export to InfluxDB + Grafana for dashboards
k6 run --out influxdb=http://localhost:8086/k6 script.js

# Cloud execution
k6 cloud script.js

Locust (Python)

from locust import HttpUser, task, between

class PaymentUser(HttpUser):
    wait_time = between(1, 3)

    def on_start(self):
        """Called once per VU — authenticate"""
        res = self.client.post('/auth/token', json={
            'email': '[email protected]',
            'password': 'password',
        })
        self.token = res.json()['access_token']

    @task(3)  # weight 3: 3× more frequent than weight-1 tasks
    def browse_products(self):
        with self.client.get(
            '/products',
            headers=self._auth(),
            name='/products',          # group dynamic URLs
            catch_response=True,
        ) as res:
            if res.status_code != 200:
                res.failure(f"Got {res.status_code}")

    @task(1)
    def create_payment(self):
        self.client.post(
            '/payments',
            json={'amount': 100},
            headers=self._auth(),
        )

    def _auth(self):
        return {'Authorization': f'Bearer {self.token}'}

# Headless CI mode
locust -f locustfile.py \
  --headless -u 100 -r 10 --run-time 5m \
  --host https://api.example.com \
  --csv results          # outputs results_stats.csv, results_failures.csv

Go Benchmarks

package payment_test

import (
    "fmt"
    "testing"
)

func BenchmarkProcessPayment(b *testing.B) {
    svc := NewPaymentService(testDB)
    b.ResetTimer()   // don't count setup time
    b.ReportAllocs() // show allocations/op in output

    for i := 0; i < b.N; i++ {
        _, err := svc.ProcessPayment(ctx, Payment{Amount: 100})
        if err != nil {
            b.Fatal(err)
        }
    }
}

// Sub-benchmarks for different scenarios
func BenchmarkProcessPayment_Sizes(b *testing.B) {
    for _, amount := range []float64{1, 100, 10_000} {
        b.Run(fmt.Sprintf("amount=%.0f", amount), func(b *testing.B) {
            for i := 0; i < b.N; i++ {
                svc.ProcessPayment(ctx, Payment{Amount: amount})
            }
        })
    }
}

# Run benchmarks
go test -bench=. -benchmem -benchtime=10s ./...
# Output: BenchmarkProcessPayment-8  50000  23456 ns/op  1024 B/op  12 allocs/op

# Compare before/after a change
go test -bench=. -count=10 -benchmem ./... > before.txt
# ... make the change ...
go test -bench=. -count=10 -benchmem ./... > after.txt
benchstat before.txt after.txt

SLO-Based Pass/Fail Criteria

Defining Thresholds from SLOs

Base thresholds on your production SLOs — not arbitrary numbers.

// If SLO: p99 < 500ms, error rate < 0.1%
thresholds: {
  'http_req_duration': ['p(50)<100', 'p(95)<300', 'p(99)<500'],
  'http_req_failed':   ['rate<0.001'],
}

Establishing a Baseline

1. Run load test against staging with production-like traffic shape
2. Record p50 / p95 / p99 and error rate
3. Set regression threshold: fail if p99 degrades > 20% from baseline
4. Set SLO threshold: fail if p99 exceeds SLO target

Bottleneck Identification Under Load

Symptom	Likely cause	How to confirm	Fix
Latency climbs with VU count	Connection pool exhausted	Check pool wait metric	Increase pool / add PgBouncer
Error spikes at N rps	Thread / goroutine limit	Check active connections	Tune concurrency config
Memory grows during soak	Memory leak / large cache	Heap profile during test	Fix leak, tune GC
High latency, low CPU	N+1 queries	Count DB queries per request	Add eager loading
CPU > 90%	Compute bottleneck	CPU flame graph	Optimize hot path, add cache
Latency spikes periodically	GC pause (JVM/Go)	GC log analysis	Tune GC, reduce allocations

CI Integration

When to Run

Type	Frequency	Trigger	Failure action
Smoke perf (5 VUs, 1 min)	Every PR	PR CI	Fail PR if p99 > 2× baseline
Full load test	Nightly	Cron	Alert on Slack
Stress test	Weekly	Cron	Report only

GitHub Actions Example

jobs:
  load-test:
    runs-on: ubuntu-latest
    if: github.event_name == 'schedule'
    steps:
      - uses: actions/checkout@v4

      - name: Run k6 load test
        uses: grafana/[email protected]
        with:
          filename: tests/load/payment.js
        env:
          API_TOKEN: ${{ secrets.LOAD_TEST_TOKEN }}
          K6_CLOUD_TOKEN: ${{ secrets.K6_CLOUD_TOKEN }}

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: k6-results-${{ github.run_id }}
          path: results/

See also: performance, observability, ci-cd

Red Flags

• Symmetric ramp-up/ramp-down without a sustained plateau — spike-then-ramp-down misses memory leaks and GC pressure; hold at target RPS for ≥10 min in steady state
• Asserting only on HTTP 200 — a cached error page or open circuit breaker returns 200; use check() to assert on specific response body fields, not just the status code
• Single load generator machine for high VU counts — one machine saturates its NIC before the target; use distributed execution (k6 cloud, multiple Locust workers) above ~500 VUs
• No baseline before the test — without a pre-change baseline you can't tell whether 300ms p99 is a regression or always was that way
• Load test traffic escaping into production — test traffic that bypasses rate limits can trigger real customer alerts; isolate by dedicated API key, IP allowlist, or a separate environment
• Zero think time between requests — real users pause between actions; 0ms think time inflates effective concurrency 5–10×, producing false bottlenecks that don't exist in production
• Setting SLO thresholds from the first test run — first-run numbers are noisy; run 3+ tests under stable conditions before codifying a regression threshold

Checklist

• Test type chosen (load/stress/soak/spike) matches the specific question being answered
• k6 / Locust thresholds tied to SLO values — not made-up numbers
• Baseline measured before setting regression thresholds
• Test users and data isolated from production
• Think time (sleep) included in VU scripts for realistic simulation
• k6 check() used for per-request assertions (not just global thresholds)
• Go benchmarks include b.ReportAllocs() and b.ResetTimer()
• benchstat used to compare before/after for Go performance changes
• Bottleneck identification checklist followed when tests fail
• Load test results stored as CI artifacts for trending over time