agentsclimarketplace

Monitoring

Skill ryukyagamilight/terminal-skills/devops/monitoring

🖥️ Discover and master essential terminal skills for Server, Kubernetes, DevOps, and Cloud Computing with this curated collection.

Install
npx -y skills add ryukyagamilight/terminal-skills --skill monitoring

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

监控与告警

SKILL.md

7.8 KB, ~2.3k tokens by cl100k_base, as published. Nobody here has run it

监控与告警

概述

Prometheus、Grafana、告警规则配置等技能。

Prometheus

基础查询（PromQL）

# 即时向量
http_requests_total
http_requests_total{job="api", status="200"}

# 范围向量
http_requests_total[5m]

# 偏移
http_requests_total offset 1h

# 聚合
sum(http_requests_total)
sum by (job) (http_requests_total)
sum without (instance) (http_requests_total)

# 速率
rate(http_requests_total[5m])
irate(http_requests_total[5m])

# 增量
increase(http_requests_total[1h])

# 直方图分位数
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

常用查询

# CPU 使用率
100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# 内存使用率
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

# 磁盘使用率
(1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100

# 网络流量
rate(node_network_receive_bytes_total[5m])
rate(node_network_transmit_bytes_total[5m])

# HTTP 请求速率
sum(rate(http_requests_total[5m])) by (status)

# 错误率
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

# 延迟 P99
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

配置文件

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

rule_files:
  - "rules/*.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets: ['node1:9100', 'node2:9100']

  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true

告警规则

# rules/alerts.yml
groups:
  - name: node
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is {{ $value }}%"

      - alert: HighMemoryUsage
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"

      - alert: DiskSpaceLow
        expr: (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 85
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Disk space low on {{ $labels.instance }}"

  - name: application
    rules:
      - alert: HighErrorRate
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate"
          description: "Error rate is {{ $value | humanizePercentage }}"

      - alert: HighLatency
        expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High latency"

Alertmanager

配置

# alertmanager.yml
global:
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: '[email protected]'
  smtp_auth_username: '[email protected]'
  smtp_auth_password: 'password'

route:
  group_by: ['alertname', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty'
    - match:
        severity: warning
      receiver: 'slack'

receivers:
  - name: 'default'
    email_configs:
      - to: '[email protected]'

  - name: 'slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/xxx'
        channel: '#alerts'
        title: '{{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'

  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: 'xxx'

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'instance']

Grafana

数据源配置

# provisioning/datasources/prometheus.yml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

Dashboard JSON 示例

{
  "dashboard": {
    "title": "Node Metrics",
    "panels": [
      {
        "title": "CPU Usage",
        "type": "graph",
        "targets": [
          {
            "expr": "100 - (avg by (instance) (irate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)",
            "legendFormat": "{{ instance }}"
          }
        ]
      },
      {
        "title": "Memory Usage",
        "type": "gauge",
        "targets": [
          {
            "expr": "(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100"
          }
        ]
      }
    ]
  }
}

常用面板查询

# CPU 使用率（时间序列）
100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# 内存使用（仪表盘）
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

# 请求速率（柱状图）
sum(rate(http_requests_total[5m])) by (status)

# 延迟热力图
sum(rate(http_request_duration_seconds_bucket[5m])) by (le)

常见场景

场景 1：Kubernetes 监控

# ServiceMonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: app-monitor
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
    - port: metrics
      interval: 15s
      path: /metrics

场景 2：自定义指标

# Python 应用
from prometheus_client import Counter, Histogram, start_http_server

REQUEST_COUNT = Counter('http_requests_total', 'Total HTTP requests', ['method', 'endpoint', 'status'])
REQUEST_LATENCY = Histogram('http_request_duration_seconds', 'HTTP request latency', ['method', 'endpoint'])

@REQUEST_LATENCY.labels(method='GET', endpoint='/api').time()
def handle_request():
    REQUEST_COUNT.labels(method='GET', endpoint='/api', status='200').inc()
    # ...

start_http_server(8000)

场景 3：SLO 监控

# 可用性 SLO (99.9%)
1 - (sum(rate(http_requests_total{status=~"5.."}[30d])) / sum(rate(http_requests_total[30d])))

# 错误预算消耗
(1 - (sum(rate(http_requests_total{status=~"5.."}[7d])) / sum(rate(http_requests_total[7d])))) / 0.999

# 延迟 SLO (P99 < 500ms)
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[30d])) by (le)) < 0.5

场景 4：告警静默

# 创建静默
amtool silence add alertname=HighCPUUsage instance=node1 --duration=2h --comment="Maintenance"

# 查看静默
amtool silence query

# 删除静默
amtool silence expire <silence-id>

故障排查

问题排查方法
指标缺失检查 scrape 配置、target 状态
告警不触发检查规则语法、Alertmanager 配置
查询慢优化 PromQL、增加采样间隔
存储满调整 retention、清理旧数据
# 检查 Prometheus targets
curl http://prometheus:9090/api/v1/targets

# 检查告警规则
curl http://prometheus:9090/api/v1/rules

# 检查 Alertmanager 状态
curl http://alertmanager:9093/api/v1/status

# 测试 PromQL
curl 'http://prometheus:9090/api/v1/query?query=up'

Gives 0 of the 12 instructions most monitoring observability skills give in ~2.3k tokens

Counted across 481 of the 483 authors here whose files we hold, read 2026-08-06

  • link every alert to a runbookin 43 of 481, across 35 files
  • use structured json loggingin 36 of 481, across 31 files
  • alert on user-facing symptomsin 20 of 481, across 15 files
  • emit structured JSON logs with stable event namesin 18 of 481, across 13 files
  • propagate trace context across boundariesin 16 of 481
  • use histograms for latency trackingin 14 of 481, across 9 files
  • use OpenTelemetry for distributed tracingin 13 of 481, across 8 files
  • include a correlation ID on every log linein 13 of 481, across 8 files
  • Define service level objectivesin 10 of 481, across 7 files
  • Call useAzureMonitor before importing other modulesin 9 of 481, across 2 files
  • stop and ask for clarification if inputs are missingin 9 of 481, across 2 files
  • define on-call questions before adding telemetryin 9 of 481, across 4 files

Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once. Length counted with cl100k_base; the agent that loads this file may tokenize it differently.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.