Snmp exporter

snmp_exporter best practices

snmp_exporter is a multi-target proxy: one central instance walks SNMP on many devices; Prometheus tells it what to scrape per request via URL params (/snmp?target=<ip>&module=<m1,m2>&auth=<name>). Everything below assumes v0.30.x. The generator and exporter versions must match — a config generated by a different version can fail to load. Pin both in CI.

This skill ends where the metrics land in Prometheus. Querying them, writing the PromQL, and building the dashboards or alerts on top belong to the prometheus-mimir-grafana skill (same observability plugin); shipping them to long-term storage is mimir-upgrade territory.

Golden rules

1. Never walk a whole vendor enterprise subtree. Full-tree walks are the #1 cause of multi-minute scrapes, device-CPU exhaustion, and even crashed SNMP agents (Cisco documents this for CBS; iDRAC full walks take 2–5 min; Onyx serves all management through one easily-overloaded daemon). Walk the specific tables worth alerting on — a curated module is typically 10–100× smaller. Measured here: Cisco CBS full-tree = 4034 metrics vs curated = 63.
2. Compose modules per scrape instead of duplicating walks. Since v0.24 one scrape can request module=if_mib,vendor_health. Keep vendor modules health-only and reuse the stock if_mib, system, hrDevice, hrStorage modules for interfaces/uptime/CPU/memory. sysUpTime lives in system (moved out of if_mib in v0.28) — include it when alerting on reboots.
3. Do the timeout math. Per-request worst case = timeout × (retries+1) = 20 s at defaults (5s × 4), which silently exceeds Prometheus's default 10 s scrape_timeout. The exporter does NOT honor the scrape-timeout header — Prometheus just hangs up and the walk is cancelled, yielding up=0 with no partial data. Set scrape_timeout above the observed snmp_scrape_walk_duration_seconds, and keep timeout×(retries+1) below it.
4. Prefer v3 authPriv everywhere; v2c only as fallback. SNMP v1 forces GETNEXT (one row per round trip) and can't read 64-bit counters — never use it. v2c community strings are plaintext on the wire: acceptable only on an access-controlled management network. Modern gear (iDRAC 9/10, Cisco CBS, Onyx) all support v3 with SHA-2 + AES — use SHA256 + AES-128 as the default pair, and match auth_protocol exactly to how the device user was created. BMCs also gate some data (e.g. iDRAC lifecycle logs) behind v3.
5. Keep credentials out of git. Generated snmp.yml embeds secrets in cleartext. Either use ${ENV_VAR} in v3 username/password/priv_password with --config.expand-environment-variables (only those three fields), or split an auths-only yml into a Secret and load it as an extra --config.file (repeatable + glob since v0.24; duplicate auth/module names across files are rejected — never reuse the stock names like public_v2).
6. Protect port 9116. Anyone who can reach /snmp can walk arbitrary targets with the configured credentials (SSRF + credential relay). Firewall/NetworkPolicy it to Prometheus, optionally add exporter-toolkit TLS
   • basic auth (--web.config.file), and set device-side SNMP source ACLs.

Writing modules (generator workflow)

Layout: one directory per device family with its own mibs/ + generator.yml to avoid MIB name collisions; generate one snmp.yml per family and load them all at runtime (no merge needed). MIB sources: make mibs for the curated upstream set; github.com/prometheus-community/snmp for ready-made vendor configs; librenms/librenms mibs/ as the broadest mirror when the vendor hides MIBs behind a login. Patch unparseable MIBs in a mib-patches/ dir rather than hand-editing generated output.

modules:
  my_device:
    walk:
      - MY-MIB::usefulTable          # ALWAYS prefer MIB::name over bare OIDs
      - 1.3.6.1.4.1.X.Y.Z            # numeric: last resort, see pitfall below
    lookups:                         # turn numeric indexes into readable labels
      - source_indexes: [ifIndex]
        lookup: IF-MIB::ifName
    overrides:
      ifName: {ignore: true}         # don't also emit lookup columns as metrics
    timeout: 5s                      # per-PDU; worst case ×(retries+1)
    retries: 2
    max_repetitions: 25              # raise for fast agents, LOWER for buggy ones

• Numeric-OID pitfall: numeric subtree walks can silently resolve to nothing when the MIB directory contains overlapping arc definitions — generation "succeeds" with the metrics simply absent. Always diff the generated file for the expected metric names before shipping it.
• A walk entry that is a single instance (e.g. ...1.8.0) becomes a cheap GET.
• filters.static (generation-time, fixed indices → GETs) and filters.dynamic (runtime, e.g. only ifAdminStatus=up interfaces) trim big tables at the device, unlike Prometheus metric_relabel_configs which only saves TSDB.
• EnumAsInfo for inventory-ish enums; EnumAsStateSet only for low-cardinality alertable states (it emits one series per possible value).
• Strings render as gauge=1 with the value in a label; use regex_extracts to turn status strings into real 0/1 gauges.
• Tables indexed by non-renderable types (DateAndTime, Bits) are rejected after v0.30.1 (#1653) — override the index type: OctetString to keep such a table.
• Generate with --fail-on-parse-errors first; inspect generator parse_errors. Using --no-fail-on-parse-errors is fine when the remaining errors are all in MIB files the walks never touch (common with large vendor MIB dumps).
• No net-snmp headers locally? Use the container: docker run --rm -v "$PWD:/opt/" quay.io/prometheus/snmp-generator:v0.30.1 generate -m /opt/mibs -g /opt/generator.yml -o /opt/snmp.yml

Running it

snmp_exporter \
  --config.file=/etc/snmp_exporter/snmp.yml \   # stock modules (if_mib, system, hr*)
  --config.file=/config/snmp-custom.yml \        # generated custom modules
  --config.file=/secrets/auths.yml \             # credentials, from a Secret
  --config.expand-environment-variables

• Validate in CI with --dry-run; reload via SIGHUP or POST /-/reload; liveness on /-/healthy; inspect effective config at /config.
• Watch the exporter's own /metrics plus the per-scrape series every target returns: snmp_scrape_walk_duration_seconds (tune scrape_timeout against this), snmp_scrape_packets_retried (alert if ratio to packets_sent exceeds a few percent — it's the packet-loss signal), snmp_scrape_pdus_returned. Liveness of the device is the job's up; there is no snmp_up metric.
• One instance comfortably handles thousands of targets; scale with plain replicas behind one Service (stateless). --snmp.module-concurrency only parallelizes modules within one scrape — raise cautiously, devices rate-limit.

Kubernetes

Read references/kubernetes.md before deploying: Helm chart values, Probe CRD (params support needs prometheus-operator ≥ v0.85.0) vs ScrapeConfig relabeling, Secret-based auth wiring, UDP/161 NetworkPolicy, and stable-source-IP options for device ACLs.

Local testing & debugging

docker run --rm -p 9116:9116 -v "$PWD/snmp.yml:/etc/snmp_exporter/snmp.yml" \
  quay.io/prometheus/snmp-exporter
curl 'localhost:9116/snmp?target=192.0.2.10&auth=public_v2&module=if_mib'

Failure-mode table — check these before blaming the exporter:

Symptom	Likely cause / fix
"request timeout (after N retries)"	Wrong community/ACL/unreachable — SNMP silently drops bad-community requests, so auth errors LOOK like timeouts. Verify with snmpbulkwalk -v2c -c <comm> <ip> system from the same host/pod.
Scrape works but empty/few metrics	Wrong module for the device (it doesn't implement those OIDs), or the community's SNMP view excludes the private tree.
"OID not increasing"	Buggy agent — set allow_nonincreasing_oids: true.
Replies ignored on multi-homed device	Device answers from a different source IP — use_unconnected_udp_socket: true.
up=0 only under Prometheus, curl works	scrape_timeout too low — walk cancelled mid-flight; check snmp_scrape_walk_duration_seconds via curl timing.
Config load: "invalid index type"	Post-v0.30.1 rejection of non-renderable index types — override to OctetString or drop the table.
Auths/modules parse error on upgrade	v0.23 auth split / version mismatch — regenerate with the matching generator.

Debug tracing: --log.level=debug logs per-subtree walk durations (find the slow subtree); add &snmp_debug_packets=true to a scrape URL for full packet traces (requires debug log level; there is no debug= param). For CI without hardware, replay a recorded snmpwalk with snmpsim.

Device references

Read the matching file before writing or modifying a module for these families — they carry verified OIDs, quirks, and the rationale for what NOT to walk:

Device	Reference
Dell iDRAC 9/10 (PowerEdge BMC)	references/idrac.md
Cisco CBS250/350, Catalyst 1200/1300	references/cisco-cbs.md
NVIDIA/Mellanox Onyx (Spectrum, SN2010…)	references/mellanox-onyx.md
Ubiquiti UniFi (switches, APs)	references/ubiquiti-unifi.md — live-tested; stock modules suffice

Working generator configs for all of the above (curated walks, lookups, v3 auth blocks with ${ENV} placeholders, plus a vendor-neutral lldp mis-cabling watchdog module) are in examples/ — generator-custom.yml is the combined production config; the per-vendor files are for iterating on one family. An opinionated alert pack for these modules is examples/alerts.yml. The stack around the exporter — traps (Telegraf snmp_trap → Prometheus), ICMP discrimination (blackbox), NetBox-driven service discovery, Dell OME for firmware, CI fixtures — is references/companions.md: adopt those, don't rebuild them.

Backlog — open items to re-check

Time-sensitive or hardware-gated items; resolve and update the relevant reference file when done. Dated provenance for every external claim lives in references/sources.md — bump its Last verified: dates when re-checking; the full research narrative behind these modules is references/research-report.md (background only — never needed to apply the skill):

1. idrac_exporter iDRAC 10 fix — watch mrlhansen/idrac_exporter#202 (power/fan/network metrics empty on 17G; fix committed but unreleased as of v2.6.1). Until a release ships, don't deploy the Redfish layer against iDRAC 10. Re-check: gh release list -R mrlhansen/idrac_exporter.
2. Hardware snmpwalks (one session with real gear settles all): Onyx SN2010 — hrProcessorLoad/hrStorageTable answer? EtherLike dot3Stats? DAC rows in entPhysicalSerialNum? CBS — which env table answers (.101.83 vs .101.53.15) on owned models; confirm DAC serial absence; do pethMainPseTable + etherStats* answer, and does etherStatsIndex == ifIndex; EtherLike dot3Stats? iDRAC 10 — v2c gets work once enabled? AES256 vs AES256C pairing? Walk bugs (virtualDiskTable skip) on current firmware?
3. Next snmp_exporter release (>0.30.1) ships the non-renderable-index rejection (#1653) — re-validate hand-written configs on upgrade; regenerate with the matching generator.
4. prometheus-operator ScrapeConfig v1beta1 graduation — update references/kubernetes.md examples when it ships.
5. Contribute the cisco_cbs module to prometheus-community/snmp — no public CBS module existed as of 2026-07; ours is novel.

When NOT to use snmp_exporter

• OS that permits installing software → node_exporter.
• Modern Dell/HPE/Lenovo BMCs → idrac_exporter (Redfish, mrlhansen) is richer and avoids MIB curation; keep SNMP for old BMCs or trap-only needs.
• SNMP traps → snmp_exporter doesn't receive them; that needs a separate receiver (e.g. Telegraf's snmp_trap input; note maxwo/snmp_notifier is the REVERSE direction — Alertmanager alerts out as traps — see references/companions.md). Don't promise trap-based alerting from this exporter.