Ollygarden otel collector k8s daemonset
Skill ollygarden/skills/skills/ollygarden-otel-collector-k8s-daemonset
OllyGarden's opinionated, optimization-first OpenTelemetry Collector configuration for a Kubernetes node agent (DaemonSet). Use when authoring or reviewing a node-level/agent collector config for logs, metrics, and traces on Kubernetes, or when the user wants to reduce telemetry volume, cost, cardinality, or noise at collection time. Triggers on "collector daemonset config", "node agent collector", "otel collector on kubernetes", "reduce telemetry cost in the collector", "tune kubeletstats/hostmetrics/filelog", "drop noisy spans/logs/metrics in the collector".From its SKILL.md
npx -y skills add ollygarden/skills --skill ollygarden-otel-collector-k8s-daemonsetAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its file declares
Copied from the file, not written here
The file declares its own license as Apache-2.0. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.
SKILL.md
6.4 KB, as published. Nobody here has run it
Opinionated OTel Collector — Kubernetes DaemonSet
Use this skill for the agent tier: one collector per node ingesting node-local OTLP,
kubelet/host metrics, same-node Prometheus targets, and pod logs. Use a separate gateway or
cluster Deployment for tail_sampling, load_balancing, k8s_cluster, and k8s_events.
This repository owns OllyGarden's decisions, not component facts. Consult otel-collector for
current component keys, defaults, and stability; otel-ottl for syntax; and the relevant
public OpenTelemetry language or instrumentation skill for source-side fixes.
Non-negotiable pipeline contract
Apply these to traces, metrics, and logs:
- Put
memory_limiterfirst so backpressure happens before downstream buffering. - Enrich with real identity; do not fabricate it. Scope
k8s_attributesto the local node withfilter.node_from_env_var: K8S_NODE_NAME, disable the system detector'shost.name, and verify detector order/overrideagainst the pinned Collector distribution. - Put resource-value truncation last among transforms. Kubernetes metadata can otherwise inflate every record.
- Persist
file_logoffsets withfile_storageon host-backed storage. Container-local storage loses offsets when the pod is recreated.
The OTLP listener binding alone does not make ingest node-local. The DaemonSet deployment must route each workload to the agent on its own node. Confirm the networking topology before claiming the agent/gateway boundary holds.
Metrics: reduce series and cadence
Metric cost is series count × datapoints per minute. The references implement these decisions:
- Curate
kubeletstatsandhost_metrics; prefer bounded utilization measurements over redundant per-state series. - Split fast and slow groups into disjoint receiver instances. Keep container/pod/node metrics at 20s, volume metrics at 60s, CPU/memory at 60s, and filesystem at 300s. Do not slow a single receiver globally and lose useful CPU/memory resolution.
- Drop read-only
secret,configMap,downwardAPI, and projected service-account-token volume metrics. RetainemptyDir: its writable usage is a disk-pressure signal. - Scrape each pod only from the agent on its node using
field: spec.nodeName=${env:K8S_NODE_NAME}; use a separate slow scrape for expensive endpoints and discard terminal pods. - Filter to monitored namespaces.
The preferred DPM pattern is separate receivers over disjoint subsets. For SDK-set OTLP cadence or
receivers without a partition knob, consult the public otel-collector routing connector and
interval processor references. Preserve unmatched passthrough and account for empty emissions.
Logs: cap and scope
- Cap individual pod-log records (
max_log_size: 100KiB) and exclude the collector's own logs. - Drop low-value severities only after structured records have a usable
severity_number. - Deduplicate only known-chatty services. Immediately before scoped
log_dedup, removelog.file.pathandlog.file.record_numberunder the same condition or those changing offsets defeat the hash. The shipped YAML deliberately does not enable dedup without a service-specific scope; consult theotel-collectorlog_dedupreference before adding it. - Fix telemetry values embedded in message text at the application rather than relying on brittle Collector regexes.
Traces: deterministic noise only
- Drop probe spans using the bounded route/path/name patterns in
references/traces.yaml; the filters cover current and legacy HTTP attributes plus framework handler names. Keep regexes anchored. - Prefer source-side suppression for static assets following the relevant public OpenTelemetry instrumentation guidance; use the Collector filter as a portable fallback.
- Do not probabilistically head-sample at the agent for cost. Keep the agent lossless except for
reviewed deterministic noise filters; whole-trace reduction requires gateway
tail_samplingbehindload_balancing.
Self-monitoring
Use detailed internal telemetry at a modest reporting interval, with views dropping the noisiest high-cardinality internal series. Retain queue, refusal, and export-failure visibility so savings do not hide an unhealthy collector.
Reference configuration
Copy the full set and search for CUSTOMIZE:
references/common.yaml— shared receiver, processors, exporter, state, and self-telemetry.references/traces.yaml,metrics.yaml,logs.yaml— one complete signal pipeline each.references/prometheus/*.yaml— bare scrape-job fragments included bymetrics.yaml.
Read references/decomposing-config.md before editing. Processor arrays replace rather than merge,
and ${file:} paths depend on the Collector working directory.
Verify before shipping
Complete every gate below; a parser-only or single-fragment check is not verification:
- Obtain
common.yaml,traces.yaml,metrics.yaml, andlogs.yaml; if one is missing, stop and request it. Fromreferences/, validate all four together against the pinned distribution. - Supply non-secret synthetic
K8S_NODE_NAME,K8S_CLUSTER_NAME, and exporter endpoint values, then inspectprint-configoutput for all pipelines, processor order, and included scrape jobs. - Use sanitized positive and near-miss telemetry to prove each filter drops only its intended target. Never use production ingest/export endpoints for verification.
Follow references/validating.md for the merged commands, off-cluster failure classification, and
version limits.
Handoffs
- Component configuration and OTTL:
otel-collector,otel-ottl. - Generic deep-merge mechanics:
ollygarden-otel-collector-config-decomposition. - Source-side telemetry design and suppression: the relevant public OpenTelemetry language and instrumentation skills.
What ships with it: 15 files
31.1 KB alongside SKILL.md
evals/
references/
- common.yaml5.3 KB
- decomposing-config.md1.3 KB
- logs.yaml2.2 KB
- metrics.yaml7.0 KB
- prometheus/scrape-kube-state-metrics.yaml1.3 KB
- prometheus/scrape-pods-slow.yaml994 B
- prometheus/scrape-pods.yaml1.2 KB
- traces.yaml3.1 KB
- validating.md2.7 KB
Gives 0 of the 12 instructions most project setup skills give
Counted across 1,553 of the 3,091 authors here whose files we hold, read 2026-09-06
- Write the configuration filein 36 of 1553
- Create the directory structurein 35 of 1553, across 33 files
- Verify the setupin 31 of 1553, across 28 files
- Run the setup scriptin 30 of 1553, across 29 files
- Pre-determine the required sample sizein 29 of 1553, across 12 files
- Check if the configuration already existsin 29 of 1553
- Document every testin 26 of 1553, across 10 files
- Start with a hypothesisin 26 of 1553, across 11 files
- Ask one question at a timein 22 of 1553
- Test a single variable per testin 21 of 1553, across 9 files
- Read product marketing context before asking questionsin 19 of 1553, across 8 files
- Do not peek and stop earlyin 18 of 1553, across 7 files
Said here and by no other author read
- Put memory limiter first in pipelines
- Scope k8s attributes to local node
- Truncate resource values last
- Persist file log offsets with storage
- Split fast and slow metric groups
- Drop read-only volume metrics
Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once.