Groq observability

Groq Observability

Overview

Monitor Groq LPU inference for latency, token throughput, rate limit utilization, and cost. Groq's defining advantage is speed (280-560 tok/s), so latency degradation is the highest-priority signal. The API returns rich timing metadata (queue_time, prompt_time, completion_time) and rate limit headers on every response.

Prerequisites

• A Groq account with an API key exported as the GROQ_API_KEY environment variable — the groq-sdk client reads it automatically (new Groq()).
• Node.js with groq-sdk and prom-client installed (npm install groq-sdk prom-client).
• A Prometheus scrape target and (optionally) Grafana for the dashboard panels.

Key Metrics to Track

Metric	Type	Source	Why
TTFT (time to first token)	Histogram	Client-side timing	Groq's main value prop
Tokens/second	Gauge	usage.completion_time	Throughput degradation
Total latency	Histogram	Client-side timing	End-to-end performance
Rate limit remaining	Gauge	x-ratelimit-remaining-* headers	Prevent 429s
Token usage	Counter	usage.total_tokens	Cost attribution
Error rate by code	Counter	Error handler	Availability
Estimated cost	Counter	Tokens * model price	Budget tracking

Instructions

Apply these six steps in order. Steps 1-2 are the core instrumentation loop — wrap the client, then feed a Prometheus instrument set from each call. Steps 3-6 add rate-limit tracking, alerting, structured logs, and dashboards on top. The lean client skeleton is below; the full code for every step lives in references/implementation.md.

1. Instrumented client — wrap groq.chat.completions.create so latency, tokens, queue time, and estimated cost are captured on the same path as the request (trackedCompletion).
2. Prometheus metrics — register a histogram (latency), counters (tokens, cost, errors), and gauges (throughput, rate-limit remaining), then feed them from emitMetrics.
3. Rate limit header tracking — parse x-ratelimit-remaining-* off every response into a gauge so you alert before a 429, not after.
4. Prometheus alert rules — ship latency/rate-limit/throughput/error/cost alerts tuned to Groq's sub-200ms, 280+ tok/s baseline.
5. Structured request logging — emit one JSON line per request for log aggregation, preserving per-request detail metrics roll up.
6. Dashboard panels — TTFT distribution, tokens/sec, rate-limit utilization, request volume, error rate, cost, and queue time.

import Groq from "groq-sdk";

const groq = new Groq(); // reads GROQ_API_KEY

async function trackedCompletion(model: string, messages: any[]) {
  const start = performance.now();
  const result = await groq.chat.completions.create({ model, messages });
  const latencyMs = performance.now() - start;
  const usage = result.usage!;
  const metrics = {
    model,
    latencyMs: Math.round(latencyMs),
    tokensPerSec: Math.round(usage.completion_tokens / ((usage as any).completion_time || latencyMs / 1000)),
    totalTokens: usage.total_tokens,
  };
  emitMetrics(metrics); // -> Prometheus (Step 2)
  return { result, metrics };
}

See references/implementation.md for the complete GroqMetrics shape, pricing table, Prometheus instruments, rate-limit tracking, alert rules, structured logging, and dashboard panel list.

Output

Applying the workflow produces:

• A trackedCompletion wrapper that returns { result, metrics }, where metrics is a GroqMetrics object (latency, TTFT, tokens/sec, token counts, queue time, estimated cost).
• A Prometheus metric set — groq_latency_ms (histogram), groq_tokens_total / groq_cost_usd / groq_errors_total (counters), and groq_tokens_per_second / groq_ratelimit_remaining (gauges).
• Five alert rules (GroqLatencyHigh, GroqRateLimitCritical, GroqThroughputDrop, GroqErrorRateHigh, GroqCostSpike).
• A structured JSON log line per request and a 7-panel dashboard spec.

Examples

Instrument a single completion and emit a structured log line:

const { result, metrics } = await trackedCompletion(
  "llama-3.3-70b-versatile",
  [{ role: "user", content: "Summarize this incident report in two sentences." }]
);
logGroqRequest(metrics, result.id);
// metrics.tokensPerSec -> 310, metrics.estimatedCostUsd -> 0.000404

For a 429-guard using rate-limit headers and a dashboard health-reading table, see references/examples.md.

Error Handling

Issue	Cause	Solution
429 with high retry-after	RPM or TPM exhausted	Implement request queuing
Latency spike > 2s	Model overloaded or large prompt	Reduce prompt size or switch to lighter model
503 Service Unavailable	Groq capacity issue	Enable fallback to alternative provider
Tokens/sec drop	Streaming disabled or large prompts	Enable streaming for better perceived performance

Resources

• references/implementation.md — full code for all six observability steps.
• references/examples.md — worked instrumentation, 429-guard, and dashboard-reading examples.
• Groq API Reference (usage fields)
• Groq Rate Limits
• prom-client on npm
• For incident response procedures, see the groq-incident-runbook skill.