agentsclimarketplace

Client perf diagnosis

Skill cardstack/boxel/.claude/skills/client-perf-diagnosis

Diagnose a client-side performance complaint from the browser telemetry the host emits — an always-on passive instrument that beacons seven signal event types (card-load, server-request, deserialize, wedge, rebuild, realm-event, client-error) plus a keepalive liveness beacon to the realm-server's `/_client-telemetry` route, which re-emits each as one `boxel:client-perf` JSON log line that alloy ships to Loki and the "Host App Client Performance" Grafana dashboard (uid `boxel-client-perf`) reads with `| json`. Every event carries the authenticated `matrix_user_id` and a per-tab `session_id`, so a report is attributable to one account. Covers (1) a user says a card was slow to load — pivot to their `matrix_user_id`, find the slow `card-load` events (the "Loading card…" window) and their `settle_ms`, then explain the cost from that user's `server-request` / `deserialize` / `wedge` events in the same session; (2) a browser froze or went unresponsive — read the `wedge` events (a main-thread freeze at or above the wedge threshold), whose `top_frames` breadcrumb names the blocking script as `fn @ url:char` (source-map-resolvable for minified builds) with optional JS self-profiler stacks; (3) the app broke without freezing — `client-error` events carry the uncaught throw or unhandled rejection, its message and stack, and the `dedup_count` that tells one fault from a loop; (4) slow realm-server round-trips — `server-request` latency by endpoint, joined to the server's own `realm:requests` / `realm:search-timing` stage breakdown via the `x-boxel-logging-correlation-id` correlation id; (5) heavy response deserialization — `deserialize` duration by `card_type` against `doc_bytes` / `included_count`; (6) loader/store rebuild churn after a code edit — `rebuild` cost grouped by the `trigger_module` that forced the rebuild; (7) realm index-event write-burst churn — `realm-event` reload work per incoming index event, own-write vs external; and (8) turning any of the above into a single-user or single-session drill-down via the dashboard's `matrix_user_id` / `session_id` template variables. Use when someone reports the app was slow, a tab froze, the app broke or threw, a card took seconds to open, or asks which user is hitting client performance problems. The client half joins to the server half via the correlation id — for the realm-server-side render / search / indexing view of the same request, layer the `indexing-diagnostics` skill on top. For reading the raw realm-server logs directly, see `tail-logs`.From its SKILL.md

Install
npx -y skills add cardstack/boxel --skill client-perf-diagnosis

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

SKILL.md

42.9 KB, as published. Nobody here has run it

Client performance diagnosis

The host runs an always-on, passive client performance instrument. It samples the main thread and hooks the client's own server interactions, batches what it sees into a bounded ring buffer, and beacons the batch to the realm-server's POST /_client-telemetry route. The route re-emits one structured JSON log line per event on the boxel:client-perf channel; alloy tails the realm-server's stdout into Loki, and the "Host App Client Performance" Grafana dashboard (uid boxel-client-perf) — the host app is the instrumented client; boxel-cli and other API clients do not beacon — reads those lines with | json.

This is the tool for turning a vague field complaint — "the app was slow loading my card", "my tab froze" — into a measured story attributed to a specific account. Every event carries the authenticated matrix_user_id and a per-tab session_id, so you can start from who and reconstruct what their browser was doing.

The pipeline

host instrument  →  fetch(keepalive) POST /_client-telemetry  →  one `boxel:client-perf`
JSON log line per event  →  alloy tails realm-server stdout  →  Loki  →  Grafana `| json`

Nothing about this is a separate data store: it rides the same log pipeline every realm-server log already uses. A client event is just a realm-server log line with a known shape.

What gets measured, and how often

The instrument gathers at each signal's natural cadence, decoupled from how often it talks to the network:

  • The main thread is probed by a ~100ms heartbeat; long-animation-frame (LoAF) / longtask observers fire as the browser reports them; the JS self-profiler (on sampled sessions) samples continuously; the event hooks (card-load, server-request, deserialize, rebuild, realm-event) record at the instant each occurs; uncaught exceptions and unhandled rejections are captured as they surface, coalesced by signature so a loop reports a count rather than a flood.
  • Those samples accumulate in a ring buffer that a separate ~1s flush loop batches and POSTs at most once per second, and only when the batch carries signal. An otherwise-quiet tab still beacons a compact keepalive on a fixed cadence, so an active session_id is visible even when nothing is wrong.

So measurement resolution is finer than network chatter, and a healthy idle tab is nearly silent on the wire. Two consequences for reading the data:

  • A tab is quiet while hidden — browsers throttle background timers, so a heartbeat gap in a backgrounded tab is not a wedge and is dropped outright (the instrument keeps the heartbeat clock current on resume rather than measuring the hidden interval). A foreground gap is filtered once more against an OS sleep/suspend wake: a genuine freeze always leaves a LoAF/longtask entry, so an uncorroborated gap (checked across a short grace window, since the observer callback can lag the heartbeat) is treated as a sleep wake and dropped — but only where such an observer is available; with none, the heartbeat gap alone stands.
  • A keepalive event with a nonzero max_gap_ms is the low-grade jank channel; a wedge event is the incident-grade freeze channel (a gap at or above the wedge threshold). Read keepalive.max_gap_ms for "the tab was a bit janky", wedge for "the tab locked up".

Where the data lives — labels vs. JSON fields

Loki indexes only low-cardinality stream labels; everything else is a JSON field extracted at query time. This split is deliberate — it keeps per-user/per-session drill-down possible without exploding Loki's index.

  • Stream labels (cheap selectors, always in {…}): service="realm-server" and env (local / staging / production, stamped by alloy per environment). These are the only labels on the client-perf stream.
  • JSON fields (extracted with | json, then filtered): channel (always boxel:client-perf), event_type, matrix_user_id, session_id, app_version, env (the host build's environment, distinct from the stream label), realm (on most events), ts (epoch ms), plus every per-event field below.

The env stream label (set by alloy, selects which environment's log stream you read) is distinct from any per-event env JSON field (the host build's environment). Queries select on the stream label: {service="realm-server", env="production"}.

Every query therefore starts:

{service="realm-server", env="<env>"} |= "boxel:client-perf" | json
  | channel="boxel:client-perf" | event_type="<type>"

and then filters by matrix_user_id / session_id and unwraps a numeric field.

The leading |= line filter is load-bearing, not decorative. Client-perf lines are a small fraction of the realm-server stream, and | json on a deployed environment's whole stream parses every request log line in the range before anything gets filtered out. Grafana splits a range query into sub-queries, so a board full of unfiltered parses shows up as cancelled queries, 400 … eof, and 504 gateway time-out from Loki rather than as one slow panel. Keep the substring filter ahead of the first parser in anything you write; the dashboard's own queries all do.

In deployed environments each line arrives wrapped by the firelens log router, so the dashboard's queries unwrap it before the real parse — | json | line_format "{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}" | json. The |= filter still works ahead of that, because the substring survives the wrapper's JSON escaping.

The event glossary

Common envelope on every line: ts, event_type, channel, matrix_user_id, session_id, env, app_version, and (on most) realm.

event_typewhat it measureskey fields
card-loadcard-to-interactive latency — the "Loading card…" windowcard_id, loading_ms (window-open until the store reports the load resolved), settle_ms (one event-loop turn later, once the render settles — the fuller card-to-interactive number), num_loads, loaded_ids[], slowest_loads[] ({id, ms, outcome})
server-requestone realm-server round-trip, client-observedendpoint (normalized, ids stripped), method, status, duration_ms, resp_bytes, retried, correlation_id
deserializeturning a response into card instancesduration_ms, doc_bytes, included_count, card_type (a code ref address, <moduleURL>/<ExportName>, so the row says where the type's code lives)
wedgea main-thread freeze at/above the wedge thresholdduration_ms, worst_gap_ms, blocked_ms, longtask_count, top_frame_function, top_frame_url, top_frame_char, top_frame_blocked_ms, top_frames (fn @ url:char breadcrumb), loaf_scripts[], profiler_stacks[] (sampled sessions)
rebuildloader/store rebuild after a code changerebuild_source (realm-event = driven by an incoming index event; write = the tab's own code-mode save re-establishing the graph at save time), duration_ms, trigger_module (grouping key), trigger_modules[], modules_refetched, cards_reloaded, coalesced_events (index events that collapsed into this rebuild; always 0 for write)
realm-eventthe tab's work processing one incoming index eventindex_type (incremental/full), invalidations_count, invalidated_ids[], reloads_triggered, own_write, processing_ms
client-erroran uncaught exception or an unhandled promise rejectionkind (error = an uncaught throw, unhandledrejection = a promise nobody caught), message (verbatim), message_key (grouping key, per-occurrence ids collapsed to *), stack, top_frame_function, source_url, line, col, dedup_count (occurrences this one event accounts for), folded_signatures (0 normally; nonzero on a budget-folded event, where the count spans that many distinct errors)
keepaliveliveness beacon from an otherwise-quiet tabwindow_ms, max_gap_ms

Array fields (loaf_scripts, slowest_loads, loaded_ids, profiler_stacks) are not extractable by | json element-by-element — that is why the wedge event also carries flat scalar top_frame_* fields for grouping. To see an array's contents, read the raw log line (see Reading raw lines). client-error.stack is a scalar, so | json does extract it, but it is multi-line: it reads as a stack only expanded on a raw line, which is what the errors row's detail panel is for.

How to actually query

The LogQL below is environment-agnostic; what changes is how you reach Grafana/Loki.

Local dev

Bring up the observability stack and push the dashboards:

cd packages/observability
docker compose up -d          # Grafana :3001, Loki :3100, alloy, prometheus
./scripts/apply.sh --env local

alloy tails the realm-server's stdout file (${BOXEL_LOG_DIR:-/tmp/boxel-logs}/realm-server.log, mounted into the alloy container) and labels the stream {service="realm-server", env="local"}. So the local realm-server must be running with its stdout going to that file (the dev service tasks do this), and the host must be running so beacons actually POST. Open Grafana at http://localhost:3001 → Host App Client Performance, with env=local.

Sanity-check the path end-to-end by querying Loki's HTTP API directly:

# Are any client-perf lines landing at all (last 15m)?
curl -sG 'http://localhost:3100/loki/api/v1/query_range' \
  --data-urlencode 'query={service="realm-server", env="local"} |= "boxel:client-perf" | json | channel="boxel:client-perf"' \
  --data-urlencode "start=$(( ($(date +%s) - 900) ))000000000" \
  --data-urlencode "end=$(date +%s)000000000" \
  --data-urlencode 'limit=20' | python3 -m json.tool | head -60

If that returns rows, the pipeline is healthy and any empty panel is a filter/time-range issue, not a plumbing one.

Staging / production

The data lives in the hosted Grafana for that environment. Each environment's dashboard has its env baked in (a hidden constant substituted at apply time), so you don't switch env — you open that environment's Grafana and use the account/session controls below. Everything else in this skill — the modes, the field meanings, the queries — is identical. No database tunnel is needed (unlike server-side render diagnostics): the whole client story is in Loki.

The drill-down controls

Two dashboard variables are the entire user-attribution mechanism; the third is a fixed per-deployment constant:

  • matrix_user_id — the account. Defaults to $__all (a match-all regex, so "All" really means all users, including anonymous/synthetic sessions). Pick a user to scope the whole board to that account. In LogQL this is | matrix_user_id=~"$matrix_user_id".
  • session_id — a free-text box, matched as a substring: | session_id=~".*${session_id}.*". Empty matches every session; paste a session id to isolate one tab.
  • env — not an operator control: a hidden constant baked into each environment's dashboard at apply time, so {… env="$env"} resolves to that deployment. You change environment by opening its Grafana, not by picking a value here.

Picking a matrix_user_id (and optionally a session_id) turns every panel on the board into a single-client view. That is the move Mode A is built on.

Mode A — a user reports their card was slow to load

This is the common one: "user XYZ said the app was slow loading their card." You have an account and a vague symptom. Turn it into a measured window and a cause.

Step 1 — get the account's matrix_user_id. It's the Matrix user id (@localpart:server). In the dashboard, the matrix_user_id variable is populated from the users table, so you can pick it from the dropdown. Select it (and leave session_id empty to start).

Step 2 — find the slow card-load(s). The card-load event is the "Loading card…" window. Look at that user's slowest card-loads by settle_ms, keyed by card_id:

topk(20, max by (card_id) (
  max_over_time({service="realm-server", env="$env"} |= "boxel:client-perf" | json
    | channel="boxel:client-perf" | event_type="card-load"
    | matrix_user_id=~"$matrix_user_id" | session_id=~".*${session_id}.*"
    | unwrap settle_ms [$__range])
))

loading_ms is the elapsed time until the store reports the load resolved; settle_ms is measured one event-loop turn later, once the render has settled — the fuller card-to-interactive number, and the one the user experiences as "still spinning". (Neither is a first-paint metric; they differ by about a tick.) A card-load with a large settle_ms and a large num_loads pulled a lot of documents to become interactive — the slowest_loads[] array on that line names which ids were slowest and whether they errored.

Step 3 — pin the session and the moment. Once you have a slow card_id, find the session_id and timestamp of that specific event by reading the raw line (see Reading raw lines), then paste the session_id into the dashboard variable. Now the whole board is that one tab, and you can read what else was happening around that ts.

Step 4 — spider into the cause. The card-load names the symptom and its magnitude; the other event types in the same pinned session name the cause. With the session isolated and the slow card-load's ts in hand, walk outward into each event type and let the one that overlaps the window explain it:

  • Slow server round-trips → Mode C. A card-load that spent its time on the network shows up as slow server-request.duration_ms; a server-request with retried="true" re-attempted the fetch after a transient auth (401). Note duration_ms covers only the final successful attempt — each attempt is timed separately and the transient 401 event is dropped from the stream — so it excludes the reauth round-trip rather than including it. The correlation_id then crosses to the server's own timing — why the server was slow.
  • A main-thread freeze → Mode B. If the tab locked up mid-load, a wedge overlaps the window and its settle_ms includes the freeze; the wedge's top_frames breadcrumb names the blocking script.
  • Something threw → Mode H. A client-error inside the window means the load didn't just take long, it failed: the stack names the code that broke, and a load that never settled has a settle_ms pinned at the watchdog rather than a real measurement.
  • Heavy deserialization → Mode D. Large deserialize.duration_ms / doc_bytes for a card_type in the load — a big linked graph deserializes slowly.
  • Rebuild churn → Mode E. If the load raced a loader/store rebuild (a rebuild event in the window), the card waited on the rebuild; trigger_module names what forced it.
  • Index-event churn → Mode F. A realm-event storm (high reloads_triggered) during the window means the tab was busy absorbing incoming index events while trying to load.

Whichever event overlaps the window with the outsized number is the lead. The whole point of pinning the session first is that this spider is then confined to one tab's timeline, so the overlap is real rather than coincidental.

Mode B — a browser froze / went unresponsive

A wedge event is emitted when a foregrounded tab's main thread freezes for a heartbeat gap at or above the wedge threshold. A backgrounded tab's throttled gaps are excluded outright; a foreground gap is checked against an OS sleep/suspend wake by requiring a corroborating LoAF/longtask entry (a real freeze always leaves one, though the observer callback can lag the heartbeat, so the check spans a short grace window) — so a sleep wake isn't misreported as a freeze.

Find the wedges, per user, grouped by the blocking frame:

topk(15, sum by (top_frame_function, top_frame_url) (
  count_over_time({service="realm-server", env="$env"} |= "boxel:client-perf" | json
    | channel="boxel:client-perf" | matrix_user_id=~"$matrix_user_id"
    | session_id=~".*${session_id}.*" | event_type="wedge" [$__range])
))

Don't add a top_frame_function!="" filter here: a wedge corroborated only by a longtask (no LoAF script attribution) carries an empty top_frame_function yet is a real freeze — filtering it out would hide exactly the symptom you're chasing. Those wedges group under the blank frame; read their top_frames / longtask_count / blocked_ms from the raw line.

Read the breadcrumb. Each wedge carries:

  • top_frame_function / top_frame_url / top_frame_char — the single worst blocking script as scalar fields (groupable). top_frame_char is a source character offset: with a source map it resolves to a line/column even for a minified production build, where the function name alone is useless.
  • top_frames — a readable multi-frame summary (fn @ url:char, worst few), so the frame can be reasoned about beyond one name. Read it from the raw line.
  • loaf_scripts[] — the worst blocking scripts the LoAF observer attributed, each with source_url, function_name, char_position, invoker, blocking_duration_ms.
  • profiler_stacks[] — on sessions the JS self-profiler sampled, real sampled call stacks spanning the freeze. Absent on sessions the profiler didn't sample, and on any session where the JS self-profiler API or its document policy isn't available; the LoAF breadcrumb still ships either way.

blocked_ms is the total blocked time in the wedge window; worst_gap_ms is the single largest heartbeat gap; longtask_count is how many long tasks piled up.

Mode C — slow realm-server round-trips (and the server-side join)

server-request records one round-trip as the client saw it. endpoint is normalized (instance ids stripped to a low-cardinality label like GET card or _search), so you can group by it.

p95 by endpoint, and most-retried endpoints:

quantile_over_time(0.95, {service="realm-server", env="$env"} |= "boxel:client-perf" | json
  | channel="boxel:client-perf" | event_type="server-request"
  | matrix_user_id=~"$matrix_user_id" | session_id=~".*${session_id}.*"
  | unwrap duration_ms [$__interval]) by (endpoint)
sum by (endpoint) (count_over_time({service="realm-server", env="$env"} |= "boxel:client-perf" | json
  | channel="boxel:client-perf" | event_type="server-request" | retried="true"
  | matrix_user_id=~"$matrix_user_id" | session_id=~".*${session_id}.*" [$__range]))

The join to the server's own view. server-request.correlation_id is the x-boxel-logging-correlation-id the client stamps on the outgoing request. The realm-server logs the same id as corr=<id> on its realm:requests (dur= total) line and, for searches, its realm:search-timing stage breakdown. So a client-observed slow request splits into network/queue time vs server processing time: grab the correlation_id from the raw client line, then search the realm-server logs for corr=<that id>. The server-side stage attribution (parse / SQL / loadLinks / serialize / queue) is covered by the indexing-diagnostics skill's search-timing mode — this is where the client and server halves of one request meet.

Mode D — heavy response deserialization

deserialize times turning a fetched document into card instances. A heavy linked graph (an index/dashboard card) is the classic outlier.

quantile_over_time(0.95, {service="realm-server", env="$env"} |= "boxel:client-perf" | json
  | channel="boxel:client-perf" | event_type="deserialize"
  | matrix_user_id=~"$matrix_user_id" | session_id=~".*${session_id}.*"
  | unwrap duration_ms [$__interval]) by (card_type)

Correlate duration_ms against doc_bytes and included_count (swap the unwrap field) to tell "big document" apart from "expensive to build". A card_type whose duration_ms grows faster than its doc_bytes is doing expensive per-instance work, not just moving bytes.

Mode E — rebuild churn after a code edit

A rebuild event fires when a .gts/code invalidation forces a loader/store rebuild. It is attributed to why it rebuilt: trigger_module is the invalidated module that forced it (the grouping key; trigger_modules[] has the full set).

Rebuilds come from two machines, named by the event's rebuild_source (so named because the firelens wrapper's own source key wins the wrapper-unwrap parse — parsed-label conflicts keep the first extraction and drop the second, without the _extracted suffix that stream-label conflicts get):

  • realm-event — the store's event-driven rebuild. Runs when an incremental index event names an executable module this tab had loaded, and only in a tab whose store is subscribed to that realm — a subscription exists while something holds a card instance from it (an open preview, a stack item, a playground). The loader is flushed the instant a module is rewritten — by a save from the app, or by the code-mode file resource reacting to someone else's write — and the index event lands later, after indexing; the store consults a snapshot of what the discarded loader held, so the flush cannot hide the module from the rebuild decision.
  • write — the tab's own code-mode save re-establishing the graph at save time, without waiting for the index event. A session editing a module in a realm the store holds no instances from does all of its rebuilding here: no subscription means the index event never reaches its store, so no realm-event (and no realm-event-sourced rebuild) will ever appear for it. An empty rebuild row for a user known to be editing code usually means you are looking for the wrong source.

A save of a loaded module in a subscribed realm pays both — one event per source, and the two events describe two complete graph re-fetches, not one counted twice. That double pass is a known cost, not an intent: the index event cannot tell that the write-time pass already ran, so the tab tears down and re-fetches every open card twice per save. Collapsing the two passes is open work; until then, two rebuild rows per save in a subscribed session is the accurate reading, not a dashboard bug.

A realm-event-sourced rebuild always has a realm-event beside it in the same session and window — read that event's own_write to tell a save made in this tab from someone else's write. A write-sourced rebuild has no such companion: the session that pays it is typically one whose store receives no realm events at all. The cross-check that carries information is the other direction: incremental realm-events carrying executable invalidated_ids that no rebuild follows means code changes are landing that this tab is not absorbing.

sum by (trigger_module) (count_over_time({service="realm-server", env="$env"} |= "boxel:client-perf" | json
  | channel="boxel:client-perf" | event_type="rebuild"
  | matrix_user_id=~"$matrix_user_id" | session_id=~".*${session_id}.*" [$__range]))

avg by (trigger_module) (... | unwrap cards_reloaded ...) and ... | unwrap modules_refetched ... show which module's edits are the most expensive to absorb. A trigger_module that reloads many cards on every edit is a hot dependency.

How many events did this rebuild absorb? coalesced_events is the number of incremental index events that collapsed into this one rebuild. Rebuilds are coalesced: at most one runs in flight and one stays pending, so a burst of executable invalidations arriving faster than a rebuild completes bounds to two rebuilds regardless of burst length, and the trailing rebuild carries the whole burst's coalesced_events. Read it against the realm-event count in the same window:

quantile_over_time(0.95, {service="realm-server", env="$env"} |= "boxel:client-perf" | json
  | channel="boxel:client-perf" | event_type="rebuild"
  | matrix_user_id=~"$matrix_user_id" | session_id=~".*${session_id}.*"
  | unwrap coalesced_events [$__interval]) by ()

coalesced_events = 1 is an isolated edit — one event, one rebuild. coalesced_events > 1 is a write burst (an agent session writing every few seconds is the typical source) absorbed into a single rebuild: the tab paid one full-graph re-fetch for many events instead of one per event. So the burst signature is few rebuild events with high coalesced_events, not one rebuild per realm-event. During a burst, one rebuild per incoming realm-event with coalesced_events stuck at 1 means the events are landing slower than a rebuild completes (each finishes before the next arrives) — no burst to coalesce — rather than coalescing failing to engage.

Why did it rebuild? A rebuild in a session the user wasn't editing usually traces to an incoming index event: look for a realm-event in the same window whose invalidated_ids[] contains this trigger_module (Mode F). That distinguishes a self-inflicted rebuild (the user saved a module) from one pushed by someone else's write.

Mode F — realm index-event write-burst churn

realm-event is often the "why" behind reloads — the causal upstream of the other modes. It measures the tab's cost to process one incoming realm index event (the write-burst churn from the client's side), and that same incoming event is frequently what set the other costs in motion — why a rebuild fired, and why a card reloaded links the user never asked for. When Mode A, D, or E shows churn without an obvious local trigger, look for a realm-event in the same window — it is the answer to "why did this happen when I didn't touch anything?"

sum(sum_over_time({service="realm-server", env="$env"} |= "boxel:client-perf" | json
  | channel="boxel:client-perf" | event_type="realm-event"
  | matrix_user_id=~"$matrix_user_id" | session_id=~".*${session_id}.*"
  | unwrap reloads_triggered [$__interval]))

Split own_write (the tab's own edits echoing back) from external events with sum by (own_write) (count_over_time(... event_type="realm-event" ...)): a tab drowning in external events with high reloads_triggered is paying for someone else's write burst, whereas high own_write churn is self-inflicted. processing_ms is the per-event reload cost; index_type distinguishes an incremental event from a full-reindex broadcast.

Trace the causal chain. The event's invalidated_ids[] (bounded, read from the raw line) is what the index event marked stale, and it links the realm-event to its downstream cost:

  • A rebuild whose trigger_module appears in a realm-event's invalidated_ids in the same window — the incoming invalidation of an executable module is what forced the loader/store rebuild. The realm-event is the cause; the rebuild is the effect.
  • A card-load (or a spike in another load's num_loads / slowest_loads) whose reloaded ids intersect a realm-event's invalidated_ids — the tab reloaded a linked card because an index event invalidated it, not because the user navigated. reloads_triggered counts exactly these knock-on reloads.

So the reading order for "why was the tab busy" is often backwards from the symptom: land on the rebuild / card-load cost first (Modes E/A), then find the realm-event in the same session-and-window whose invalidated_ids explain it.

Mode G — scope anything to one user or one session

Every mode above already threads matrix_user_id and session_id. To go the other direction — which users/sessions are hitting problems — count distinct sessions and rank users.

Mind the difference between "active now" and "seen in the window." The lookback duration is the whole distinction, and getting it wrong is the easiest way to misread this data:

# sessions active now — a rolling 5m lookback, so a tab counts only while it is
# still beaconing. Any quiet tab keeps beaconing a keepalive on a fixed cadence,
# so a live session always lands inside a 5m window.
count(count by (session_id) (count_over_time({service="realm-server", env="$env"} |= "boxel:client-perf" | json
  | channel="boxel:client-perf" | matrix_user_id=~"$matrix_user_id"
  | session_id=~".*${session_id}.*" [5m])))

# sessions seen anywhere in the window — swap the lookback for [$__range]. This
# is cumulative: it only grows as you widen the time picker, and a single person
# reloading the app contributes a session per load, because a session id is
# minted per tab.

The dashboard's Overview row is built entirely from rolling 5-minute windows: Card loads (5m), Wedges ≥10s (5m), Rebuilds (5m), and Errors (5m) each show the latest 5-minute count as the number and the count's movement across the range as the sparkline. The errors tile counts events rather than occurrences, so a looping error reads as a small number there — the errors row is where the dedup_count sum lives. Who is on — and how many sessions, now and across the range — is the Active sessions over time by user panel, stacking the 5m-window session count per account: total height is concurrency, the colours are who, and the legend doubles as the current-user list (its Last column is strict-last, so a band that aged out reads blank). The same panel carries the build story in two extra layers: grey build in use: <sha> legend rows name each host build the active sessions run (from the app_version every beacon stamps) — one row is the quiet steady state, a second row right after a deploy is a tab that has not reloaded and still runs the previous code — and hovering the chart names which user is on which build (user → build per point). A user split across two builds also shows as two same-named bands. Reading a stale tab is therefore: second grey row appears → hover → the user and sha to tell them to reload.

Swap the outer count by (session_id) for count by (matrix_user_id) to get accounts instead of tabs. That per-account form over the same 5m window is what the Overview board publishes as its Concurrent Users headline (minus the drill filters), so the two boards agree by construction. It is app-sourced: matrix_user_id comes from the beacon's authenticated session token rather than the browser, and only the host emits these events, so Matrix accounts that sync without loading the app (the AI bot runs a sliding-sync client of its own) are not in it. Unauthenticated browsing is not in it either — the instrument buffers but cannot beacon without a realm-server session token.

Group any of the health queries by (matrix_user_id) instead of the default aggregation to get a per-user leaderboard. The dashboard's By user row does exactly this, and each table carries its "why": wedges by user name the blocking frame and its worst blocked time (the frame is what makes a row actionable), worst card-to-interactive names the card that was slow, and the signal mix shows one column per event type — volume alone is activity, weight in the wedge/rebuild columns is suffering. From a row there, copy the matrix_user_id back into the variable to drill in.

Mode H — the app broke without freezing

"The app broke." No freeze, no spinner — it just stopped working, or a tab went silent. This is the channel for that: client-error reports every uncaught exception and unhandled promise rejection a tab suffers, so a fault that would otherwise live only in the user's own console is attributable to their account.

Start from the account. The Errors by user panel on the errors row is the pivot: it names the account and what broke for them. Or go straight to LogQL:

topk(20, sum by (message_key, top_frame_function) (
  count_over_time({service="realm-server", env="$env"} |= "boxel:client-perf" | json
    | channel="boxel:client-perf" | event_type="client-error"
    | matrix_user_id=~"$matrix_user_id" | session_id=~".*${session_id}.*" [$__range])
))

Read the stack, not just the message. The message says the app broke; the stack says where. Read it from the raw line (or the row's detail panel) — frames are kept from the top down and a deep stack loses its tail rather than its head, so the throw site is the first frame. The message the engine writes above the frames is bounded separately from them, so a huge message (one carrying a response body or a serialized document) is cut rather than costing you the frames; a … marks any value that was cut.

top_frame_function is the same frame as a scalar, for grouping, and source_url / line / col locate the throw — for a rejection, which carries no location of its own, from the stack's first frame. Both kinds of frame locate real code: a built asset (/assets/*.js) source-maps back, and a card module carries its own realm url, because the loader appends //# sourceURL=<module url> when it evaluates one. So a card-module frame names the module a reader can open.

An empty stack is not a missing capture. It means the throw carried nothing to take a stack from: a cross-origin script error (the browser withholds the detail; the message reads Script error.), a promise rejected with something that isn't an Error, or a rejection reason that is merely error-shaped — an object with a message but no stack. The message is all that exists in those cases.

Count events against occurrences. Identical errors coalesce: one event carries a dedup_count covering every occurrence it stands for, so a render loop throwing thousands of times cannot flood the beacon or push the other signal types out of the tab's buffer. That makes volume a two-part reading:

  • Many events, dedup_count near 1 — distinct faults. Read them individually.
  • Few events, high dedup_count — one fault looping. The count is the severity; the stack is the same every time.

So count_over_time measures distinct-error pressure and sum_over_time(… | unwrap dedup_count …) measures what the user actually suffered. The errors row plots both on one panel for that reason.

Past a per-window event budget, further distinct errors in the same second fold into a single event, and folded_signatures says so: 0 on an ordinary event, where dedup_count counts occurrences of the one error described, and nonzero on a folded event, where the count spans that many distinct errors and the message and stack are only a sample of them. Check it before reading a large dedup_count as one error looping — that is the one case where the two look alike.

message_key is the grouping key. It is the message with the parts that vary per occurrence collapsed to *, so the same failure against a hundred different cards is one row instead of a hundred. Group by message_key; read message for the verbatim text of one occurrence.

The rule is deliberately narrow: a token of four or more word characters containing a digit becomes *. That covers uuids, content hashes, timestamps, and the generated ids most instance urls carry, and it leaves short numbers that carry meaning — an HTTP status, a small count — intact. It does not collapse a short or digit-free id, so …/Person/abc and …/Person/xyz stay separate rows: nothing distinguishes those from an ordinary word. Treat the key as a grouping aid, not a guarantee of one row per failure. A key longer than its budget ends in …#<8 hex>, a digest of the whole message — two long messages that differ only near the end stay distinct rather than merging.

A session that goes quiet. A tab that dies from a throw stops beaconing, which on the session-history panel looks exactly like a closed tab. A client-error at the tail of a session tells the two apart: the session ended because something broke, not because someone navigated away. Pin the session_id and read the events leading up to it.

What this channel does not carry. Errors from a prerender tab. A render that throws is an expected indexing outcome — a card with a broken link or a bad module — and the instrument stays dormant in a render context, so nothing here is server-side render noise. For those, use the indexing-diagnostics skill.

Reading raw lines

Aggregations can't show you array fields (loaf_scripts, slowest_loads, top_frames, correlation_id on a specific event). To read the full JSON of specific events, query the lines directly instead of an aggregation — in Grafana Explore, or via the API:

curl -sG 'http://localhost:3100/loki/api/v1/query_range' \
  --data-urlencode 'query={service="realm-server", env="local"} |= "boxel:client-perf" | json
      | channel="boxel:client-perf" | event_type="wedge" | matrix_user_id=~"@user:example.com"' \
  --data-urlencode "start=$(( ($(date +%s) - 3600) ))000000000" \
  --data-urlencode "end=$(date +%s)000000000" \
  --data-urlencode 'limit=50' | python3 -m json.tool

Each returned line's values[][1] is the full JSON object — read top_frames, loaf_scripts, slowest_loads, or correlation_id from there.

Calibrating thresholds

The event fields are facts; the line between "fine" and "slow" is per-realm. A card-to-interactive that's normal for a heavy dashboard realm is alarming for a realm of trivial cards. Read a realm's own baseline (the overview row over a healthy window) before calling a number pathological, and prefer the high quantiles (p95/p99) and the topk panels over averages — a client complaint is a tail event, and averages hide tails.

Related skills

  • indexing-diagnostics — the server-side half. When a server-request correlation_id points at a slow realm-server response, that skill attributes the server's time across its request→response stages (parse / SQL / loadLinks / serialize / queue) and, for renders, across the prerender pipeline. The correlation id is the seam between the two skills.
  • tail-logs — reading the realm-server's raw log stream directly (including the corr=<id> lines you join to).
  • aws-access — the AWS session and read-only DB tunnel for deployed environments, needed only when a client investigation crosses into server-side data that isn't in Loki.

Keep looking

Skills are one crate of 613,757. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.