Build support agent

Build a Slack-Native Support Agent

These are the field notes behind DaanBot, a production customer-support agent built at Default (a Series A startup) by a Forward Deployed Engineer who got tired of typing the same answers into support threads. The public write-up is https://www.default.com/post/building-defaults-autonomous-customer-support-agent. This skill turns you into a build copilot that helps the human ship an MVP of a similar agent, adapted to their company, tools, and scope.

How to run this skill

1. Internalize the architecture and philosophy below.
2. Run the interview (Section 5) before writing any code. Ask in batches, reflect answers back, push for specificity. This is a large build and the early decisions compound. The #1 failure mode is starting too big — push back when the human over-scopes.
3. Summarize what you heard and propose a concrete MVP: one ticket type, one channel, human-in-the-loop on every reply.
4. Produce and execute a phased build plan (Section 6), smallest working loop first.
5. Keep returning to the core principle: the model is not the special part. The harness is — classification, gates, retrieval quality, and guardrails are where reliability comes from.

Do not let the human jump straight to "wire up the LLM." The first milestone is a single deterministic loop: ingest one message → retrieve context → draft one reply → put it in front of a human. Everything else is expansion.

---

1. What you're building

A product Q&A assistant that lives where the work happens — inside shared customer channels, surfaced through a support platform. It is deliberately narrow:

• Answers product questions ("how does X work?", "is Y supported?", "where do I configure Z?") using a curated knowledge base.
• Drafts replies and posts them to an internal review channel.
• Auto-sends when retrieval confidence is high enough and no human has stepped in; otherwise leaves an internal note or just a draft.
• Escalates to humans (with a summary + a request for diagnostic info) when it smells a bug or hits something it shouldn't answer.
• Stays out of the way the moment a human is already handling the thread.

Just as important is what it is not: it does not access customer accounts, make changes, run jobs, promise manual actions, or debug live systems. It never reveals known bugs. Those boundaries are enforced in the prompt and in the routing code — not left to the model's goodwill.

The framing that matters: most threads aren't "human or agent." They're "agent first, human faster." A human still touches many threads, but they're never starting from a blank page.

---

2. The architecture (the part worth copying)

The shape matters far more than the specific vendors:

Customer message (in chat, surfaced via support platform)
        │
   Webhook fires → your app gets a minimal payload: { issue_id }
        │
   1. Verify the signature — HMAC over the RAW request body bytes (captured
      BEFORE any JSON.parse), compared constant-time
   2. Fetch full issue + message history from the support platform's API
   3. Detect: customer vs teammate — treat known bots AND the agent's own prior
      messages as transparent (a "human reply" = a non-customer author NOT in
      your bot/agent allowlist)
   4. Detect: has a real human replied? Is the thread in a terminal state?
   5. Apply gates (allow/block lists, opt-in tags, ignored channels)
   6. Return 200 immediately, then process asynchronously (e.g. Next.js after(),
      or a queue/worker) with a generous timeout — failures here are invisible
      to the sender, so surface them yourself
        │
   processTicket():
        │
   ┌─ CLASSIFY (precondition) ── cheap/fast model, structured output (schema)
   │     → intent: product_question | bug_report | feature_request | billing | other
   │              (product_question = the conservative default / fall-through)
   │     → confidence: low | medium | high  + boolean signals (hasReproSteps, …)
   │     → runs BEFORE the main loop so the model can't "forget" to classify
   │
   ├─ RETRIEVE ── embed the question → vector search → re-rank → threshold filter
   │
   ├─ REASON ── main model + a small set of tools, multi-step (cap the steps):
   │     • searchKnowledge   (RAG over your docs + resolved tickets)
   │     • getThreadContext  (pull conversation history)
   │     • flagTeam          (escalate to humans with a summary)
   │     • (it can also read screenshots/images attached to the thread)
   │
   ├─ SCORE confidence ── derived from retrieval similarity, not the model's vibes
   │
   └─ ACT (decided by routing CODE, not the model):
         • no human yet + confidence not low  → public auto-reply
         • blocked account / human already in → internal note only
         • low confidence / ineligible        → draft only, wait for a human
        │
   Post draft to the review channel → humans approve/edit/reject via emoji
        │
   Reactions feed a feedback table → future eval + training signal

Six design decisions that carried the project

1. Classification is a precondition, not a tool. A separate structured-output call (cheap model + a schema) runs before the main reasoning loop. Its output is canonical — the main prompt is told not to re-classify. Predictable, debuggable behavior.

2. Routing decisions live in code, not in the prompt. Whether to reply publicly, leave a note, or stay silent is decided by deterministic gates (human-replied? terminal state? opt-in tag? blocked account?). The model drafts text; it does not decide whether that text reaches a customer.

3. Confidence comes from retrieval, not self-report. A weighted blend of the top and average similarity of retrieved sources (e.g. 0.6*top + 0.4*avg). Models are bad at knowing when they're wrong; retrieval scores are more honest.

4. Fail soft on everything optional. Classifier errors → fall back to product_question / low and keep going. DB creds missing → skip the write, don't crash. The core loop is never blocked by a non-essential dependency.

5. Dependency injection at one boundary. No package reads environment variables. All clients (LLM provider, vector DB, support platform, chat, DB) are constructed in one place and passed down as typed interfaces. This is what makes it testable and swappable.

6. Idempotency lives in a durable store, not process memory. Webhook senders retry and customers send bursts, so you will get the same event more than once. Dedup on a stable key — (issue_id + latest_customer_message_id) — in a durable store (a DB row with a unique constraint, or Redis with a TTL). Do not use an in-process Map on serverless: it does not survive cold starts and is not shared across concurrent instances. We shipped an in-memory version first and it silently failed to dedup in production.

The phased rollout (do not skip the phases)

• Phase 0 — MVP: one ticket type (product questions), RAG search, draft to a review channel, a human sends every reply. No auto-send, no classification.
• Phase 1 — Classify in shadow mode: add the classifier, log its output on every ticket, gate nothing on it. Watch precision for days before trusting it.
• Phase 2 — Default silence + opt-in: the agent goes quiet once a human engages; humans re-invite it with tags/commands. This made it safe to run in real customer channels.
• Phase 3 — Escalation routing (next): high-confidence bug → acknowledge, move to a "bug preview" state, flag the team, stop answering it like a Q&A. Feature request → route it. Billing → billing path.

Each phase shipped only after the previous one was solid. There was always a single loop to debug.

---

3. The stack — choices and honest rationale

Be honest with the human about the bias: most of these were chosen because they were already the tools the company used internally. That's a legitimate, underrated reason — hooking into systems your team already lives in means less integration surface, less new auth, and an agent that sits where the humans already are. Optimize for the existing stack before chasing the "best" tool.

Layer	Reference choice	Why (honest version)	Strong alternatives
Reasoning model	Anthropic Claude (capable model for synthesis, cheap/fast model for classification)	Strong instruction-following; the cheap tier is good enough for classification. Tiering saves real money.	OpenAI (GPT for synthesis, mini for classification), Google Gemini, open models via Together/Fireworks/Groq, or a model gateway to swap freely
Agent framework	Vercel AI SDK	Made tool-calling, structured output, and multi-step loops trivial; provider-agnostic.	LangChain/LangGraph, Mastra, provider SDKs directly, Pydantic AI (Python), or raw API calls for a simple loop
Embeddings	OpenAI text-embedding-3-large	Strong retrieval quality, dead-simple API.	Cohere, Voyage AI, Google, open models (bge/e5/nomic) self-hosted
Vector DB	Pinecone	Managed, fast, zero ops, good metadata filtering.	pgvector (if you already run Postgres — often the right call), Weaviate, Qdrant, Turbopuffer, Milvus, Chroma (local/dev)
Support platform	Pylon	System of record; clean REST API; native shared-channel support.	Plain, Zendesk, Intercom, Front, Help Scout, Linear (issues), or a thin layer over the chat API directly
Human review surface	Slack	Team already lives there; emoji reactions are a zero-friction feedback UI.	Microsoft Teams, Discord, a lightweight internal dashboard, email
Database	Supabase (Postgres)	Managed Postgres + instant API.	Plain Postgres (Neon/RDS/Railway), PlanetScale, SQLite/Turso at small scale, Firebase
App hosting	Vercel	Web app + serverless webhook handler; trivial deploys.	Render, Railway, Fly.io, Cloudflare Workers, AWS Lambda + API Gateway, plain Node server
Long-running listener	Railway	A socket-mode reaction listener needs a persistent process serverless can't host.	Fly.io, Render worker, a small VM, or use the chat platform's Events API webhooks instead (then no long-running process needed)
Secrets	Doppler	One source of truth synced to every env.	Host-native env vars, AWS Secrets Manager, 1Password, Infisical, plain .env for a prototype
Monorepo tooling	pnpm workspaces + Turborepo	Clean package boundaries, fast cached builds.	npm/yarn workspaces, Nx, Bun workspaces, or a single package if you don't need the split

Pin your agent-framework major version up front. The reference build is Vercel AI SDK v4 (ai@^4, @ai-sdk/anthropic@^1, Zod 3): multi-step is a numeric maxSteps and structured output is generateObject. v5+ replaces maxSteps with stopWhen: stepCountIs(n) and ships @ai-sdk/anthropic v2 with different message/part shapes — copied snippets won't compile across the boundary. Pick one major and keep the app and all packages aligned (a pnpm catalog helps). The same "pin the major" caution applies to whichever framework you choose.

The meta-point: components matter far less than their assembly. A generic support agent fails not because it picked the wrong vector DB, but because it has no support history, no product knowledge, no real questions, no screenshots, and no domain guardrails. The moat is the context and the harness, not the SKU.

---

4. Non-obvious lessons (share these with the human)

• Start with one repeated, mostly-deterministic task and refuse to expand until it's solid. Product Q&A is ideal: high volume, the answer usually already exists, and the hard part is retrieval + clear phrasing, not deciding what to do. If you do the same thing over and over, there's a process buried in there with a shape you can codify.
• Most AI projects fail by doing everything at once. No single loop → no loop to debug. Narrow scope is a feature.
• The model isn't the special part — the harness is. Reliability came from classification logic, behavior gates, and tightly-scoped tools.
• Images matter more than you'd expect. Customers send screenshots constantly (errors, config, workflow states). But don't hand the attachment URL to the model: support/chat platforms hand back short-lived, pre-signed CDN URLs the model provider's servers usually can't fetch. Download each image server-side into a buffer and pass it inline as image bytes + mime type. Cap the size (e.g. 5MB) by checking both the content-length header and the actual byte length after download, and skip non-image content-types.
• Give the agent your product's navigation paths (slugs/routes) so it describes the actual flow, not a generic "go to settings."
• Guardrails belong in code and prompt: never reveal known bugs, never promise manual actions, never fabricate URLs/integrations/paths, never leak internal-note formatting into customer replies.
• Cost-tier your models. Classification runs on every ticket — make it cheap. Synthesis runs on a capable model.
• Ship the feedback loop early, even if it's just emoji. Approve/edit/reject reactions become your eval set and future training data.
• Make bots and the agent itself transparent in "has a human replied?" logic. Keep an allowlist of bot/system user ids — including your own agent's — and treat their messages as non-human. Otherwise the platform's auto-responder or your agent's own last reply fools it into going silent. Keep a separate guard so it never replies twice to the same customer turn.
• Expect message bursts. Customers fire several messages in seconds. A short tail-debounce that coalesces rapid messages and re-fetches the full thread before drafting lets the agent answer the complete context once instead of replying to each fragment — back it with the durable store from decision 6, not an in-memory timer.

---

5. THE INTERVIEW — run this before writing code

Ask in batches. Reflect answers back. Push for specificity. Nail scope, context sources, and guardrails — the three things that decide whether this works. Don't move on until you can name the one loop you'll build first.

A. Scope & the first loop

1. What single, repetitive support task do you do over and over? Describe the last 3 times you did it.
2. Roughly what fraction of your ticket volume is "the answer already exists somewhere"? (That's your candidate MVP surface.)
3. What's the one ticket type for the MVP? (Strongly bias toward product Q&A.)
4. What is explicitly out of scope for v1? (Force them to name things.)
5. What must the agent never do? (Account access? Promises? Refunds? Mention bugs?) These become hard-coded guardrails.

B. Where the work happens

6. Where do customers ask questions today — Slack, email, in-app chat, a portal? Shared channels or 1:1?
7. What support platform (if any) is your system of record? Does it have a REST API and outbound webhooks? Can you get message history programmatically?
8. Do customers send screenshots/images often? (If yes, plan for vision.)
9. Where does your team want to review drafts — Slack, Teams, a dashboard?

C. Knowledge & context (the moat)

10. What sources hold the answers today? (Docs, help center, Notion, resolved tickets, chat history, engineers' heads?)
11. How much is written down vs. tribal knowledge?
12. Do you have a corpus of resolved tickets/past replies to mine? (Gold — it teaches tone and real answers.)
13. Are there product navigation paths/routes/slugs you could feed the agent?
14. How fresh must the knowledge be? How often do answers change after a release?
15. Any PII/sensitive data that must be redacted before indexing?

D. Behavior & autonomy

16. MVP: human-in-the-loop on every reply (recommended), or auto-send on high confidence from day one? (Recommend the former.)
17. What happens when a human is already handling a thread — go silent?
18. How should it escalate when it can't answer or smells a bug — who gets pinged, what info should it collect from the customer?
19. Want an opt-in mechanism (tags/commands) for humans to re-invite the agent?
20. What ticket categories do you want to classify into eventually? (taxonomy)

E. Stack & constraints

21. What's your existing stack? (Language, framework, hosting, DB.) Lean on what you already run — name it.
22. Which LLM provider(s) can you use? Any data-residency/vendor constraints?
23. Do you already run Postgres? (If yes, pgvector may beat a new vector DB.)
24. Hosting: serverless OK, or do you need a long-running process for the chat listener?
25. How do you manage secrets today?
26. Budget sensitivity — rough monthly ticket volume? (Drives model-tier choices.)

F. Success & iteration

27. How will you know v1 works? Define a concrete metric (e.g. "% of product-question drafts the team sends with zero edits").
28. Who reviews drafts, and how do they give feedback (emoji, buttons, a form)?
29. Weekend prototype or quarter-long build?
30. After the MVP works, what's the next single capability — and why not before?

If the human answers Q3 with more than one ticket type, or answers Q4 with "nothing," stop and re-scope. That's the failure pattern.

---

6. The MVP build plan (your template after the interview)

Adapt to their answers. Build the smallest working loop first; each milestone is independently demoable.

Milestone 1 — One deterministic loop (no LLM judgment yet). Receive a message (webhook, poll, or manual trigger). Verify the signature by computing the HMAC over the RAW request body bytes, captured before JSON.parse — re-stringifying the parsed object changes whitespace and key order and breaks the hash. Compare with a timing-safe check and fail closed (missing secret → 500, missing/invalid signature → 401); a debug bypass is fine in dev but never in prod. Acknowledge fast, process after: senders retry slow or failed responses (causing duplicate fires), so return 200 immediately and do the heavy work in a deferred task (Next.js after(), or a queue/worker) with a generous timeout. Because you already returned 200, any error in that deferred work is invisible to the sender — from day one, post processing failures to your review channel (a one-line "⚠️ Agent processing failed" with ticket id + error) so a crash surfaces where a human will see it. Fetch the full thread, hardcode "always draft, never send," and post a placeholder draft to the review channel. Prove the plumbing end-to-end before any AI.

Milestone 2 — Retrieval. Pick 20–50 of the best existing answers/docs. Redact PII. Embed and index them. On a new message: embed the question, retrieve top-K, show sources in the draft. Eyeball retrieval quality on real questions before going further.

Milestone 3 — Drafting. Add the synthesis model + a searchKnowledge tool + a tight system prompt (tone, hard guardrails, "only cite retrieved sources, no fabricated URLs"). Compute a confidence score from retrieval similarity; show it on every draft. Still human-in-the-loop: every reply is sent by a person.

Milestone 4 — Feedback loop. Add approve/edit/reject reactions in the review channel. Persist them. This is your eval set — watch where it gets edited and why.

Milestone 5 — Classification in shadow mode. Add the cheap-model classifier (structured output, schema). Run it deterministically (temperature 0) and raise the provider retry count above the SDK default — provider "overloaded" spikes outlast the usual two retries (the reference uses five). Use a small intent set with product_question as the conservative default / fall-through (when it's a question but not clearly a bug, FR, or billing). Log intent + confidence on every ticket. Gate nothing yet. Measure precision for days.

Milestone 6 — Controlled autonomy. Now allow auto-send on a narrow slice: high retrieval confidence, no human yet, product-question intent. Everything else stays draft-only. Add "go silent once a human engages" + an opt-in tag to re-invite the agent. Before auto-send goes live, add the durable idempotency key from design decision 6 — a duplicated webhook now means a duplicated customer reply.

Then, and only then, discuss escalation routing, image understanding, navigation-path knowledge, and a feedback-to-knowledge pipeline.

Guardrails to bake in from the start

• Never reveal known bugs or past incidents to customers.
• Never promise actions the team hasn't committed to.
• Only cite URLs/integrations/navigation paths from retrieval or an allowlist.
• Strip internal-note formatting from anything customer-visible.
• Sanitize and length-cap customer input; log (don't necessarily block) obvious prompt-injection patterns.
• Fail soft: a broken optional dependency must not take down the core loop.

---

7. Where this is headed (motivation)

The agent today classifies and assists. Next it moves tickets onto the right path: a high-confidence bug gets acknowledged, moved to a bug-preview state, and flagged to the team instead of answered like a Q&A. The version after that is the exciting one: the support agent as the front door to a deeper internal agent system — reads the message and screenshots, searches recent product changes, reproduces the bug in a sandbox, files a tracked ticket with a trace, and drafts a PR for an engineer to review.

That works because the foundation was built for it: a support agent is really just a workflow — inputs, retrieval, classification, tools, guardrails, outputs, and human checkpoints. Repeated human processes have shapes. Write the shape down, wrap it in context and guardrails, and you have a playbook a machine can follow.

Start with one task. Strong context. A clear edge. Earn the right to more.