Build voice agent

Build a voice agent with Patter

Patter is an open-source SDK that turns any AI agent into a phone agent. Pick one of three architectures (Realtime, ConvAI, Pipeline), describe the behaviour in a system prompt, and run a local server that connects to your Twilio or Telnyx number — no Patter Cloud, no managed service. The four-line pattern below is the entire surface.

Architecture in one diagram

┌────────────────────────────────────────────────────────────────────┐
│                          Patter server (local)                      │
│   ┌────────────────┐    ┌───────────────────┐    ┌───────────────┐ │
│   │ Carrier WS     │───▶│  Agent (engine /  │───▶│  Carrier WS   │ │
│   │ in (audio in)  │    │  pipeline) loop   │    │ out (TTS out) │ │
│   └────────────────┘    └───────────────────┘    └───────────────┘ │
│           ▲                                              │          │
│           │                Built-in tools:               │          │
│           │             transfer_call · end_call         │          │
└───────────┼──────────────────────────────────────────────┼──────────┘
            │                                              │
            │ WSS (mulaw 8kHz / pcm 16kHz)                 │
            ▼                                              ▼
       ┌──────────┐                                   ┌──────────┐
       │  Twilio  │  ────────  real phone call  ───── │   User   │
       │  Telnyx  │                                   └──────────┘
       └──────────┘

You write: system prompt, optional tools, optional guardrails, choice of engine. Patter writes: WebSocket framing, audio transcoding, barge-in, VAD, cost tracking, tunnel, dashboard.

Decide the mode

Mode	When to use	Latency	Cost
OpenAIRealtime2 (GA, default)	Default for everything. Lowest latency, single key.	~200 ms turn	$$$
OpenAIRealtime	Legacy gpt-realtime-mini; pick only if explicitly required.	~200 ms turn	$$
ElevenLabsConvAI	Turn-taking conversations (slower but better at long pauses).	~600 ms turn	$$
Pipeline	Custom STT/LLM/TTS combinations. Use when the user wants Anthropic Claude / Cerebras / Whisper / a specific TTS voice.	~800 ms turn	$-$$$$

Default pick: OpenAIRealtime2. Switch only if the user explicitly asks for custom voice/LLM, lower cost, or a non-OpenAI provider.

Quick start

Python

import asyncio
from getpatter import Patter, Twilio, OpenAIRealtime2

async def main():
    phone = Patter(carrier=Twilio(), phone_number="+15550001234")
    agent = phone.agent(
        engine=OpenAIRealtime2(),
        system_prompt=(
            "You are Mia, the AI receptionist for Acme Plumbing. "
            "Greet the caller warmly. Help them book a service visit. "
            "Keep replies under two sentences."
        ),
        first_message="Hi, this is Mia at Acme Plumbing — how can I help?",
    )
    await phone.serve(agent, tunnel=True)

asyncio.run(main())

TypeScript

import { Patter, Twilio, OpenAIRealtime2 } from "getpatter";

const phone = new Patter({
  carrier: new Twilio(),
  phoneNumber: "+15550001234",
});

const agent = phone.agent({
  engine: new OpenAIRealtime2(),
  systemPrompt:
    "You are Mia, the AI receptionist for Acme Plumbing. " +
    "Greet the caller warmly. Help them book a service visit. " +
    "Keep replies under two sentences.",
  firstMessage: "Hi, this is Mia at Acme Plumbing — how can I help?",
});

await phone.serve({ agent, tunnel: true });

Both expose a tunnel URL on startup (tunnel ready: https://abc.trycloudflare.com). Point your Twilio number's voice webhook at that URL (see configure-telephony), call the number, talk to Mia.

Pick the right mode

The default works for 80% of agents. Switch when:

Want this	Pick this mode	Reference
Lowest possible latency, OpenAI voice	OpenAIRealtime2 (default)	references/realtime-mode.md
Better long-pause handling, ElevenLabs voice	ElevenLabsConvAI	references/convai-mode.md
Mix custom STT / LLM / TTS (Anthropic, Deepgram, Cartesia, …)	Pipeline	references/pipeline-mode.md
Save cost on high-volume outbound	Pipeline with Cerebras LLM + Deepgram STT + ElevenLabs Turbo	references/pipeline-mode.md
Noisy line / speakerphone (denoiser, high-pass, AGC, semantic turn detection)	Pipeline audio levers	references/pipeline-mode.md

Read the corresponding reference only when the user picks that mode — do not preload all three.

Outbound calls

The same Patter instance places outbound calls — but phone.call() always needs a running server to dial through (raises PatterConnectionError otherwise), and you choose how the call's lifecycle is reported:

• wait=True (completion-aware, since 0.6.3) — call() blocks until the callee hangs up and returns a CallResult with the outcome, duration, transcript, and cost. This is what you want for scripts and campaigns: one await, one resolved call.
• wait=False (default, fire-and-forget) — call() returns at the moment the carrier dials (not at hangup) and yields None/void. The call then lives entirely inside the running server — so something has to keep that server alive (a long-running serve(), or the async with / await using block below) for the conversation to continue.

The completion-aware form is the recommended pattern. async with Patter(...) (Python) / await using (TypeScript) boots the local server, keeps it alive for the call's lifetime, and tears it down cleanly on exit — no dangling process, no manual serve() task to cancel:

import asyncio
from getpatter import Patter, Twilio, OpenAIRealtime2

async def main():
    # `async with` runs the local server for the duration of the block.
    async with Patter(carrier=Twilio(), phone_number="+15550001234") as phone:
        agent = phone.agent(
            engine=OpenAIRealtime2(),
            system_prompt="You are Mia from Acme. Keep replies under two sentences.",
            first_message="Hi, this is Mia from Acme.",
        )
        result = await phone.call(
            to="+14155551234",
            agent=agent,
            voicemail_message="Sorry we missed you. Call back at +1...",
            wait=True,                 # block until the call ends → CallResult
        )
        # CallResult is frozen: call_id, outcome, status,
        # duration_seconds, transcript, cost, metrics.
        print(result.outcome)          # answered | voicemail | no_answer | busy | failed
        print(f"{result.duration_seconds:.0f}s · ${result.cost.total_usd:.4f}")

asyncio.run(main())

import { Patter, Twilio, OpenAIRealtime2 } from "getpatter";

// `await using` runs the local server, then disposes it when the block exits.
await using phone = new Patter({
  carrier: new Twilio(),
  phoneNumber: "+15550001234",
});

const agent = phone.agent({
  engine: new OpenAIRealtime2(),
  systemPrompt: "You are Mia from Acme. Keep replies under two sentences.",
  firstMessage: "Hi, this is Mia from Acme.",
});

const result = await phone.call({
  to: "+14155551234",
  agent,
  voicemailMessage: "Sorry we missed you. Call back at +1...",
  wait: true,                          // block until the call ends → CallResult
});
// CallResult is readonly: callId, outcome, status,
// durationSeconds, transcript, cost, metrics.
console.log(result.outcome);           // answered | voicemail | no_answer | busy | failed
console.log(`${result.durationSeconds}s · $${result.cost.totalUsd}`);

wait=True is timeout-bounded on ring_timeout (default 25 s), so a number that never picks up resolves to outcome="no_answer" rather than hanging forever. machine_detection is on by default since 0.6.3: Patter auto-detects voicemail, plays the call's voicemail_message before hanging up, and the CallResult comes back with outcome="voicemail". A non-empty voicemail_message implicitly enables AMD even if you pass machine_detection=False. Otherwise the live person hears first_message and the agent starts the conversation. See configure-telephony for the full set of outbound options.

System prompt — what matters for voice

Voice prompts are different from chat prompts. Patter doesn't add hidden instructions, so be explicit:

1. Identity: "You are <name>, calling on behalf of <company>."
2. Goal: "Help the caller book a service visit." (one line)
3. Length rule: "Keep every reply under two sentences." (LLMs over-explain on voice.)
4. Stop conditions: "If the caller wants a human, call transfer_call."
5. Persona: "Warm, professional, never robotic. Say 'mm-hm' when listening."

The first_message is what the agent says immediately on pickup — keep it to 1 sentence under 8 seconds of audio.

Gotchas

• Patter has no Patter Cloud as of 0.7.0 — Patter(api_key=...) raises NotImplementedError. Always use carrier=... + phone_number=....
• Built-in tools are always present: every agent gets transfer_call and end_call for free. You don't have to register them.
• Mulaw vs PCM audio: Twilio carrier = mulaw 8 kHz, Telnyx = PCM 16 kHz. Patter transcodes both transparently — you never touch raw audio unless you write a pipeline hook.
• tunnel=True is dev-only. For production, use a static webhook URL (your own subdomain or paid ngrok). Cloudflare quick tunnels work but occasionally have a ~3 s WSS upgrade race on first call.
• first_message is played before the LLM responds. If you omit it, there's an awkward 1–2 s pause while the LLM warms up.
• Barge-in works by default since 0.6.3 (VAD activation 0.8, deactivation 0.65 — tuned for room noise). If your callers are interrupting at the wrong moments, see inspect-calls-and-metrics to read the VAD events from the call log.
• Callers cut off mid-pause? In Pipeline mode add a semantic turn_detector (smart-turn or NAMO) and, on noisy lines, the denoiser / high_pass_hz / agc levers — see references/pipeline-mode.md. In Realtime mode use openai_realtime_noise_reduction="far_field" and realtime_turn_detection — see references/realtime-mode.md.
• prewarm_first_message=False is the default since 0.6.3 — it was briefly flipped to True mid-release and reverted because it conflicted with barge-in.

Common errors

Symptom	Fix
RuntimeError: Patter Cloud is not implemented	Don't pass api_key=. Use carrier=Twilio() + phone_number="...".
PatterConnectionError on call(..., wait=True)	No running server to dial through. Wrap the call in async with Patter(...) / await using, or place it while a serve() task is alive.
Agent says nothing on pickup	Missing first_message=. Add one.
LLM responses are 4 paragraphs long	Add "Keep replies under two sentences" to system prompt.
Caller hears garbled audio	Wrong audio rate. Check you're using the right carrier (mulaw 8 kHz Twilio, PCM 16 kHz Telnyx). Patter handles this; user code only breaks it via custom pipeline hooks.
Connection drops after 30 s	Either the tunnel died (use static URL) or barge-in hung. Check MetricsStore for the last event.

Related skills

• setup-patter — install + env vars (run this first if user hasn't).
• configure-telephony — set up the Twilio/Telnyx webhook.
• add-tools-and-handoffs — custom tools, transfer_call, guardrails.
• inspect-calls-and-metrics — live dashboard, cost per call, transcript.

References

• Patter overview: https://docs.getpatter.com/python-sdk/overview · https://docs.getpatter.com/typescript-sdk/overview
• Concepts: https://docs.getpatter.com/concepts
• Agent configuration: https://docs.getpatter.com/python-sdk/agents · https://docs.getpatter.com/typescript-sdk/agents
• Examples: https://docs.getpatter.com/examples