OpenAI Realtime API: Building Voice Agents That Actually Feel Conversational

Voice has always been the most natural interface, but most voice bots feel clunky. You speak, wait, hear a reply, and can't interrupt. The OpenAI Realtime API changes that. It lets you build speech-to-speech agents that respond in real time, handle interruptions, and call tools mid-conversation.

Here's what it is, how to start, and how to prompt it well.

What is the Realtime API?

Traditional voice agents chain three steps: speech-to-text, an LLM, then text-to-speech. Each hop adds latency and loses tone, emotion and emphasis.

The Realtime API skips the chain. The model works directly with audio, keeps conversation state, and can call tools. The result:

• Low first-audio latency, so replies start fast
• Barge-in, so users can interrupt naturally
• Natural turn-taking, with voice activity detection
• Realtime tool use, such as looking up an order while talking

Use it when the interaction should feel immediate and conversational. If you need a chained pipeline instead, OpenAI's voice agents guide compares the architectures.

Choosing a model

| Model | Best for | | --- | --- | | gpt-realtime-2 | The strongest reasoning, tool use and instruction following. It can think before it speaks, and has a 128k context window (up from 32k). | | gpt-realtime-1.5 | A fast, reliable, non-reasoning speech-to-speech model. |

For most new projects, start with gpt-realtime-2.

How it works in the browser

The usual flow has four steps:

1. Your server creates an ephemeral client secret for the session.
2. Your frontend creates a RealtimeSession.
3. The session connects over WebRTC (browser) or WebSocket (server).
4. The agent handles audio turns, tools, interruptions and handoffs.

The ephemeral key matters: it keeps your real API key off the client.

With the Agents SDK, the core is small:

import { RealtimeAgent, RealtimeSession } from "@openai/agents/realtime";

const agent = new RealtimeAgent({
  name: "Assistant",
  instructions: "You are a helpful voice assistant.",
});

const session = new RealtimeSession(agent, {
  model: "gpt-realtime-2.1",
});

await session.connect({
  apiKey: "ek_...(ephemeral key from your server)",
});

Keep audio transport in the session layer and business logic in the agent. Tools, handoffs and guardrails attach to the RealtimeAgent just like a text agent.

Prompting a reasoning voice model

gpt-realtime-2 isn't a basic voice bot. Treat it as a reasoning agent and prompt it with intent. A few lessons from OpenAI's guide:

Start simple. Begin with a minimal prompt, run evaluations, then add instructions only for behaviors that fail.

Use short, labeled sections, for example Role, Tone, Language, Tools, Unclear Audio and Escalation. The model finds rules faster.

Set reasoning effort low. low suits most production agents. Go minimal for simple commands, and medium or higher only when multi-step reasoning pays for the extra latency.

Be precise, not loud. The model follows wording literally. Too many always, never and must rules cause rigid behavior and conflicts. Scope rules narrowly: "ask for confirmation before write actions" beats "always confirm everything".

Preambles: stay responsive

Preambles are short spoken updates like "I'll check that order now." They keep the agent feeling alive during tool calls or longer reasoning. Use one short sentence, describe the action rather than the thinking, and skip them for direct answers.

Tools: define when to act, ask or confirm

| Tool type | Behavior | | --- | --- | | Read-only lookup | Call when intent and fields are clear | | Account changes | Confirm first | | Payments, cancellations | Confirm amount, target and consequence | | Irreversible actions | Confirm explicitly, offer escalation |

Also tell the model what to do when a tool fails: explain simply, don't expose raw errors, retry once, then offer an alternative.

Exact entities: order IDs, emails, codes

Voice makes exact values hard. Collect one value at a time, read numbers back digit by digit, have users spell emails, and confirm before any tool call. Never act on guessed or partial values.

Silence and unclear audio

Voice agents want to answer everything, including TV noise and side conversations. Give the model a no-op wait_for_user tool so it has a valid way to stay quiet. For genuinely unclear speech, have it ask for a repeat rather than guess.

Language and accent

Control them separately. A user with a strong accent still expects their own language back. Set a default language and switch only on an explicit request or a substantive utterance in another language.

Production notes

• Safety identifiers. If you identify end users, send a stable, hashed ID in the OpenAI-Safety-Identifier header. It helps target enforcement to one user rather than your whole organization.
• Migrating from beta. Remove the OpenAI-Beta: realtime=v1 header, create credentials with POST /v1/realtime/client_secrets, use /v1/realtime/calls for WebRTC, and update to the GA event names such as response.output_audio.delta.
• Long sessions. Give the model structure: current state, authoritative sources and stale background, so it knows what to trust when they conflict.

Where this fits for businesses

Realtime voice suits customer support, appointment booking, order lookup, internal helpdesks and multilingual assistants. Anywhere a person would otherwise wait on hold or navigate a phone menu, a good voice agent can resolve the request in a single natural conversation.

The key to success isn't just the model. It's tight prompts, clear tool rules, exact-value handling, and relentless evaluation.

Getting started

1. Build the browser example above with the Agents SDK.
2. Add one read-only tool.
3. Write a minimal prompt, then test with real conversations.
4. Add rules only where behavior fails.

Voice agents are finally ready to feel like conversations. Now is a good time to build one.

Try the demo

I built a working example with the Realtime API. The full source is on GitHub: nisalgunawardhana/open-ai-realtime.