GPT-Live-1 Voice Agent Architecture: Connecting Speech and Background Logic

Building a voice assistant has long come down to chaining three separate components: speech-to-text (STT), a large language model (LLM), and text-to-speech (TTS). In practice, this cascade creates noticeable latency and complicates live conversation management. When a speaker pauses, changes their thought, or interrupts the assistant, developers must manually track state, reset the audio stream, and synchronize context across three distinct services.

On September 10, 2026, OpenAI made the GPT-Live-1 model available in the API (official release). Rather than stitching services together step-by-step, the model offers a full-duplex audio layer: it can simultaneously ingest an incoming audio stream and generate spoken responses.

Separating Speech and Computation

When backend tasks take time to complete, they are separated from the speech layer to maintain ongoing voice interaction. A practical architecture built on GPT-Live-1 relies on a separation of responsibilities:

1. Voice frontend. The model processes inbound and outbound audio concurrently. According to the developers, this approach handles background noise, pauses, and interruptions better than an STT–LLM–TTS cascade, while also supporting native turn detection.
2. Background backend. Complex data analysis, database queries, and tool calling are offloaded to dedicated text models or external agents.

This setup is designed to help the interface maintain conversational engagement and avoid hanging in silence while the background service prepares a substantive response. Even so, actual latency and transition smoothness must still be verified on your specific stack.

How Task Delegation Works

Voice and the backend operate asynchronously. When a user requests an order status or a repository search, the application routes the task to its backend.

The official documentation provides a conceptual example of this coordination using the Codex SDK:

import { Codex } from "@openai/codex-sdk";

const thread = new Codex().startThread({
  workingDirectory: "./repo",
  sandboxMode: "read-only",
  approvalPolicy: "never",
});

async function answer(live, delegationId, context) {
  const { finalResponse } = await thread.run(
    `Answer the latest question using this repo.
     Reply in two short spoken sentences.\n${context}`
  );

  live.send({
    type: "session.commentary.append",
    delegation_id: delegationId,
    content: finalResponse,
  });
}

The code above is only an official integration excerpt: connection initialization and delegation event handling are omitted, so it is not intended to run standalone.

This snippet illustrates the general principle of interaction: the application passes the turn’s context into the tool worker thread and returns the resulting answer back to the audio session via a session.commentary.append event. The voice channel remains active, allowing the agent to deliver a brief introductory phrase if needed while the backend finishes its computations.

Architecture Selection Criteria

As of the publication date, the voice layer costs $0.05 per minute, which does not include expenses for background models or tool calls. This architecture is relevant in scenarios where conversational continuity is critical:

• Phone calls and appointment booking. Workflows where any unnatural pause between turns prompts callers to ask if they are still being heard.
• Unstructured support conversations. Dialogues where speakers frequently hesitate, rephrase thoughts mid-sentence, or speak in fragments.
• Voice-driven pair workflows. Interactive collaboration on code or documents where users think out loud and prefer not to wait for every single turn to conclude.

If the task is limited to rigid command input, note dictation, or standard form filling, comparing this approach against a traditional STT-based cascade makes sense.

Getting Started with Prototype Verification

The steps below are recommendations for prototype verification, not a report of completed tests. Before moving a workflow to the new architecture, completing these baseline steps is worthwhile:

• Measure backend response times. If querying your database or external model takes time, configure the voice layer to acknowledge that the operation has started with a natural, brief phrase instead of remaining silent.
• Verify behavior in noisy environments. Test the prototype under real-world conditions: ambient street noise, background conversations, or unstable microphone input.
• Constrain the background model’s output format. In the backend system prompt, explicitly specify that responses must fit within one or two concise sentences suitable for spoken delivery.
• Set session duration limits. At a rate of $0.05 per minute for the voice layer, setting programmatic limits on maximum test call duration helps prevent unnecessary charges if a client hangs.

Following this sequence helps surface actual tool latencies early and allows prompt refinement before scaling the system.