Voice

Voice is two things, and it is worth being precise about which two.

Dictation. Capture an utterance, get the text back. You decide what to do with it.
Read-aloud. Hand us text, get audio back. You decide what gets spoken and when.

It is not a spoken loop. Nothing sends on its own and nothing speaks on its own. stop() returns a transcript — it does not submit it, and the agent’s reply stays silent unless you call speak(). Build on those two primitives rather than on continuous conversation. The division of labour is the same as everywhere else in Patchwork: your app renders the interface, we run the pipeline. Provider keys are ours, the browser only ships and plays audio.

Turn it on

const { send, voice } = useAgent({ voice: true });

That is the whole setup. voice is a single object rather than ten members on the hook, and it does nothing until you call something on it.

A working mic button

import { useAgent } from "@usepatchwork/react";

export function MicButton() {
  const { send, voice } = useAgent({ voice: true });

  async function toggle() {
    if (!voice.isListening) {
      await voice.start();
      return;
    }

    const text = await voice.stop();
    if (text) await send(text);
  }

  return (
    <button onClick={toggle} disabled={voice.isTranscribing}>
      {voice.isListening ? "Stop" : "Speak"}
    </button>
  );
}

start() asks the browser for the microphone, so it must be called from a user gesture. stop() ends the take and resolves with the transcript — empty if the take was silent, cancelled, or failed, which is why the example checks before sending. Telling those three apart is below. cancel() throws the take away. Use it for an escape key or a swipe-away, so a mistake costs nothing.

Reading a reply aloud

const { messages, voice } = useAgent({ voice: true });
const last = messages.at(-1);

<button
  onClick={() => last && voice.speak(last.content, last.id)}
  disabled={voice.isSpeaking}
>
  Read aloud
</button>;

The second argument is an id you choose. It comes back as voice.speakingId while that clip is sounding, which is how you show a playing state on the right message rather than on all of them:

{messages.map((message) => (
  <Message
    key={message.id}
    message={message}
    isSpeaking={voice.speakingId === message.id}
  />
))}

stopSpeaking() cuts playback short.

Replies you have already spoken are cached for the session, so replaying one costs nothing and returns instantly.

Markdown is handled

Agents write markdown, and markdown read literally sounds like punctuation. We render it into something worth hearing before synthesis — you do not need to strip it yourself.

Drawing the microphone

waveformRef attaches a canvas that we draw the live input into while recording:

<canvas ref={voice.waveformRef} width={160} height={32} />

Attach it whenever you like; it idles until start().

Picking a voice

Pass voiceId to use a specific synthesizer voice. Omit it and the platform default applies.

useAgent({ voice: true, voiceId: "your-voice-id" });

To build a picker, list what is available:

const client = usePatchworkClient();
const voices = await client.listVoices();

Each entry is { id, name, category?, preview_url? }. preview_url is a sample you can play in the picker itself.

The state you render against

Field	Type	Means
`isListening`	`boolean`	the microphone is capturing
`isTranscribing`	`boolean`	the take ended, text is on its way
`isSpeaking`	`boolean`	audio is playing
`speakingId`	`string \| null`	which `speak()` call is sounding

isTranscribing is the one people forget. There is a gap between releasing the button and the text arriving, and without a state for it the interface looks broken.

Without the React SDK

Three endpoints, authenticated exactly like the text path — the same subject token, and Patchwork-Connection if you use it. See the API reference for the full shapes.

Endpoint	Does
`POST /v1/voice/transcribe`	multipart `audio` → `{ text, confidence }`
`POST /v1/voice/speak`	`{ text, voice_id?, format?, language? }` → raw audio bytes
`GET /v1/voice/voices`	the voices available to you

speak returns audio, not JSON — it is the one endpoint in the platform that does.

How errors reach you

The voice controls never throw. Every failure goes to the hook’s error and to onError, the same as the rest of useAgent:

const { voice, error } = useAgent({
  voice: true,
  onError: (e) => {
    if (e.code === "voice_unavailable") setMicAvailable(false);
  },
});

This matters for one case in particular: stop() returns "" for a silent take, a cancelled take, and a failed one. The empty string does not tell you which. If you need to tell a silent recording apart from a broken pipeline, watch error — do not infer it from the transcript. start() only touches the browser, so what fails there is permission or hardware. The server is not reached until stop() transcribes, which is where voice_unavailable actually appears — not at the point you asked for the microphone.

When voice is not configured

Every voice endpoint returns 503 voice_unavailable when the workspace’s API has no voice provider configured. This is a normal state, not a fault: a deployment without voice keys still runs agents perfectly well. Treat it as “hide the mic button”, not “show an error”. The text path is unaffected.

Errors

Code	Where it comes from	Means
`voice_disabled`	the SDK, locally	you called `start()` without `{ voice: true }`
`voice_unavailable`	the server, on `stop()` or `speak()`	no provider configured — hide the affordance
`voice_provider_error`	the server	the provider failed; retrying is reasonable
`voice_invalid_format`	the server	`format` was not `markdown` or `plain`
`voice_empty_text`	the server	nothing left to speak after rendering

speak() swallows voice_empty_text deliberately. A reply that is all code or all data renders to an empty script, and a read-aloud button that asked for it should quietly do nothing rather than raise. You will not see that one on error.

Metering

Voice is metered as its own product, so transcription and synthesis appear in Activity as their own entries rather than being folded into the runs they belong to.

Core objects

Conversations

Delivery

Integrating

Operating

Workbench

Turn it on

A working mic button

Reading a reply aloud

Markdown is handled

Drawing the microphone

Picking a voice

The state you render against

Without the React SDK

How errors reach you

When voice is not configured

Errors

Metering

​Turn it on

​A working mic button

​Reading a reply aloud

​Markdown is handled

​Drawing the microphone

​Picking a voice

​The state you render against

​Without the React SDK

​How errors reach you

​When voice is not configured

​Errors

​Metering

Turn it on

A working mic button

Reading a reply aloud

Markdown is handled

Drawing the microphone

Picking a voice

The state you render against

Without the React SDK

How errors reach you

When voice is not configured

Errors

Metering