I Gave Claude Code a Realtime Voice Mode on My MacBook

That's me building a small web page with Claude Code without touching the keyboard. I ask for the page, it asks out loud before running a command, I say "yes", and when I change my mind halfway through its answer I just talk over it and it stops.

The voice part runs entirely on the laptop: listening, working out when I've finished, transcribing and speaking take about 1.4 GB, with no PyTorch and no cloud speech service. Claude Code is the same Claude Code I use every day, on my normal subscription, driven over ACP (the Agent Client Protocol).

Last week's post, A Local Voice Agent on a MacBook in Under 2 GB, built the voice loop. This one points it at a coding agent, which needed three more things: a project to work in, a way to say yes or no to the agent's permission requests, and something to fill the silence while it thinks.

Try it #

git clone https://github.com/johnrobinsn/macvoice && cd macvoice
uv sync && uv run macvoice download              # about 0.85 GB of models, once
npm i -g @agentclientprotocol/claude-agent-acp   # Claude Code over ACP

uv run macvoice run --llm acp --cwd ~/demo-site

--cwd is the folder the agent works in, created if it doesn't exist. Use laptop speakers if you like: echo cancellation is on by default, so you can interrupt it without headphones.

Asking permission out loud #

Coding agents ask before they do anything risky, and a voice mode has to handle that, or you end up reaching for the keyboard every thirty seconds. When Claude Code wants to run a command, macvoice says something like:

"Claude wants to create site folder with an index page. Allow?"

and prints the exact command:

? Claude wants to run: mkdir site && echo hi > site/index.html
    (say yes or no, or press y / n / a (always))

The spoken version uses the agent's own plain-English description of the command where it gives one. Reading mkdir site && echo hi > site/index.html aloud sounds terrible, so the details stay on screen.

Then it listens for the answer. The rules are deliberately cautious:

• "Yes", "sure", "go ahead" allow it once. "Always" allows it for the rest of the session. "No", "don't", "stop", "wait" decline.
• Anything negative wins. "No, yes" is a no. A misheard answer can only ever decline.
• Unclear answers are asked once more, then declined. So is 20 seconds of silence.
• The question can't answer itself. Echo cancellation should keep the agent's voice out of the mic anyway, but nothing heard while the question is playing counts as an answer.
• You can also press y, n or a. Keys typed before the question are thrown away, so nothing you pressed earlier can approve it.

To approve fewer things, use the agent's own modes. Claude Code's, as it reports them over ACP:

$ uv run macvoice modes
 * default  (Manual)  Always ask before making changes
   acceptEdits  (Accept edits)  Automatically accept all file edits
   plan  (Plan)  Create a plan before making changes
   auto  (Auto)  Claude handles permission decisions
   bypassPermissions  (Bypass permissions)  Accepts all permissions

--mode acceptEdits is a nice middle ground for voice: file edits go through, and shell commands still ask. Nothing here is specific to Claude. Modes and permission requests are part of ACP, so the same flags work with any ACP agent, and the prompt uses the agent's own name ("Gemini wants to...").

Filling the silence #

The voice pipeline hands off to the brain about 350 ms after you stop talking. With a small local model, the first word of the answer arrives 0.6–0.8 s after you stop. Claude Code is a different animal. In my tests it took 3 to 4.5 seconds to start answering simple requests, and longer once it's reading files or running commands.

Four seconds of silence in a conversation feels like the other side hung up. So when a reply hasn't started after 1.2 seconds, macvoice plays a soft blip once a second until the first word arrives, then cuts it off mid-blip. It plays through the same output as speech, so echo cancellation removes it from the mic, and you can still talk over it. --no-thinking-tone turns it off.

To be clear about where the time goes: the voice pipeline is fast, and most of the wait is the agent thinking and working. The tone doesn't make Claude faster. It makes the wait feel like the agent is working, not broken.

Short answers, not code #

A coding agent's natural output is markdown and code blocks, which are miserable to listen to. ACP has no system prompt, so macvoice sends a short brief with the first message: replies are spoken, keep them to one to three short sentences, and when you change files or run commands, say briefly what you did instead of showing the code. The result sounds like a colleague reporting back, while the code lands on disk where you can read it.

Interrupting #

This is the part that makes it feel real-time. While Claude is talking, start speaking and it goes quiet within a fraction of a second. It needs a quarter second of continuous speech, so a cough doesn't count, and it ignores the first half second of a reply, while the echo canceller is still learning the room.

Going quiet isn't the same as stopping. Talking over someone usually means "I get the idea, let's move on", not "undo what you're doing", so Claude keeps working. Its full reply still lands on screen, and what you said goes to it as the next prompt once its turn ends. Two soft blips tell you when it's free, if you haven't already moved on. When you really do want it to stop, say just "stop" (or press Ctrl+C), and macvoice cancels its turn over ACP (session/cancel). Permission questions are the exception to going quiet: those are always spoken.

Type or talk #

Some things are easier to type: a file path, a variable name, a stack trace. So the terminal has an input box under the conversation, and typed and spoken prompts go to the same Claude Code session. It works like you'd expect from Claude Code itself: Enter sends, Shift+Enter adds a line, the up arrow brings back earlier prompts, Ctrl+C stops the reply. Tool calls scroll past as one-liners (▸ Edit index.html, ✓), and replies are rendered as markdown on screen while the voice skips the code blocks.

Why ACP #

macvoice doesn't screen-scrape a terminal or wrap a CLI. It speaks the same JSON-RPC protocol that editors use to host coding agents: start a session in a folder, send a prompt, stream the reply, answer permission requests, cancel. That's about 300 lines of standard-library Python, and it works with Claude Code, Gemini CLI, Codex, OpenCode and the rest. I wrote more about why that shared surface matters in ACP: the shared surface for coding agents.

Caveats #

• Permissions are yours to manage. --permissions allow approves everything without asking. Only use it in a folder you don't mind losing.
• One-word answers are short. "Yes" is right at the edge of what the endpointer keeps (250 ms of speech). If a quick "no" gets missed, you'll be asked again, and silence declines anyway.
• It's a Mac project. The voice stack uses MLX and Apple's voice processing. The ACP side is portable; the audio side isn't, yet.

Beyond one agent: vibr8 #

macvoice is a tech project on GitHub: one agent in one terminal, small enough to read in an afternoon, and that's as far as I want it to go. vibr8 is the product, the thing I actually use every day, and I'm getting it ready for other people to install. It runs on Linux today with a GPU doing the voice work; the first installable release is coming to the Mac, with a full voice pipeline that runs locally on your Mac (the same Apple Silicon stack as this post, no GPU server and no cloud speech service), and other platforms follow.

The difference is what sits between your voice and the agents. In vibr8 you talk to Ring Zero, a meta-agent that runs several coding sessions at once: it starts them, sends them work, relays their permission requests, and tells you when one finishes or gets stuck. Because it can see all of those sessions, it can answer "what did the API session just do?" with a sentence that knows the context, instead of reading you a transcript. Ring Zero: The Meta-Agent for Coding Agents explains the idea, and Anatomy of a Ring Zero Session walks through a real one.

The voice side goes further too. A guard word means it only acts when you address it ("vibr8, ..."), so it's comfortable to leave listening all day (Towards an always-listening voice agent). Note mode is for thinking out loud: say "vibr8 note", ramble for as long as you like while it stays quiet, say "vibr8 done", and Ring Zero gets the whole thing as one voice note to act on. And you can just ask vibr8 to put something on your clipboard, a command, a snippet or an image, ready to paste wherever you're working.

Bring your own coding agent: the sessions underneath are ACP, so Claude Code, OpenCode and the rest plug in the same way (ACP: the shared surface for coding agents). I recommend Claude Code for now. Why one person ends up running a fleet of agents at all is in Software Factories.

If macvoice is talking to one agent, vibr8 is talking to the team. More soon, starting with the Mac.

The code for this post is at github.com/johnrobinsn/macvoice. The first post has the details of the voice pipeline: why Smart Turn ends turns in under 300 ms where a silence timer takes 1.5 seconds, and how echo cancellation makes barge-in work on laptop speakers.

If you try it, let me know how it goes, especially which agent you pointed it at.

Interested in being a beta tester when Ring Zero (vibr8) is released? Sign up for the mailing list now.

Follow along at storminthecastle.com — or reply if you're building in this space.

• Previous: A Local Voice Agent on a MacBook in Under 2 GB