Towards an always-listening voice agent

This post is part of an ongoing series on vibr8, a personal AI meta-harness that I'm building.

Last week's Anatomy of a Ring Zero Session had one line about the voice pipeline: it locks onto your voice specifically, so the car radio, the kitchen fan, the news playing in the background, or someone else in the room talking doesn't derail the interaction. That's a lot to hide behind one line. This post opens it up.

What "always-listening" would actually require #

The phrase gets used loosely. What it should mean is: the agent is present in your environment the way a person is. You speak, it listens, it understands whether the speech was for it, and it responds — with no wake word, no button, no rehearsed cadence. The interaction is conversational rather than transactional.

That vision has to survive contact with a real room. Your kitchen has a fan and a radio. Your car has road noise and a passenger. Your office has other people talking who are not talking to the agent. Your phone microphone is not a studio mic. The device is small, its battery is finite, and most of the interesting compute cannot live on it.

To feel like a person in the room, the system has to solve at least four problems at once. It has to hear you through whatever noise is in the room. It has to know when a given utterance is meant for it and when it isn't. It has to do that reliably enough that false positives don't erode your trust in a week. And it has to do all of that with a realistic split between what the phone can do (capture, stream) and what a server can do (currently GPU-required for the heavier work).

Every voice product on the market handles a subset. None handle all of it well. The gap between "recognizes speech" and "is present in the room" is where most of the interesting work still lives.

The hard problems inside the vision #

Naming them helps, because they look like one problem from the outside and are actually four.

Whose voice is this? Even in a quiet room, more than one voice is usually reachable. In a family kitchen, in a car with a passenger, in a coffee shop, the microphone catches multiple speakers at once. Before the agent can do anything else, it needs some way of picking your voice out of a mixture.

Was that utterance for me? Even after you've isolated the target speaker, most of what they say all day is not directed at the agent. It's directed at their kid, their phone call, the dog, themselves. Getting this wrong in either direction is bad — an agent that responds to everything is exhausting, an agent that misses you when you actually address it is worse.

Noise robustness. Real environments are not the clean recordings that acoustic models were pretrained on. Fan hum, car cabin, HVAC, another TV in the room. Speech recognition degrades unevenly across these; models can look great in a demo and fall apart in a kitchen.

False-positive management. If the agent thinks it was addressed and it wasn't, what happens next? Speaking out of turn is a visible failure. So is dispatching an action nobody asked for. The cost of a false positive scales with what the agent can actually do — a chatbot that says the wrong thing is embarrassing; an orchestrator that files a PR is worse.

vibr8 has gone deep on two of these — target speaker identity and noise robustness. Addressee detection and false-positive management remain open.

Where we are: target speaker extraction #

Target speaker extraction — TSE, for short — is the name for the "whose voice is this?" piece. Given a segment of audio that might contain multiple voices plus noise, and given a reference for who you're trying to hear, produce audio that contains only that speaker.

It is not the same as speaker verification, which just answers yes/no, was this you? Verification acts as a filter — it can reject a segment, but it can't clean it. TSE is generative. It reconstructs the target speaker's audio out of the mixture. What comes out the other side is a version of the input where the other speakers and much of the background have been attenuated.

vibr8 uses a model called BSRNN — a band-split recurrent network — from the open-source WeSep project. It runs on the server GPU as a lazy-loaded PyTorch model. The audio it operates on is 16 kHz mono float. The conditioning signal — the "who to extract" input — is a 192-dimensional embedding produced by an ECAPA-TDNN speaker encoder that comes bundled with the same WeSep checkpoint. That embedding is what tells the separator whose voice to keep.

The way you get that embedding is enrollment. When you sit down to teach vibr8 your voice, the pipeline enters an enrollment mode — same WebRTC transport as normal sessions, different server-side handling. You speak; the server transcribes normally through Whisper, but on every confirmed speech segment it also computes two speaker embeddings from that segment — one from SpeechBrain's ECAPA-TDNN, one from WeSep's — and compares them against any existing profiles. When you save the fingerprint, both embeddings get stored on the entry, together with an optional reference audio clip and a device-or-environment label. A user typically ends up with multiple entries per profile: one from the phone, one from the earbuds, one from the desk mic. Different capture devices produce measurably different embeddings for the same voice — mic frequency response, distance, digital filtering, and acoustic environment all shift the fingerprint. Storing per-device entries lets the speaker gate match reliably regardless of which path is active in the moment.

Two embeddings is deliberate. The SpeechBrain one and the WeSpeaker one live in different vector spaces — different training data, different normalization conventions — and they are not interchangeable. The SpeechBrain vector is L2-normalized and used for the speaker gate: given a new segment, does its embedding match your stored profile closely enough to be you? The WeSpeaker vector is stored un-normalized, which is what the BSRNN separator was trained to receive; it feeds into TSE as the conditioning input.

In a real session, WebRTC delivers 20 ms Opus frames from the browser at 48 kHz stereo. The server accumulates roughly 160 ms of frames, downmixes to mono, resamples to 16 kHz, and runs a two-stage voice-activity detector (an RMS threshold followed by Silero VAD) to find segment boundaries. When a segment closes, the speaker gate runs first: compute the SpeechBrain embedding, compare against the stored fingerprint entries, reject the segment if the best cosine similarity falls below threshold. If the segment passes the gate and TSE is enabled, the stored WeSpeaker embedding for the matched entry is handed to BSRNN along with the raw 16 kHz audio; the separator returns a peak-normalized cleaned version. That cleaned audio — not the raw audio — is what Whisper transcribes. If TSE is disabled, or the matched entry has no WeSpeaker embedding, or the separator errors, the pipeline falls back to raw audio.

Concretely, this is what gives you the property from last week's post. A car radio playing music in the background is a competing signal that the separator is trained to attenuate. A kitchen fan is stationary noise; the separator suppresses it. Someone else in the room asking a question at the same time as you: the gate rejects segments where that voice is dominant, and TSE cleans segments where your voice is dominant but the other voice is present. The result feels less like "the agent hears everything" and more like "the agent hears me."

The result works well enough in practice to feel a little magical. You can chat to the agent in a busy cafe, with the news on TV in the next room, or with the car radio playing — and it just works. Competing sounds don't derail; other people in the space aren't confused for you.

The real limit is the one that connects back to the two problems we haven't solved yet — addressee detection and false-positive management. Start talking to someone else in the room and the agent still picks up your voice and treats it as addressed to it. TSE only answers "here is a cleaner version of what you said." It doesn't answer "was that meant for me?"

The addressee problem #

That distinction is the one worth staying on. Target speaker extraction solves whose voice. It does not solve was this utterance meant for the agent? Those are different problems and they need different machinery.

Addressee detection is genuinely hard for reasons that are less about audio and more about context. Whether an utterance was directed at the agent depends on where you're looking, what you just said to it, whether anyone else is in the room, whether you've raised your voice or lowered it, and whether the last thing the agent said left an open thread. Humans use all of these signals fluently. Software has mostly not tried to.

The literature covers a handful of approaches — head-pose and gaze cues, prosodic features, conversation-state models, LLM classifiers that read the transcript in context — and none of them is a solved problem in the way that verification or extraction now are. Getting it wrong in either direction is expensive, which is why the deployed systems mostly duck the problem: they require a wake word, or a button press, or a specific frame of interaction that makes the addressee unambiguous.

vibr8 ducks it too, currently, but visibly. The mechanism is a guard word: the transcript out of Whisper is scanned for the string "vibr8" or "vibrate," and if the server is in guard mode the transcript is only routed onward when the guard word appears (optionally followed by a known command). This is deterministic string matching on the output of a speech recognizer that is always running — not a separate wake-word acoustic model. That distinction matters. There's no low-power always-on detector; the STT stack is transcribing continuously, and the guard word is a filter on the result. It works, but it's a placeholder for real addressee detection, and the code is honest about that.

The direction points at replacing that placeholder with something more like intent classification over the current conversational context — a classifier or LLM check that takes the recent transcript history, the current session state, and the utterance itself, and returns a calibrated probability that the utterance was addressed to the agent. Doing that well without a wake word is a research problem, not a plumbing problem.

Worth naming an implication: in the target design, guard-word doesn't disappear. It becomes a mode you switch on when the room is full of people you're actually talking to and you don't want the agent responding to every sentence as if addressed to it. The always-listening direction is about making addressee detection reliable enough to be the default — not about eliminating your ability to explicitly gate the agent when you want that.

In either mode, when voice is on the pipeline is always streaming to the server; the difference between guard-word and always-listening is what happens with the transcript after, not whether audio is captured.

The switch between modes is voice-based: "vibr8 guard" toggles into guard mode, "vibr8 listen" back to always-listening. Mode is a first-class thing the user names, not a hidden setting.

What still needs to happen #

The gap between "listens to you cleanly" and "is present the way a person is" has recognizable pieces.

Wake-word-free interaction, with an addressee classifier good enough that the false-positive rate is negligible in a busy household. Multi-speaker context that goes beyond a single enrolled target — households have more than one person, and each person needs their own profile and their own routing rules. Cross-modal context — calendar, presence, screen focus, gaze — feeding the addressee classifier so it has more than just audio to work with. And an explicit mode model that the user controls: always-listening as the default (addressee detection deciding what's for the agent), guard-word as a switchable fallback for social contexts where you're actively conversing with humans. The user's model of the system matches what the system is actually doing.

None of these are fully solved, but are open areas of research.

One more mode worth naming: dictation #

Not every voice input is a conversation with the agent. Sometimes you want to dictate — an email, a note, a message — without any agent trying to interpret or route it.

Vibr8 has a third mode for this: note mode. You say "vibr8 note" and everything you speak from that point gets captured, transcribed, and returned as a voice-note object — until you say "vibr8 done", which closes the note. Nothing is routed to an agent. Nothing is interpreted. It's just becoming a thing you can hand to an agent when you're ready.

Then you act on it. "Mail this to Sarah." "Save this to my Google Doc." "Turn this into a PR description." The note is a first-class object; routing happens after capture, not during it.

That distinction matters. Speech-to-text is a service. Routing speech through an agent is a very different service. Note mode separates them so you can use each on its own terms — dictate cleanly without an agent trying to help; hand off the captured content when you're ready.

It's a small feature that removes a lot of friction. And it reinforces the broader modes idea: your voice pipeline should have distinct behaviors you can choose, not one behavior that tries to guess.

Close #

Anatomy of a Ring Zero Session had one line about locking onto your voice. I hope this article sheds some light on how I've tackled the problem in my voice-enabled meta harness.

If you're building voice-first multimodal agents and any of this maps to problems you've been thinking about or working on I'd love to hear your thoughts.

Interested in being a beta tester when Ring Zero (vibr8) is released? Sign up for the mailing list now.

Follow along at storminthecastle.com — or reply if you're building in this space.

• Previous: Anatomy of a Ring Zero Session