bullshitai.dev

RESEARCH Experiments in AI voice

Full-duplex AI voice: the app I needed was already there

I wanted to talk to my Hermes agents while they did things for me. Several days of browser experiments, Discord voice and a PWA later, a third-party iPhone app covered much of what I wanted on the phone. My own client still has a reason to exist, just a smaller one.

A cartoon white goose with a long orange beak and an amused, half-lidded look, wearing a black voice headset with a microphone.

Listen: two AI hosts discuss this article

An AI-generated two-host discussion of this article. It is a separate adaptation, not a verbatim narration.

Download the audio (M4A, 12:46, 6.4 MB)

I wanted to talk to my agents without treating the conversation like a sequence of voice notes. Speak, get an answer, interrupt when necessary, ask an agent to do something and keep talking while it works.

That sent us into browser audio, a Discord voice bot and a small web client. We got parts of it working. We also got silence, dropped calls and tests that sounded more reassuring than the experience deserved.

Meanwhile, we had Conduit connected to my Hermes server. Someone else’s native iPhone app did the phone part reasonably well. My first draft still talked about building a PWA for that same job.

There is something funny about spending days investigating how to build a thing while an app that covers much of the original need is already available. Conduit didn’t make the experiments pointless. It changed which of them I wanted to keep pursuing.

This is where that investigation got to. The Discord version never became something I could rely on, the phone has an alternative, and the custom desktop client is still an experiment.

01 / What do I still need to build?

One wish, different places to use it

On my phone

Use Conduit

Someone else's native app covers much of what I wanted.

Useful in my own use
In Discord

Leave this experiment paused

A few spoken exchanges, then a dropout. We did not activate the candidate fixes.

Paused
At my desktop

Keep exploring a smaller client

Voice beside my work, with visible tasks and the right agent context.

Still an experiment
Spoken description The original wish split into different needs. Conduit covers the author's phone use reasonably well. The Discord experiment is paused after an unsuccessful human conversation. The custom desktop client continues as a narrower experiment, not a replacement phone app. These are current choices, not a reliability ranking.

I wanted a conversation, with an agent behind it

Push-to-talk gives you a button that decides whose turn it is. A speech-to-text, model, text-to-speech pipeline can remove the button and still make you wait through each stage.

Full duplex lets audio travel both ways during the same live session. The model can hear you while it speaks. In a useful conversation, you should also be able to interrupt it and hear it stop.

I wanted that through the ChatGPT subscription I already pay for. Codex Voice uses an unofficial subscription transport, and that was the route we explored. I wasn’t looking for a paid realtime API or a fallback to a sequential voice pipeline.

The unofficial part matters. This isn’t a supported public API, it could change, and the model name sent in a request doesn’t tell me which model actually answered.

Talking was only part of the idea. I also wanted to ask an agent to work on something without stopping the conversation until the task finished. That’s where the experiments started involving more than audio.

The browser worked well enough to try Discord

On 20 October, Codex Voice got a GPT-Live mode. The browser sent audio over WebRTC. A small Rust server handled the subscription login and control connection.

A real call went through. We supplied a public speech recording, got a correct transcription and heard a spoken answer. Pause and resume worked. Stop released the microphone and closed the connection. Those were useful results, though recorded speech isn’t a human testing interruptions. Accepting the privacy notice also wasn’t a guarantee that nothing would be retained.

Discord seemed like a useful next place to try it because my agents were already there. Hermes managed the Discord connection and who could speak to the bot. A separate Rust process handled the audio, using Songbird, Discord’s DAVE encryption, Opus and WebRTC to the voice model.

Hermes had to renew a short lease to keep the media process in the channel. If the controller stopped responding, the bot was supposed to leave rather than keep an open microphone indefinitely.

We had several failed probes before a run called live-7 passed. Generated speech went in, decoded model audio came back, a second utterance worked in the same session, and the logs showed input and output overlapping. The subscription transport could carry the exchange.

Then I tried it. I got a few spoken exchanges before the bot dropped out of the channel and didn’t come back.

The generated-speech test had answered a narrower question than mine. It showed that the audio could get through. It hadn’t shown that I could have a conversation and recover when something went wrong.

02 / What did the tests actually answer?

Audio got through. The rest varied.

Browser

20 October

Audio exchange
Recorded public speech and a real spoken reply.
Human interruption
Not tested by a person.
Background agent work
Not attached.
Stop and recovery
Clean stop. Reconnect was not covered.

Discord

Paused

Audio exchange
Generated speech passed. I had a few real exchanges before the dropout.
Human interruption
Code stopped sending audio. Human-heard interruption was not established.
Background agent work
Not tested.
Stop and recovery
Dropped out in my use. Candidate lifecycle fixes stayed offline.

Desktop

25 October

Audio exchange
Generated speech, real transport and real Hermes.
Human interruption
Not tested by a person.
Background agent work
Real file task with the call open. Result appended; correct spoken answer unverified.
Stop and recovery
Clean stop. One immediate reopen timed out; a later one passed.
Spoken description All three custom routes demonstrated audio moving in both directions, but under different conditions. None established reliable human interruption. Only the desktop slice demonstrated a background file task while voice stayed open, and its result was appended rather than verified as correctly spoken. Passing the audio test did not establish a finished conversation experience.

We found bugs, but didn’t prove why my call dropped

The service journal repeatedly showed standby_lease_expired and restarts. Follow-up tests reproduced two problems in the production code.

One controller operation could wait indefinitely before renewing a 1.2-second lease. If it stalled, the lease expired. Separately, a failed command to leave Discord could prevent the remaining cleanup, leaving the old connection owned and potentially interfering with a rejoin.

The candidate fix moves reconciliation into its own cancellable task and runs cleanup even if leaving or revoking fails. Both regression tests went from failing to passing.

That doesn’t establish what caused my call to drop. An earlier observation of a Python garbage-collection pause around a slow status read wasn’t enough to blame GC either. We had reproduced bugs and a candidate with bounded, metadata-only lifecycle logging. We hadn’t activated it or repeated my human test with it. The Discord work stayed paused.

There were other limits to the test results. The interruption check showed our code stopping encoded audio, not a person hearing the bot stop mid-sentence. A 61-second offline codec run and a mock advancing a virtual clock through two hours weren’t long human conversations.

I don’t want to throw those results away. I also don’t want to read a passing engineering report and pretend it answered the question I started with.

Conduit covered the phone use case

The PWA plan was a small Rust service that would serve the interface, own the voice session and start Hermes work separately when the voice model delegated a task. The call would keep going. A result would return only to the still-open session that requested it.

We tested that arrangement locally against the real Hermes core with a synthetic model and voice peer. Duplicate delegation events started one run, newer sessions didn’t receive stale results, and cancelling a task didn’t close voice.

Conduit, by kaishi00, already offered a native iPhone app with a Hermes server plugin. We had it connected, and for talking to my agents from my phone, it worked reasonably well. That’s my experience using it, not a feature-by-feature reliability test.

My agents also read its code at pinned commits. The phone can create a background Hermes session, submit a prompt, watch for completion and add the result to the voice conversation that requested it. Work attached to an existing chat can go there instead. Stopping voice and cancelling a task are separate actions.

That was close to the arrangement we had been working towards, with an actual native iPhone app around it. Well, fuck. Credit to kaishi00. It made much less sense to keep building a phone client just because we had started one.

There are still things I wouldn’t claim from reading its code. The ordinary phone task ledger lives in memory. Optional server-side callback watches persist, but that alone doesn’t establish task or call recovery. A persona export isn’t the full operational context of an agent. We didn’t measure restart or background reliability.

The subscription covers the voice connection. Delegated Hermes work still uses ordinary backend inference. Finding a useful app didn’t make that work free.

03 / Working while we talk

Voice keeps going while Hermes works

Swipe sideways to see the client and Hermes.

Sequence diagram with four participants: me, the voice model, the client and Hermes. I ask the voice model for work. It delegates the task to the client, which starts a Hermes session and submits the prompt. In parallel, I keep talking with the voice model while Hermes runs the task. Then one of three things happens. If the result arrives while voice is open, Hermes returns the result for this delegation to the client, the client appends it to the original voice session, and the voice model gives a short summary. If I stop voice first, the client closes the voice session and the Hermes task keeps running. If I cancel the task first, the client cancels that Hermes run.
  1. I ask the voice model for work, and it delegates the task through the client.
  2. The client starts a Hermes session and submits the prompt. The conversation keeps going while Hermes works.
  3. If voice is still open when Hermes finishes, the result comes back to the client tied to that delegation, and the client appends it to the voice session that asked. The voice model can then give a short summary.
  4. Stop voice and Cancel task are separate actions, available while Hermes works. Closing voice does not cancel the task, and nothing guarantees a result is spoken after the call ends.

This is the interaction we read in Conduit's code, not proof that my desktop client voiced task results correctly. Appending a result is not proof that the right result was spoken.

Spoken description I ask the voice model for work. It delegates through the client to Hermes, and the conversation continues in parallel while the task runs. The result returns through the client, bound to the original delegation, and is appended to the voice session that requested it, so the model can summarize it. Stopping voice and cancelling the task are separate choices. Closing voice does not cancel the task, and nothing guarantees a result is spoken after the call ends. This pattern comes from reading Conduit's code. The desktop experiment demonstrated a real file task and append, but did not prove that the correct result was spoken.

The desktop experiment is a different question

I still want a browser client beside the rest of my desktop work. I want to see background tasks, know which agent profile I’m speaking to and understand what context it has. Those are reasons to keep investigating a desktop client. They aren’t features I can claim to have finished.

The Rust implementation stays because the transport code already exists. Reusing it is cheaper than starting again. I haven’t measured a latency or build-speed advantage, and we haven’t chosen a rewrite.

On 25 October, a private desktop version ran with the real served interface, subscription-backed GPT-Live and Hermes. Automated tests with generated speech exercised audio in both directions and a spoken request that started a real Hermes file task while the call remained open. Typed file work survived a history reload. Stop released the microphone and session.

The client appended the task result to the voice conversation. We still hadn’t proved that the model spoke the correct result. The first check accepted an assistant caption and loud audio after the append, which could have been speech about something else. Review rejected that stronger claim.

One immediate attempt to reopen after Stop timed out. A later attempt passed. We didn’t know why. Human interruption quality and my client’s iPhone background audio or AirPods behaviour remained untested. Conduit working on my phone didn’t answer those questions for my PWA.

My first hands-on desktop session exposed another mismatch. I asked for coding fixes, but this version used a limited Hermes test backend with file tools only and built-in memory off. It couldn’t hand work to a coding manager because that capability wasn’t connected. A completed task in that interface wasn’t necessarily the work I had asked for.

Context occupancy wasn’t visible. Voice choice and profile identity were missing or unreliable. A read-only review also found file-tool records reporting edits where profile memory lives, despite built-in memory being off. At the time of this account, I hadn’t checked or repaired that state. Switching off a memory feature doesn’t restrict where file tools can write. That needs proper isolation before the client gets more capabilities.

What I’m keeping from this

Conduit covers enough of the phone use case that I don’t need to rebuild it right now. The Discord candidate stays paused. The desktop browser remains worth trying for the parts I want beside my work, but it still needs to speak the right task result, connect to an appropriately isolated agent backend and survive me interrupting it.

The useful distinction from these experiments is between getting audio through and having something I want to use. We got audio through in several ways. We didn’t get our own clients to something I want to use.

I started because I wanted to talk to my agents. Using someone else’s app for part of that is a good outcome. The desktop client is still work in progress.

Sources and limits

This account covers work from 20 to 25 October 2026. It draws on my own use and my agents’ reports and test logs. The internal reports stay private. The technical details above come from those records; reproduced bugs aren’t presented as proven explanations for my call dropping, and none of my calls is quoted.

The description of Conduit combines practical phone use with a static read of those commits. It isn’t a measured restart or background-reliability comparison. Later versions may differ.

← things/

Cartoon goose generated with Nano Banana Pro using an earlier goose artwork as a reference. That artwork was AI-edited from a real goose photograph by Francesco Canu, CC BY-SA 3.0. The adapted image is under the same licence; the article text is separate.