bullshitai.dev

RESEARCH

Moving Hermes speech-to-text to codex-asr

My agents transcribe my voice notes. Moving recognition from a local model to an unofficial ChatGPT transcription shim cut six notes from 236 to 57 seconds, with fewer errors. Here's what that bought and what it costs.

Illustrative stock photo, not my room: Dejan Krsmanovic, CC BY 2.0, cropped

METHOD
Six private voice notes, the same audio and the same cleanup model and prompt on both sides. One run per backend.
WINDOW
11 Oct 2026, one evening, then a separate vocabulary run on the frozen transcripts.
RESULT
Wall time 236.0 → 57.3 s. Mean per-note word error rate after cleanup 13.08% → 5.11%.
wall time for the same six voice notes
236 → 57 s
mean word error rate per note, after the same cleanup
13.08 → 5.11%
critical words kept after cleanup, up from 35
41 / 50
local CPU for all six; recognition now runs on OpenAI's side
1.2 s

The find: a Whisper-shaped door into ChatGPT

I send my Hermes agents voice notes, in Ukrainian with English mixed in. My plugin, hermes-stt-enhance, turns those voice notes into text in two steps. First a local model hears the audio, NVIDIA’s Parakeet in my setup. Then a short LLM pass fixes punctuation, terms and the language mix.

In August I tried to fix the text after the fact and benchmarked cleanup models on frozen Parakeet output. The best one took mean word error rate from 17.06% to 11.97%. Better than nothing, but cleanup works on whatever the recognizer heard, and the recognizer was also the slow step.

Then I found Wangnov/codex-asr. It’s a small Rust server that listens on localhost and speaks the Whisper API shape. Behind that it uploads the audio to the transcription endpoint the ChatGPT apps use, using the ChatGPT login my Codex install keeps on disk. Its own README calls it “a reverse-engineered local automation surface, not an official API”. It accepts model=whisper-1 for compatibility and ignores it, so I don’t know which model actually hears me.

I added it to my plugin as a backend called codex_asr and kept Parakeet as an optional fallback.

Before and after on the same six notes

Six of my voice notes, 285 seconds of audio, the same files on both sides. The same plugin revision, the same cleanup model (gpt-6.1-sol, low effort, through my Codex subscription) and a byte-identical cleanup prompt. Only the recognizer changed.

Before and after on six voice notes: wall time 236.0 s to 57.3 s; mean word error rate after cleanup 13.08% to 5.11%.Same six voice notes, before and afterOne run per backend · same cleanup model and promptWALL TIME FOR ALL SIX, SECONDSBefore: local Parakeet236.0 sAfter: codex-asr57.3 sspeech recognitionLLM cleanupother4.1× faster end to end; recognition alone 13.3×How many words came out wrongMean word error rate per note against reviewed references. Lower is better; all bars share one scale.Raw transcript (outlined)After GPT cleanup (solid)Parakeet, local (before)Raw transcript17.06%After GPT cleanup13.08%codex-asr, cloud (after)Raw transcript13.29%After GPT cleanup5.11%0%5%10%15%20%After GPT cleanup: 13.08% → 5.11%, a 61% lower mean error rate per noteLocal CPU: 311.5 s → 1.2 s. Recognition moved to OpenAI’s servers; their compute isn’t measured.References started as ChatGPT transcriptions and may favour codex-asr.
Before and after on six voice notes: wall time 236.0 s to 57.3 s; mean word error rate after cleanup 13.08% to 5.11%.Same six voice notes,before and afterOne run per backend, same cleanup modeland promptWALL TIME, ALL SIXBefore: local Parakeet236.0 sAfter: codex-asr57.3 sspeech recognitionLLM cleanupother4.1× faster end to end;recognition alone 13.3×How many words came out wrongMean word error rate per note againstreviewed references. Lower is better;all bars share one scale.Raw transcript (outlined)After GPT cleanup (solid)Parakeet, local (before)Raw transcript17.06%After GPT cleanup13.08%codex-asr, cloud (after)Raw transcript13.29%After GPT cleanup5.11%0%5%10%15%20%After GPT cleanup: 13.08% → 5.11%,61% lower mean error rate per noteLocal CPU: 311.5 s → 1.2 s. Recognition moved toOpenAI’s servers; their computeisn’t measured. References started asChatGPT transcriptions.
One run per backend, not a production benchmark. Six of my voice notes, 285 seconds of Ukrainian with English mixed in, the same audio files on both sides. Error rate is the mean of per-note word error rate (WER) after alias normalisation, not pooled over all words. Cleanup is the same GPT pass on both sides. The references started as ChatGPT transcriptions, so they may favour codex-asr. The CPU line is my machine only. Aggregate data (JSON).

The chart’s values

Measure, all six notesBefore: local ParakeetAfter: codex-asr
Wall time, total236.0 s57.3 s
Speech recognition185.8 s13.9 s
LLM cleanup46.6 s42.3 s
Mean word error rate, raw transcript17.06%13.29%
Mean word error rate, after GPT cleanup13.08%5.11%
Critical words kept, after GPT cleanup35 / 5041 / 50
Local CPU time (after: plugin + shim)311.5 s1.2 s

A separate run reused the “after” raw transcripts and added a vocabulary file to the cleanup prompt (plugin revision 306772b): 4.42% mean WER, 44 / 50 critical words. It has no recognition step, so it says nothing about speed. Peak memory: 1,633 MiB before; after, 164 MiB for the plugin process and 42 MiB for the shim, measured separately, so their sum is an upper bound.

Speed. All six went from 236.0 to 57.3 seconds, 4.1 times faster. Recognition alone went from 185.8 to 13.9 seconds. Most of what’s left is the cleanup call, which now takes three quarters of the time.

Errors. The raw transcript was already better, 17.06% to 13.29%. The interesting part is what cleanup did with it: from the old raw it reached 13.08%, from the new raw 5.11%. That’s 61% fewer word errors on average, measured per note. Cleanup kept 41 of the 50 critical words in the references (names, terms, English phrases), up from 35.

One caveat changes how far to trust the error numbers. The reference transcripts started as ChatGPT audio transcriptions; one of my agents reviewed them and I corrected a term. codex-asr uploads to ChatGPT’s transcription endpoint, which may well be the same service, so the references can lean its way. The speed result doesn’t depend on that.

A later run gave the cleanup step a short vocabulary file and reached 4.42%. It reused the frozen transcripts at a newer plugin revision, so it’s a separate condition, not part of this comparison.

What it costs

My server’s CPU time for the six notes fell from 311.5 seconds to 1.2. The recognition didn’t disappear; it moved to OpenAI’s servers, and I can’t see what that costs them.

The audio leaves my machine now. The cleanup step already sent transcripts out, but the recording itself is a different thing to hand over.

The endpoint is undocumented. It can change, get rate-limited or stop working without notice, and codex-asr doesn’t refresh my login on its own. The plugin can fall back to local Parakeet, but only when I configure that explicitly.

I can’t give you a price. The usage meter on my subscription mixes everything I do, so transcription has no number of its own. “Free with a subscription you already pay for” is a guess.

My own voice notes go this way now. A recording with other people’s voices in it would be their call too.

The plugin

hermes-stt-enhance is public under MIT. The current public version, 0.5.1, does local recognition with faster-whisper or Parakeet, plus the cleanup pass, built so that a failed cleanup never loses the speech:

hermes plugins install dmmeteo/hermes-stt-enhance#hermes-stt-enhance

The codex_asr backend described here runs on my machine and isn’t in a public release yet. That install command doesn’t include it.

Method, sources and limits

One primary run per backend on 11 October 2026. It’s a before/after for my setup, not a production benchmark.

How it was measured

Samples. Six private voice notes, Ukrainian with English code-switching, 285.4 seconds in total. Audio hashes matched between the two runs. Audio, transcripts and the reference texts stay private.

Scoring. Word error rate against reference transcripts that started as ChatGPT audio transcriptions, reviewed by an agent and corrected by me in one place, after normalising accepted spelling variants, averaged per note. It isn’t pooled over all words. Critical words are 50 terms listed across the six references. The scorer is the one from my August cleanup benchmark.

Not controlled. No cold/warm protocol and no repeats of the full comparison. A later check re-ran three of the six notes on the new backend only. The vocabulary run reused the new raw transcripts at plugin revision 306772b, so it has no recognition timing. The August numbers come from a separate protocol and aren’t a leaderboard for this run.

Resources. CPU and memory are local process measurements. After the switch: the plugin process used 0.4 s of CPU and peaked at 164 MiB, the codex-asr shim 0.75 s and 42 MiB. Those peaks came at different moments, so their sum is an upper bound. Before: 311.5 s of CPU and a 1,633 MiB peak.

Sources and revisions

  • Wangnov/codex-asr at 479f6a7, version 0.1.2, MIT. The quoted README line and the ignored model field come from this revision.
  • dmmeteo/hermes-stt-enhance at 71fddac (public 0.5.1): local backends only. The tested codex_asr integration was plugin revision 229a844; the current local branch is 3add224. Neither is public.
  • Aggregate numbers: the JSON linked under the chart, regenerated from the raw run files.
  • Banner and card photo: Small Bedroom Recording Studio by Dejan Krsmanovic, CC BY 2.0. Cropped and resized, metadata removed, text added; no endorsement implied. An illustrative stock photo, not my room.