Speech models and where they still fall over
Why “which model is best” is the wrong first question
Every speech to text model comparison I see online treats this like a leaderboard problem: pick the model with the lowest word error rate on some public test set and ship it. That’s not how these things fail in real pipelines. Word error rate on LibriSpeech or Common Voice tells you almost nothing about what happens when your audio comes from a phone call recorded on a cheap headset, in a room with an air conditioner running, with two people talking over each other about a product name your model has never seen.
The right question is: where does this specific architecture break, and does that break in a place that costs me money or embarrasses me in front of a customer. That’s a design question, not a scoreboard question, and it’s the one this article is actually trying to answer.
Whisper hallucinates, and it’s not a bug you can patch away
If you’ve run OpenAI’s Whisper on anything longer than a clean 30-second clip, you’ve seen it invent text. Feed it a stretch of silence, background hiss, or a long musical intro, and it will sometimes output a full sentence that was never spoken. This isn’t a training data gap you can fix by fine-tuning on more silence. It comes from how the model was trained and how it decodes.
Whisper is trained on paired audio and text with beam search decoding at inference time. When the acoustic signal is weak or ambiguous, the language model half of Whisper (it’s fundamentally a sequence to sequence transformer, so it always has a strong prior over what a “plausible” transcript looks like) takes over and generates the most likely next tokens based on prior context, not based on what’s actually in the audio. On clean, well spoken input this is a feature: it fills in mumbled words correctly. On silence or noise, it’s a liability: it fills in words that aren’t there at all, sometimes repeating a phrase from three sentences earlier, sometimes producing something that sounds vaguely plausible and is completely wrong.
If you’re transcribing customer support calls for a searchable archive, a hallucinated sentence sitting next to real transcript text is genuinely dangerous, because nothing in the output marks it as low confidence. Whisper doesn’t give you a reliable per-word confidence score out of the box the way some commercial streaming APIs do. You have to build your own silence detection and voice activity detection (VAD) layer in front of it, chunk the audio on detected speech boundaries, and drop segments the VAD didn’t flag as speech, before you ever hand audio to the model. That’s not optional engineering, it’s the cost of using Whisper at all.
Streaming changes the math entirely
Batch transcription (send a whole file, get a whole transcript back) and streaming transcription (partial results as someone is still talking) are different engineering problems wearing the same name. A model that scores well on offline word error rate can be mediocre in a live captioning or voice agent context, because streaming forces a tradeoff between latency and accuracy that batch models never have to make.
Streaming ASR systems, whether that’s a cloud API like Deepgram or AssemblyAI’s real-time endpoint or Google’s streaming recognizer, work by processing short audio windows and emitting partial hypotheses that can be revised as more context arrives. The shorter the window before you commit to an emitted word, the lower your latency, but the higher your chance of emitting something the model later has to walk back and correct. If you’re building a voice agent that needs to know when a caller has finished a sentence so it can respond, you’re tuning an endpointing threshold (how long a pause counts as “done talking”) and that threshold is a direct tradeoff between the agent feeling responsive and the agent cutting people off mid-sentence. Get it too aggressive and you interrupt anyone who pauses to think. Get it too conservative and every turn feels like the agent is stalling.
This is also where the “which model has the best word error rate” comparisons quietly stop being useful, because none of the public benchmarks measure endpointing behavior or partial-result stability under real network jitter. You find that out by running your own audio through it with a bad wifi connection and a person who says “um” a lot.
Accents, code switching, and the vocabulary problem
Every general purpose ASR model is trained on a distribution of speech that’s skewed toward whatever data was easiest to license and label at scale, which in practice means it’s skewed toward standard American or British English, read or scripted speech, and single speakers using one language at a time. Step outside that distribution and error rates climb, sometimes sharply.
Two failure modes show up constantly in production audio and rarely show up in demo videos. The first is code switching, where a speaker moves between two languages in the same sentence, which is completely normal in Singapore, India, the Philippines, and plenty of immigrant communities in the US and UK. Models trained primarily on monolingual corpora will often just transcribe the whole utterance in one language, silently dropping or garbling the other language’s words, because the decoder has no mechanism to switch language context mid-sequence unless it was explicitly trained to.
The second is domain vocabulary: product names, internal jargon, drug names, part numbers. A general model has never seen your company’s product name in training data, so it will transcribe it as the nearest phonetically similar dictionary word it does know. This is exactly the gap that custom vocabulary or keyword boosting features exist to close (most commercial ASR APIs let you supply a word list to bias decoding toward), but that feature only helps if you know in advance which words are going to show up, and it degrades if your boost list gets too long because you’re fighting the model’s own language model prior.
Diarization is still guessing
Speaker diarization, figuring out who said what, is usually bolted on as a separate step after transcription rather than being a native part of the ASR model. Most pipelines run a diarization model (something like a clustering algorithm over speaker embeddings extracted at intervals across the audio) in parallel with or after the transcript, then merge the two outputs by aligning timestamps. That merge step is where things go wrong.
Overlapping speech is the classic failure case. When two people talk at once, even briefly, the diarization system has to assign that audio segment to a single speaker, and it often gets it wrong or splits it awkwardly, which then shows up in your transcript as a sentence attributed to the wrong person or a line that jumps between two speaker labels mid-thought. Short interjections (“yeah,” “right,” “mhm”) are also a common miss, either getting merged into the previous speaker’s turn or dropped from the diarization entirely because they’re too short for the clustering window to confidently assign.
What I actually do
None of this means these models are bad. It means the failure modes are structural, not incidental, so you design around them instead of hoping a model update fixes them. In practice that means running VAD ahead of any batch transcription to strip silence before it can be hallucinated into text, treating diarization output as a starting point that needs a human or a heuristic pass for anything customer facing, keeping a custom vocabulary list current for domain terms and revisiting it whenever you ship a new product name, and picking streaming versus batch based on the actual latency requirement of the feature rather than defaulting to whichever API was easiest to integrate first. The honest version of a speech to text model comparison isn’t a table of word error rates. It’s knowing which of these five failure modes your audio is going to hit, and building the layer that catches it before it reaches a user.
If you want more of this kind of practical, no hype breakdown of what’s actually shipping in AI tools right now, come check out the rest of AI Tool Gazette.