What is backchanneling? The “mm-hmm” in voice AI
Backchannels are the “mm-hmm”s a listener drops in while someone else talks. People do it without thinking. A voice agent has to tell a caller’s backchannel from an interruption, and decide when to give one of its own.
In this guide
Backchanneling is how a listener shows they’re following without taking over: “mm-hmm,” “uh-huh,” “right,” “oh no,” “got it.” The speaker keeps the floor, and the listener signals in passing that the message is landing.
The linguist Victor Yngve named it. In a 1970 paper he described conversation as running on two channels at once. The person with the turn speaks on the main channel, and the listener sends short messages back on the “back channel,” which the speaker hears without stopping. Listen to any phone call and you’ll hear long stretches of one person talking, dotted with the other’s small sounds.
Kinds of backchannel
| Kind | Examples | What it says |
|---|---|---|
| Continuer | “mm-hmm,” “uh-huh,” “yeah” | Keep going, I’m with you |
| Acknowledgement | “I see,” “got it,” “okay” | I’ve taken that in |
| Sympathy | “oh no,” “oh dear” | That sounds bad |
| Assessment | “nice,” “that’s great,” “oh good” | That sounds good |
| Nonverbal | Nods, smiles, raised eyebrows | Any of the above, silently |
Timing matters as much as wording. Listeners seldom backchannel mid-phrase. They do it in small pauses, at the end of a clause, or when the speaker’s pitch drops. The kind has to fit, too: “that’s great” after someone describes a broken furnace is worse than silence.
Why they matter on the phone
Face to face, a nod does the work of “mm-hmm.” On the phone the voice carries all of it. A listener who stays silent while a caller explains a problem for thirty seconds sounds absent, and the caller fills the gap with “hello? are you still there?”
Contact center calls are full of these stretches. Callers read out addresses, describe what went wrong with an order, or explain what they’re shopping for, and they expect the small signals a person would give. Too few and the line feels dead. Too many, or mistimed, and the listener sounds impatient or mechanical.
The two backchannel problems for voice agents
Hearing the caller’s
When the agent is talking and the caller says “mm-hmm,” the right response is to carry on, because the caller is agreeing. To a simple voice-activity detector, though, “mm-hmm” looks like the start of an interruption. An agent that stops whenever it hears sound will stutter through every explanation, stopping, restarting and losing its place.
The fix is a turn-taking model that understands what the sound means, and decides quickly, since a real interruption deserves an immediate stop. That’s the barge-in problem, where the agent should stop for “wait, can I change that to Thursday?” and keep going through “uh-huh.”
Giving its own
The other direction is harder. While the caller talks, the agent has to keep deciding whether this pause wants a sound at all, since most don’t and a cue every few seconds sounds like someone hurrying you along. If it does, the agent has to pick the right kind, such as a continuer mid-thought, “oh no” after bad news, “I see” after an explanation. The cue has to land inside the pause, neither mid-word nor so late that the caller is already talking again.
Then there’s delivery. The cue should be in the agent’s own voice and quieter than its speech, the way a listener sounds. And the agent’s turn-taking must not mistake its own “mm-hmm” for the agent taking the floor, or it will cut off the caller it meant to encourage.
How Simple handles backchannels
On the listening side, our Tango model tells a real interruption from a quick “mm-hmm,” so the agent keeps talking when a customer is only agreeing. It recognizes a real interruption about 100 ms after the caller starts talking, and it cut interruptions by 64% against our previous agent in production. The Tango post and the turn-taking page explain how.
On the speaking side, Simple’s agent gives short listening cues of its own. When the caller pauses for about half a second mid-turn, a small model reads what they’ve said so far and scores whether a listener would respond and with which kind of cue (a continuer, an acknowledgement, sympathy or a short assessment). Mid-sentence, only a continuer or sympathy fits, so those are the only cues it uses there.
It rations them to a couple per turn, spaced several seconds apart. When the caller finishes and the agent takes its turn, it can say a brief “Okay.” straight away, so the caller hears a reaction while the full answer is generated; the answer then drops its own opening acknowledgement, so nobody hears “Okay. Okay, so…”
Each cue is rendered in the agent’s own voice, played more quietly than its replies, on a separate audio track that turn-taking ignores, so a cue never counts as the agent taking the floor. Cues are recorded on their own channel, so call reviews can hear them.
Backchannels and latency
A reply takes a moment to generate, and on the phone even a short gap sounds like hesitation. An “okay” at the right moment tells the caller they were heard while the answer is on its way, which is what a person does while they think. It doesn’t replace low latency or good end-of-turn detection, but it makes the remaining gap sound like part of the conversation.
How to test an agent’s backchanneling
Call the agent and read out a long address slowly, with pauses. It should wait for the whole thing, maybe with the odd “mm-hmm,” rather than jumping in at the first gap. Then tell it a delivery arrived broken, and listen for whether any cue fits the mood.
While it’s talking, say “mm-hmm” and check that it keeps going, then interrupt properly and check that it stops at once. Over a two-minute call, count the cues; more than a few a minute starts to sound mechanical. Last, play the recording back and listen for cues at full speech volume, clipped, or on top of your words.