The Night Before the Demo, My AI Drive-Thru Started to Stutter

You know this one.

You pull up to the drive-thru speaker. It crackles. “Welcome, what can I get you?” You order a burger, no onions, a large fries and a Coke. Silence. Then: “Sorry, can you repeat that?” You repeat it, louder, leaning out of the window. Behind you a car honks. At the window they hand you a bag with onions in it and no fries.

Nobody in that story did anything wrong. The person on the headset is also bagging orders, watching a timer and taking a second lane. The speaker is cheap, the lane is noisy, and the job is to understand fifty strangers an hour through both.

Now the other version. You walk up to a kiosk and just talk to it, the way you would talk to a person. “The classic Smashburger. Actually, make it two. And a medium Coke.” It answers in a second. It knows the menu, knows what is sold out, never forgets the “no onions”, never gets tired at the end of a shift, and reads the order back before it takes your card. You tap, a receipt prints, you leave.

Every restaurant chain wants that second version. Some have already tried it in public and pulled it after videos of the AI adding hundreds of nuggets to one order went viral. It is harder than it looks.

I built one. A Flutter kiosk that greets you, takes the whole order by voice, reads it back, sends you to the card terminal and prints the receipt. No screen taps until the card. This is how it works, what nearly broke it the day before it ran live, and why the simple version won.

What is inside

Three parts, the same three you would use to describe a person at the counter:

• Ears. Deepgram, streaming speech to text over a WebSocket.
• Brain. A small OpenAI model (gpt-5.4-mini) with tools for the menu, the cart and checkout.
• Mouth. A native audio plugin that owns the microphone and the speaker.

The code below is rewritten and trimmed. The decisions are the ones that shipped.

The loop

mic (48 kHz, native)  ──►  Deepgram streaming STT  ──►  "is the guest done?"
                                                              │
speaker  ◄──  TTS  ◄──  reply text  ◄──  LLM + tools  ◄──  turn text
                                            │
                                     cart, menu, checkout
                                     (plain Dart, on device)

Everything runs on the tablet. An early version had a small Python service between the app and the model providers. I removed it. One less network hop per turn, and one less thing to deploy.

The one rule

The model talks. The code decides.

The model never edits the cart. It asks for an action through a tool call, and plain Dart code checks it, runs it and returns a result. Prices, stock and what is in the cart come only from those results, never from what the model remembers.

Everything below follows from that rule.

When is the guest done talking?

This is the first hard problem, and it is not the model. A drive-thru agent that answers half a sentence feels broken. One that waits three seconds after every sentence feels slow.

Deepgram gives you two signals. endpointing closes a segment after a short pause. utterance_end_ms fires when no new words have arrived for a while. We started with 300 ms endpointing, and it cut people off mid-order (“a classic burger and…”). We settled on this:

model=nova-3  encoding=linear16  sample_rate=16000
interim_results=true  endpointing=500  utterance_end_ms=1000
vad_events=true  smart_format=true

A turn ends on whichever comes first: a final segment with speech_final, or an utterance-end event. The second case has a trap. Sometimes the utterance-end arrives and no final segment ever did. If you only submit finals, the mic listens forever. So the adapter keeps the last interim and submits that:

void _onUtteranceEnd() {
  final text = _finalParts.isNotEmpty
      ? _finalParts.join(' ')
      : _lastInterim;           // no final arrived, use what we heard
  _emitTurn(text.trim());
}

View on GitHub Gist

Then the turn passes a few cheap gates before it costs a model call. Fewer than two characters, or only digits and punctuation, counts as a miss. Three misses in a row and the agent says it is having trouble hearing you, instead of guessing.

Teaching the STT the menu

Deepgram lets you send keyterm hints. The obvious move is to send the whole menu. That made things worse: with a long list of common words, ordinary speech gets pulled toward menu names. We send at most 12, and only distinctive ones: multi-word names or names of eight characters or more. “Smashburger” helps. “Coke” does not need help.

We also tried a second model pass to correct the transcript before the main model saw it. It added latency and over-corrected, so we turned it off. The main model is told about likely mishearings instead (“large price” probably means large fries), which is cheaper and good enough.

The brain: a small model with tools

Each turn sends one system message and the recent conversation. The system message has three parts:

1. Persona and rules. The agent is called Alex. Short replies, plain text only because it will be spoken, rotate acknowledgments (“Done.”, “Added.”, “On it.”) so it does not sound like a loop, recommend real items, not categories.
2. Knowledge. Store info and the menu and category names, built once per session from the local database and capped at 4,000 characters.
3. Current cart. Rebuilt from live state on every round and marked as authoritative. It is never stored in the history.

There is no RAG and no embeddings. A restaurant menu is small and structured. Putting the names in the prompt and letting the model look up details through a tool was faster and easier to debug than a vector search, and building the knowledge block once per session instead of per turn saved 100 to 500 ms on every turn.

The tools:

search_menu(query)                      list_menus()      list_menu_items(menu)
add_to_cart(menu_item_id, quantity)     remove_from_cart(...)
describe_menu_item_customization(id)
finalize_menu_item_customization(selections: [{group, choices, mode: normal|extra|none}])
get_cart_summary()
propose_checkout(guest_confirmed_checkout: bool)   // required, no default

search_menu is a plain lexical scorer, not a model. A name that contains the query scores high, token matches score lower, and it returns up to 12 rows with live price and availability.

Guards live in the executor

The model will sometimes do something silly. The fix is not a longer prompt. The fix is code that refuses and says why:

Future<ToolResult> addToCart(String id, int qty) async {
  final item = catalog.find(id);
  if (item == null || item.soldOut) return ToolResult.error('item_unavailable');

  final inCart = cart.quantityOf(id);
  if (inCart + qty > maxPerItem) {      // 50
    return ToolResult.error('quantity_exceeds_max',
        {'already_in_cart': inCart, 'max': maxPerItem});
  }
  cart.add(item, qty);
  return ToolResult.ok(cart.summary());
}

View on GitHub Gist

The error goes back to the model as a tool result, and the model explains it to the guest in its own words. The same pattern covers checkout: propose_checkout with guest_confirmed_checkout: false returns checkout_not_confirmed, and an empty cart returns empty_cart. The model cannot talk its way into payment.

Remember the nuggets? That is what the 50-unit cap is for. A cap in code makes that order impossible, whatever the model decides.

Bounding the turn loop

One guest sentence can take several model rounds: search, add, describe customization, finalize, reply. The loop has hard limits:

• At most 14 rounds per turn. After that the agent says “Sorry, let me start that over” instead of spinning.
• Identical tool calls are cached within a turn. If two rounds in a row are all cache hits, the model is stuck, and we break out.
• History keeps the last 20 messages. Pruning has a sharp edge: if you cut between an assistant message with tool_calls and its tool responses, the API rejects the request with a 400. The pruner drops orphaned tool messages from the front and dangling tool calls from the back.

The real thing, start to finish

Enough diagrams. This is the real system on a real kiosk: greeting, a burger, a Coke, “nothing else”, read-back, card, receipt.

The captions say full-duplex and barge-in. That was the goal. Watch the screen instead: it switches between listening, processing and speaking, and the mic is closed while the agent talks. Why that is, is the best part of this story.

The mouth, and the echo problem

The agent speaks through the speaker, and the mic is a few centimetres away. If the mic is open while the agent talks, Deepgram hears the agent, the agent hears itself, and it answers its own sentence.

There are two ways out. Close the mic while speaking (half-duplex), or cancel the echo and keep the mic open so the guest can interrupt (full-duplex). Full-duplex is the dream version. I built it, and then I turned it off.

What full-duplex needs

Android has a hardware echo canceller. It only works if it knows what the speaker is playing, so it can subtract it from the mic. That means recording and playback must share one audio session in communication mode:

val record = AudioRecord(MediaRecorder.AudioSource.VOICE_COMMUNICATION,
    48_000, CHANNEL_IN_MONO, ENCODING_PCM_16BIT, bufferSize)
AcousticEchoCanceler.create(record.audioSessionId)?.enabled = true
NoiseSuppressor.create(record.audioSessionId)?.enabled = true

val track = AudioTrack.Builder()
    .setAudioAttributes(AudioAttributes.Builder()
        .setUsage(AudioAttributes.USAGE_VOICE_COMMUNICATION).build())
    .setSessionId(record.audioSessionId)      // same session = echo reference
    .setTransferMode(AudioTrack.MODE_STREAM)
    .build()

View on GitHub Gist

Our first TTS played through a normal media player. The canceller never saw that audio, so it could not remove it. Moving TTS into the same native engine fixed the reference: raw 48 kHz PCM from the TTS provider, written straight to that AudioTrack. Mic audio never crosses into Dart. The plugin downsamples it 3:1 to 16 kHz and sends it straight to Deepgram’s WebSocket.

On top of that we had software guards: drop a transcript that repeats the last thing we finalized within 2.5 seconds (Deepgram’s rolling window can resend words), and let a 3-character interim from the guest stop the agent mid-sentence. Automatic gain control had to go, because it boosted everything to full volume and erased the loudness gap between a guest near the mic and the agent’s own voice.

The day before

It all worked. The canceller had its reference, the guards caught the echoes, you could cut the agent off mid-sentence and it would stop and listen. It felt like the dream version.

Then, the day before it had to run live, I put it on the actual kiosk tablet. The model was fine. It knew the menu, kept the cart straight, asked “anything else?” at the right moments. But every reply came out of the speaker broken into little scratches, like a record with dust on it. Weeks of work, and the one part a guest would notice first was the part that sounded wrong.

The cause was below the app. On that tablet the communication output path, the one the echo canceller needs, has a tiny playback ring buffer, the vendor’s audio effects on the output, and a track that drops into standby between chunks and wakes up late.

I tried a bigger native prebuffer (100 ms, then 750 ms). I tried buffering the whole clip before playing, which was clear but far too slow. I tried a Dart-side cushion and a larger AudioTrack buffer. None of it fixed the stutter without adding seconds of delay.

The finding, which I wish I had known on day one: on that device, hardware echo cancellation and clean voice output came as a package. You only get the canceller on the output path that stutters.

So that evening I had a choice. Keep the clever version and hope the tablet behaved on the day. Or rip it out and ship something simpler that would work every time.

I ripped it out. Half-duplex: the mic is paused before the agent speaks and resumed after the last sample drains. You can’t interrupt it, but it never hears itself and it never stutters. The next day it ran live, and I was more nervous than I would like to admit. It worked.

One fix from the full-duplex work stayed, and it is the one I would carry to any Android audio project: when the playback buffer is empty, write a 20 ms frame of silence. The track never goes to standby, never has to wake up, and the startup cushion dropped to about 250 ms.

private val silence = ByteArray(1920)   // 20 ms, 48 kHz, mono, 16-bit

while (running) {
  val chunk = queue.poll() ?: silence    // keep the track warm
  track.write(chunk, 0, chunk.size)
}

View on GitHub Gist

The proper full-duplex fix is a software echo canceller (WebRTC’s AEC3) that does not depend on the vendor’s output path. The hard part there is delay alignment between what you play and what you hear. That is the next version.

For the voice we started with ElevenLabs. The latest build uses Deepgram Aura over REST, streaming raw PCM at the engine’s sample rate, which plays straight into the native track with no decoding. It delivers 1.4 to 1.8 times realtime, with the first byte in under a second. Each reply is still one request. Splitting replies into sentences and streaming over WebSocket is the obvious next cut in latency.

Checkout: the agent stops at the door

When the guest confirms, propose_checkout returns a flag, not a payment. The app waits until the agent’s closing line has been spoken, then hands over to the existing kiosk checkout: card terminal, order submission, receipt printing. That flow was already tested in production. The agent does not get a new path to money. The prompt says it plainly: never claim payment is done until the device confirms it.

Measuring it

You cannot improve accuracy you do not measure, and nobody reads chat logs. Every session records its turns in memory and writes them once at the end, text only. Alongside the transcript we count signals that need no labelling:

• how often the agent asked again for the same thing
• items removed right after being added
• turns that hit the round limit
• phrases like “I said” or “I already told you”
• average turn latency, and how the session ended (checkout, abandoned, idle reset, error)

A sample of 10% of sessions also goes to an LLM judge after checkout. It reads the transcript and the final cart and returns whether the order matches what the guest asked for, with a score and a list of errors. That is how you find the bugs that no single turn shows.

What I took from it

• The model was the easy part. Most of the time went to turn-taking, audio and guards. A small, cheap model with good tools was enough.
• Put limits in code, not in the prompt. Caps, confirmations and round limits are a few lines each, and they cannot be argued with.
• The live cart goes in every request, never in the history. The model always sees the truth, and the truth never goes stale in context.
• Test audio on the real device early. The canceller worked on paper. The tablet decided.

The clever version is still in the git history, waiting for a software echo canceller. The simple one is the one that took the order.