We recently helped a founder implement a voice-to-voice system. The first version followed the path most engineers would probably start with:
Record the person speaking.
Upload the recording and wait for a transcript.
Send the transcript to a language model and wait for its complete answer.
Send that answer to a speech model, wait for an audio file, then play it.
It worked, but it was also quiet for around 5.5 seconds after the person stopped speaking. That felt way too long.
Today’s version starts playing audio in 444 milliseconds on the same recording. It uses Kyutai STT and Pocket TTS on an L4 GPU rented by the second from Modal. The language model is hosted Celeris-1.
Celeris-1 is a diffusion language model, which is the unusual part of the stack. Ordinary language models write an answer one small piece after another. Celeris starts with a rough answer and refines several pieces together. Short replies can appear almost at once, which is particularly useful when another model still has to turn them into speech.
The two stacks were:
First pass: GPT-Transcribe → GPT-5.6 Terra → GPT-4o Mini TTS.
Final: Kyutai STT → Celeris-1 → Pocket TTS, with the two speech models sharing one Modal L4.
The final system is hybrid: open-weight speech models on rented hardware, with a hosted diffusion model in the middle.
What we’re measuring
Our goal is to reduce the time to first audio. In our experiments, every run uses the same 1.393-second recording of “How far away is the moon?” as the input. We measure the time from the end of that recording to the first bytes of sound the system returns. In our original stack, that means we wait until the complete MP3 arrives. In the streaming path, we only wait until the first raw audio chunks arrive.
The 5.52-second and 444-millisecond numbers compare the first version to the last one. Between those versions the models and architectures changed. To measure what each change actually contributed, we swapped one thing at a time and everything else was fixed.
Where we started
The first implementation uses OpenAI directly for every model stage. GPT-Transcribe receives the complete WAV. Terra receives the complete transcript. GPT-4o Mini TTS, using the Marin voice, receives the complete answer and returns one MP3. Only then does the application decode the file.
Here is the median Terra turn:
Speech generation was the largest source of delay. The 40-word answer ended up being about 20 seconds of audio, and the application wouldn’t play any of it until the whole file was ready.
The first optimization: switch to Luna
For the very first optimization, we suggested switching from Terra to GPT-5.6 Luna. Everything else stayed the same.
Luna helped, but the user still had to wait over five seconds to get a reply. The language model was only one part of a sequence in which every step waited for the previous one to finish.
The second optimization: stream transcription
Our second angle of attack was transcription. Instead of waiting for the user to finish speaking before we began transcribing, we could stream it as they talked.
This led to our second model change: replacing GPT-Transcribe with Kyutai STT, self-hosted on a Modal GPU. We needed this swap for streaming to work at all: the live, partial-transcript flow this stack needs runs over a WebSocket, which Kyutai slots into nicely.
Blocking on the whole recording took 3,498ms. Streaming the input while it was recorded brought that to 3,181ms. Nearly 2 seconds and 37.5% faster than our previous best.
The third optimization: stream audio out
The same idea that helped on the way in helped again on the way out: instead of blocking audio output on a complete audio file, we stream the audio as it’s generated.
This required another model swap. We replaced GPT-4o Mini TTS with Pocket TTS, because Pocket TTS can emit raw, uncompressed audio in small chunks as it generates them rather than only returning a complete MP3 file once the entire input is processed.
Waiting for the whole audio file took 3,181ms. Streaming the audio out as raw chunks brought that to 1,778ms: 1,403ms faster, or 44%.
However long the upstream model takes to finish its reply, the pipeline no longer has to wait for a complete audio file to be synthesized from it afterward.
The fourth optimization: try a diffusion model
At this point, we’d done all the obvious things. What was still glaring at us was the Luna latency. We’d heard about recent promising developments around diffusion models for text generation. Our research led us to Celeris, who claim their Celeris-1 model is “the fastest LLM on Earth” outputting a shocking (if true!) 2,000 tokens per second.
So we swapped Luna for Celeris-1 on the streaming stack and measured it end to end. Streaming with Luna still in place took 1,778ms. Swapping in Celeris-1 brought that down to 441ms. 75% faster.
In our own tests, Celeris-1 produced replies at a rate of roughly 400 tokens per second, not quite the 2,000 we were promised, but still at least 9 times faster than Luna by the same measure. To be fair- we didn’t run anything approaching a real benchmark here; this is just output tokens divided by wall-clock time and includes overhead.
Open-weight moves the cost; it does not erase it
Kyutai STT and Pocket TTS do not carry a per-token model charge themselves, but the GPU running them certainly costs money. And the Celeris-1 diffusion model is a paid hosted API.
Using the published pricing for these providers (OpenAI, Modal’s L4, Celeris), representative turns work out as follows:
For the OpenAI rows, we used the measured clip duration and language-model token counts, then replayed the five saved replies through GPT-4o Mini TTS with usage reporting. The median speech charge alone was about $0.0061 with Terra’s replies and $0.0063 with Luna’s. That is why the much cheaper language model had only a modest effect on the complete turn.
For the hybrid row, we measured from the first microphone frame through the final spoken chunk. The warm turn occupied a median of 6.08 seconds of wall time. At $0.000222 per L4-second, that is $0.00135 of equivalent GPU time. Celeris adds roughly $0.00004 for a short prompt and reply.
The complete progression
The first implementation was not badly written. It was a reasonable composition of good APIs. Its latency came from the boundaries between them: complete recording, complete transcript, complete answer, and finally complete audio file.
The final implementation is much faster because it moves partial results forward. It transcribes during the question, starts speech from the first useful sentence, and plays raw audio while it the rest is still being generated. The stack also changed to fit that shape: Kyutai and Pocket TTS share a rented GPU, while Celeris supplies fast hosted text generation.
The results?
Milliseconds saved: 5,080.
Waiting time reduced: 92**%**.
Not having to ask “Are you still there?” Priceless.
You can try the replay at seconds.redspring.workers.dev, built from 13 questions recorded once against both stacks. The reproducible first-pass script, exact WAV, raw measurements, cost ledger, and diagram sources are in the repository: https://github.com/redspringxyz/seconds
Appendix: cold starts
The Modal service scales to zero between visits. A cold L4 wake takes about 28 seconds. Blending that into warm-turn medians would muddy the measurements, so the interface reports cold startup on its own.