1. Record the person speaking.

2. Upload the recording and wait for a transcript.

3. Send the transcript to a language model and wait for its complete answer.

4. Send that answer to a speech model, wait for an audio file, then play it.

It worked, but it was also quiet for around 5.5 seconds after the person stopped speaking. That felt way too long.

Today’s version starts playing audio in 444 milliseconds on the same recording. It uses Kyutai STT and Pocket TTS on an L4 GPU rented by the second from Modal. The language model is hosted Celeris-1.

Celeris-1 is a diffusion language model, which is the unusual part of the stack. Ordinary language models write an answer one small piece after another. Celeris starts with a rough answer and refines several pieces together. Short replies can appear almost at once, which is particularly useful when another model still has to turn them into speech.

The two stacks were:

• First pass: GPT-Transcribe → GPT-5.6 Terra → GPT-4o Mini TTS.

• Final: Kyutai STT → Celeris-1 → Pocket TTS, with the two speech models sharing one Modal L4.

The final system is hybrid: open-weight speech models on rented hardware, with a hosted diffusion model in the middle.

What we’re measuring

Our goal is to reduce the time to first audio. In our experiments, every run uses the same 1.393-second recording of “How far away is the moon?” as the input. We measure the time from the end of that recording to the first bytes of sound the system returns. In our original stack, that means we wait until the complete MP3 arrives. In the streaming path, we only wait until the first raw audio chunks arrive.

The 5.52-second and 444-millisecond numbers compare the first version to the last one. Between those versions the models and architectures changed. To measure what each change actually contributed, we swapped one thing at a time and everything else was fixed.

Where we started

The first implementation uses OpenAI directly for every model stage. GPT-Transcribe receives the complete WAV. Terra receives the complete transcript. GPT-4o Mini TTS, using the Marin voice, receives the complete answer and returns one MP3. Only then does the application decode the file.

Here is the median Terra turn:

Speech generation was the largest source of delay. The 40-word answer ended up being about 20 seconds of audio, and the application wouldn’t play any of it until the whole file was ready.

The first optimization: switch to Luna

For the very first optimization, we suggested switching from Terra to GPT-5.6 Luna. Everything else stayed the same.

Luna helped, but the user still had to wait over five seconds to get a reply. The language model was only one part of a sequence in which every step waited for the previous one to finish.

The second optimization: stream transcription

Our second angle of attack was transcription. Instead of waiting for the user to finish speaking before we began transcribing, we could stream it as they talked.

This led to our second model change: replacing GPT-Transcribe with Kyutai STT, self-hosted on a Modal GPU. We needed this swap for streaming to work at all: the live, partial-transcript flow this stack needs runs over a WebSocket, which Kyutai slots into nicely.

Blocking on the whole recording took 3,498ms. Streaming the input while it was recorded brought that to 3,181ms. Nearly 2 seconds and 37.5% faster than our previous best.

The third optimization: stream audio out

The same idea that helped on the way in helped again on the way out: instead of blocking audio output on a complete audio file, we stream the audio as it’s generated.

This required another model swap. We replaced GPT-4o Mini TTS with Pocket TTS, because Pocket TTS can emit raw, uncompressed audio in small chunks as it generates them rather than only returning a complete MP3 file once the entire input is processed.

Waiting for the whole audio file took 3,181ms. Streaming the audio out as raw chunks brought that to 1,778ms: 1,403ms faster, or 44%.

However long the upstream model takes to finish its reply, the pipeline no longer has to wait for a complete audio file to be synthesized from it afterward.

The fourth optimization: try a diffusion model

At this point, we’d done all the obvious things. What was still glaring at us was the Luna latency. We’d heard about recent promising developments around diffusion models for text generation. Our research led us to Celeris, who claim their Celeris-1 model is “the fastest LLM on Earth” outputting a shocking (if true!) 2,000 tokens per second.

So we swapped Luna for Celeris-1 on the streaming stack and measured it end to end. Streaming with Luna still in place took 1,778ms. Swapping in Celeris-1 brought that down to 441ms. 75% faster.

In our own tests, Celeris-1 produced replies at a rate of roughly 400 tokens per second, not quite the 2,000 we were promised, but still at least 9 times faster than Luna by the same measure. To be fair- we didn’t run anything approaching a real benchmark here; this is just output tokens divided by wall-clock time and includes overhead.

Open-weight moves the cost; it does not erase it

Kyutai STT and Pocket TTS do not carry a per-token model charge themselves, but the GPU running them certainly costs money. And the Celeris-1 diffusion model is a paid hosted API.

Using the published pricing for these providers (OpenAI, Modal’s L4, Celeris), representative turns work out as follows:

For the OpenAI rows, we used the measured clip duration and language-model token counts, then replayed the five saved replies through GPT-4o Mini TTS with usage reporting. The median speech charge alone was about $0.0061 with Terra’s replies and $0.0063 with Luna’s. That is why the much cheaper language model had only a modest effect on the complete turn.

For the hybrid row, we measured from the first microphone frame through the final spoken chunk. The warm turn occupied a median of 6.08 seconds of wall time. At $0.000222 per L4-second, that is $0.00135 of equivalent GPU time. Celeris adds roughly $0.00004 for a short prompt and reply.

The complete progression

The first implementation was not badly written. It was a reasonable composition of good APIs. Its latency came from the boundaries between them: complete recording, complete transcript, complete answer, and finally complete audio file.

The final implementation is much faster because it moves partial results forward. It transcribes during the question, starts speech from the first useful sentence, and plays raw audio while it the rest is still being generated. The stack also changed to fit that shape: Kyutai and Pocket TTS share a rented GPU, while Celeris supplies fast hosted text generation.

The results?

Milliseconds saved: 5,080.

Waiting time reduced: 92%.

Not having to ask “Are you still there?” Priceless.

You can try the replay at seconds.redspring.workers.dev, built from 13 questions recorded once against both stacks. The reproducible first-pass script, exact WAV, raw measurements, cost ledger, and diagram sources are in the repository: https://github.com/redspringxyz/seconds

Appendix: cold starts

The Modal service scales to zero between visits. A cold L4 wake takes about 28 seconds. Blending that into warm-turn medians would muddy the measurements, so the interface reports cold startup on its own.

There are already some tools for this, like Freedom and Cold Turkey, and Brick for your phone. But all the desktop ones had unnecessary features like cross-device sync, scheduling, browser extensions, and dashboards. We only wanted one thing: block a list of websites for a set amount of time, with no way to undo it until the time runs out. This was simple enough that we thought we should just build it ourselves.

The app is called Retreat. It’s free, open source, and works on macOS 15 or later. You can download it at retreat.computer.

Diving into loop engineering

Building Retreat was also an experiment. We’ve been thinking about loop engineering, where you give an agent a high-level goal, let it generate its own subtasks, execute them, and iterate until the goal is achieved. Ideally, the process is largely unattended and you come back to finished software.

We’ve seen loop engineering work in a few specific contexts. There’s a much-discussed case of the JavaScript runtime Bun migrating an enormous codebase from Zig to Rust largely through autonomous agent loops. That’s the kind of task loop engineering is probably suited for: a clear input, a clear output, a test suite that can verify correctness without human eyes.

We wanted to see how it would hold up for building a new native app from scratch. The short answer is that it didn’t, really.

Agentic tech bureaucracy

We let the agents run for around twenty-four hours, using frontier models and a setup where they’d generate tasks for themselves, execute them, and loop. The agents wrote code and features, and something resembling Retreat took shape.

But when we came back to it, the agents had created a system of elaborate and incomprehensible internal tickets, an agentic tech bureaucracy. Reading through what the agents had decided mattered and why, was like reading meeting notes from a company that didn’t exist (for good reason).

Also, a lot of the code didn’t work.

The core problem is that loop engineering needs verifiable success criteria that the agent can check without a human in the loop. For a library or framework with unit tests, that works pretty well. For a native macOS app that blocks network requests, the agent doesn’t have a good enough way to verify that the blocking actually works. Computer use has gotten better, but isn’t fully solved yet.

On top of that, building any product involves hundreds of small decisions. Every time the agent made one differently than we would have, the gap between what the agents were creating and the result we wanted would grow. After enough of these branching decisions, the product goes in an entirely different direction.

Eventually we ended up doing what we always do: sitting down with the agents, being hands-on, guiding the code, testing manually, and fixing things. And now, the app is what we want it to be, but it definitely wasn’t loop engineering that got it there.

The future of loop engineering?

Loop engineering is probably the right choice, in the right conditions. If you have clear success criteria, verifiable outputs, and well-defined scope like migrating a codebase, running a batch job, or transforming data in a predictable way, loop engineering could save you a lot of work.

But new product development isn’t the right fit yet. Products involve taste and nuanced decisions, and often, involve you changing your mind about the result. The agents can handle a lot, but they can’t handle all that.

The tooling also isn’t there yet for native app development specifically. Agents can more or less run the app with computer use, but it’s still flawed and burns tons of tokens, which leaves a huge gap in development. You still need to do the most important part yourself.

We’ll keep experimenting, but for now, Retreat was built the old new way: we had an idea, some agents helped write the code, and we did a lot of manual iteration.

Download it at retreat.computer. Source is on GitHub. Time to break out of bad habits!

If you’re planning an AI product, there’s a piece of work you can do right now, before you hire an engineer or open a code editor, that will save you months of wandering: write your evals.

An eval is just a repeatable test for an AI system. You give it an input, it produces an output, and something — a rule, a rubric, or another model — judges whether the output is good. Simple, and doable for non-engineers. There’s a growing consensus among AI product leaders that evals are the key spec for AI products, rather than product requirement documents (PRDs). Traditional PRDs lean on words like “helpful,” “accurate,” and “concise,” which can’t really be tested. AI systems are probabilistic, meaning the same input can produce different outputs, and models change under your feet. An eval collapses the spec, the acceptance criteria, and the launch review into one runnable artifact that answers a single question: what does good look like, and how do I measure it?

One key benefit is that you don’t need an actual product to write evals. All you need is judgment about your domain. And if you’re the founder, you have more of that than anyone you’ll ever hire.

Step 1: Write down 20 real inputs

Every eval has three ingredients: a dataset (example inputs), a task (the AI doing the work), and scorers (how you judge the output). You can build the dataset and the scorers before you even have a product.

Before anything else, build the dataset. Write out twenty to thirty examples of what users will actually ask your product to do. Make them as natural as possible, based on language your target customers would use in real life. For instance, if you’re building a tool that summarizes legal contracts, go find twenty contracts. If the product is a customer support agent, write out twenty support tickets (make sure to include some “unhappy customer” ones and some “I don’t know what I actually need” ones).

Step 2: Define what good looks like in binary

Next, for each input you created, write what a great output would contain. Then turn your quality bar into yes or no questions. A few great examples: Did it cite the actual clause? Did it stay under 150 words? Did it avoid inventing numbers? Did it escalate when the customer mentioned a refund?

We’re often tempted to score things on a 1–10 scale, but resist that — it won’t be helpful in the long run. Pass or fail is hard enough, and it forces you to be brutally honest about your requirements. Most importantly, make the criteria fully yours. Avoid generic metrics and make your scorers specific to your product. High quality looks different for every product.

Step 3: Grade by hand first

Next, run your inputs through an AI model (Claude, ChatGPT, or whatever you use most often) using a rough prompt that describes what your product should do. Grade each output yourself: thumbs up or down, and include one sentence explaining why it was good or bad. That written justification is incredibly valuable: it becomes your failure taxonomy (”made up a policy,” “buried the answer,” or “wrong tone for a complaint”) and each failure mode becomes a scorer later on.

This stage is sometimes called vibe-checking. Don’t skip it. Never automate a judgment you haven’t first made yourself, repeatedly, on real examples. A spreadsheet with your labels is a completely legitimate eval system.

Step 4: Make it repeatable with Braintrust

Once your hand-graded spreadsheet exists, a tool like Braintrust turns it into something you can run over and over without writing code. You can paste your inputs and expected outputs into its dataset editor, write your prompt in the browser-based playground, and run it against the whole dataset at once, comparing prompts and models side by side. Its AI assistant, Loop, will even draft scorers from a plain-English description (”check that the summary never invents a dollar figure”). You describe your bar for quality in a sentence or two, then Loop will write the grading logic, and then you can verify if the logic agrees with your hand labels before using it. There’s a free tier, and the whole workflow is usable by a non-engineer. (Langfuse is a strong open-source alternative.)

Now, without hiring an engineer, you can change the prompt, rerun the eval, and know whether it got better.

Evals build a strong foundation

By the time you engage engineers, whether that’s a technical co-founder or a team like ours, you’ll be able to hand them a runnable definition of success instead of a prose wishlist. They can test three models against your dataset in an afternoon and tell you which one clears your bar (and what it costs). Every future change will get measured against the same benchmark, so improvements build on each other.

There are two warnings before you dive in. First, don’t get set on a perfect pass rate. If everything passes, your eval is too easy, and you’re optimizing for your own comfort rather than quality. Second, don’t let perfectionism stall you: an imperfect set of twenty examples you can come up with in a week beats “perfect” dataset that takes months to be designed by a committee. Once your product launches, real user data will reshape the dataset anyway. Production failures quickly become new test cases, and the eval will grow with the product.

There’s no secret access to better AI models. Everyone has the same ones. The secret to success is defining “good” outputs precisely and early on — and using those benchmarks going forward. You can start that today. All you need is a spreadsheet.

Redspring is a boutique AI and software consultancy in New York. Your founding engineering team, on demand. If you’ve got a dataset and a dream, we’d love to talk.

For most of human history, there was a stark line between audience and story. The audience heard the story, read the book, and eventually watched the movie. The story happened while you observed.

Interactive storytelling changed that. There were choose-your-own-adventure books (remember those?), then RPG and narrative games. Now, AI is changing things again.

This week, The Atlantic released an interactive murder mystery co-developed by Redspring, written by Lemony Snicket and illustrated by Michael Kupperman. Players step into the role of detective in a closed park, talk to a cast of eccentric characters, and try to figure out who committed the murder. The characters are powered by AI, but the story architecture is still human.

Dubious Park, illustrated by Michael Kupperman

We sat down with Caleb Madison, former director of games at The Atlantic and the person who originally conceived of the project, to talk about what we learned over the last year building a human-authored game with characters you can interrogate in real time.

The problem of authority

Caleb has been making games since he was fourteen, mostly crosswords. To him, the appeal is simple: games create a bounded arena with a final, true answer (something we don’t often find in the real world). A game designer leads a player towards a certain outcome, ideally in a dynamic and entertaining way.

We were all captivated by the idea of bringing AI into that world. Can we use this new medium as a way to tell stories that you can engage with, that feel surprising, and that pique your curiosity because they evolve?

That idea runs directly into a fundamental challenge of AI today: uncertainty.

In a traditional story, the author is the source of truth. If the novel says it was a dark and stormy night, the reader accepts it. But in an AI-mediated experience, where you discover the story through conversation, who is speaking with authority? The author? The system? The character? The model?

This became our central design problem. The original concept had psychologically layered suspects who could be caught in lies. It was an Agatha Christie-style cross-examination where players would triangulate the truth by finding contradictions. Caleb described why that didn’t work: “If this one thing is a lie, then how many other things are lies? It becomes a hall of mirrors.”

Lemony Snicket, it turned out, understood this intuitively. His creative instincts pushed the game away from forensic interrogation and toward something more expansive. We were soon building an oddball ensemble in a fully realized world, where the characters had distinct personas and the players were fully inhabiting the space.

Lemony helped us realize that the solution was to give the characters a strong enough identity that players trust them even when they surprise you.

Building the game

Early in the project, we developed a character builder that let an author train a character through dialogue, refining its persona until it feels consistent. It was a good idea, but it wasn’t quite right for this project.

Working with a writer like Lemony Snicket, who already had a rich and specific creative universe, the character builder felt stifling. Instead, the best approach was to give him room to define the characters in longhand, in his own voice, with his own instincts. Our job became translating those authored characters into personas that could hold up inside an interactive system.

That required a lot of testing. We put the game in front of as many people as we could, watched their conversations, and learned which paths worked and which led nowhere. Coordinating characters, world knowledge, player knowledge, and the investigatory path toward the solution were all refined through that process until the experience felt natural.

Players never see most of what makes the game work, and that’s part of what makes it a success. The fact that it feels simple is a big achievement. Underneath that simplicity is a system that we iterated on for months.

AI is just one wire in the sculpture

The question Caleb gets asked most about this project is some version of: isn’t AI just a way to cut corners? His answer, in this case, is definitely not.

“I see creative work and its relationship to tools or the medium of creativity as pretty consistent, whether it’s oil paint, film, or the computer. The amount of work that the creative person, or the people on the back end, put into the thing is directly proportional to how good it is. There’s no hack or way of getting around that.

We didn’t use AI as a shortcut for creativity here. Our desire, our curiosity, was to figure out how to integrate AI and LLMs thoughtfully into a larger storytelling experience.

This game would not be fun without Lemony’s story, which he put so much work and effort into; without Michael Kupperman’s beautiful illustrations; and without the hours and hours of collaborative work that all of us have done to fit this together in a way that feels satisfying.

My sense is that a game, in many ways, is like a big, complex digital sculpture. AI is like a little thing hanging on one of the wires in the sculpture. It’s not the base of the sculpture at all. It’s ornamental in a way that I hope lends itself to satisfaction and play.

This is an attempt to use AI as a medium in and of itself: as a material integrated into other creative materials, the visual arts and the storytelling arts, to be one part of the mosaic of the expression of a story and an experience.”

AI ≠ creativity

Towards the end of our conversation, Caleb voiced the most useful thing we took away from this project: making AI art means chiseling away at a vast space of possibility.

AI can generate almost anything. Creativity comes in when deciding what AI won’t do in a specific context. For the game, it was what persona a character holds to, what knowledge it has access to, and what the boundaries of the world are. We found that the more precisely we defined those constraints, the more the experience felt authored rather than generated.

The principle of constraining AI applies beyond games. Any time you build something with AI at its center, the quality of the output is determined by the choices you make about what the model is and isn’t allowed to do. Your choices as the system architect are the creative center of the output.

A peek into a bigger room

In the gaming world, we’re still in the early stages of AI integration. Caleb compared the current moment to the Lumière brothers projecting footage of a train arriving at a station: audiences ran because they didn’t yet understand the boundaries of the medium. (Probably an urban legend, but the comparison stands). Today, audiences and users are still figuring out where AI ends and where the human begins. That confusion is going to take time to resolve, and that’s why we’re continuing to experiment.

Building this game left us with a set of better questions. What does authorship mean when a reader can interrogate the characters? What does truth mean in a story where the player discovers it through conversation? What does constraint mean when the system could, in theory, say almost anything?

We don’t have tidy answers to any of those. But we know how to ask the questions more precisely than we did before. That feels like the right place to be when a medium is this young. We’ll be watching this space closely. As Caleb said, “I’m very fascinated by interactive storytelling, puzzles, games, storytelling proper, the classic storytelling of the 20th century and previous eras, and the intersection of all of them in the beautiful intermedia future that we are all barreling toward at breakneck speeds.”

Lemony Snicket’s Suspicious Incident in Dubious Park is live. Go solve a murder!

We picked a workload of 100,000 pages. That number was mostly arbitrary, but it was large enough to be interesting, and large enough that we had to figure out how to make it not a financially ruinous endeavor.

tl;dr - three things that surprised us:

1. Self-hosting open-source OCR is cheaper and far less painful than you might think. Our full 100k-page run with dots.ocr-1.5 finished in 56.5 minutes for $223 — about $2.27 per 1,000 pages. Staging weights, renting GPUs, and standing up the serving stack took hours, not days; Modal did most of the heavy lifting, and we never touched a CUDA driver or a container registry.

2. The cheapest GPU per second may be the slowest to finish and unacceptable at scale. An L4 can land in the same dollars-per-page band as an H100 while taking 5–6× longer to drain the queue. Cost-per-page doesn’t matter if you blow your SLA.

3. Benchmark your workload, not the leaderboard. Two public-ranking favorites lost once we measured them on the shape of our data.

This post is for engineers, technical founders, and anyone who has mostly lived on hosted APIs from the big model providers and wants a concrete sense of what self-hosting actually looks like in practice.

Why open-source models?

Control, cost, and quality that is good enough for the vast majority of document workflows. We focused on VLMs (rather than text-only OCR) because real documents are messy: handwritten forms, scanned PDFs, awkward multi-column layouts, and text baked into images all break naive text extraction.

Running your own model turns a fixed price sheet into a set of knobs. The weights, serving engine, batch strategy, GPU type, and quantization are all yours to tune. You stop paying a per-token markup and start paying for GPU-seconds, which flips the economics once volume is non-trivial.

There’s a strategic angle too, and it’s the one that’s easy to underweight until it bites you. If a hosted API is the foundation of your product, you don’t own the cost curve, the model lifecycle, or the deployment path. Prices move, default models change underneath you, and the exact model you tuned around can be deprecated on the vendor’s schedule, not yours. Self-hosting trades convenience for control over the variables that matter in production.

The three axes that matter

Everything below comes back to three measurements:

• Throughput — pages per second per GPU. Sets wall-clock time and how many GPUs you need.

• Cost — dollars per page. Sets whether the project is viable at scale.

• Quality — does the model actually read the page right? A page OCR’d cheaply is worthless if it silently garbles the text. We treat this qualitatively (more on why in Model Selection) and ship a viewer so you can check our work.

We’ll take them roughly in the order they mattered to us: which model, then how fast and how cheap, then how close to correct, and finally how all of that stacks up against the proprietary APIs.

Who we are

We’re the founders of Redspring, an AI and product-development consultancy. We build applied ML systems, including large-scale document and data processing, so this was equal parts client-relevant and an excuse to satisfy our own curiosity.

Overview

1. Model and engine selection

2. Modal basics

3. Benchmark methodology

4. Results

5. What we learned

6. Conclusion

Model Selection

We picked models based largely on how they performed on OCR-oriented public benchmarks, including olmOCR-bench, OCRBench v2, and the OCR-related leaderboards collected by LLM-Stats (OmniDocBench 1.5, OCRBench). Because these benchmarks cover different tasks and not every model appears on every leaderboard, we treated them as directional rather than definitive.

Here’s the rough ranking we used to decide what to test:

This is not meant to be a universal ranking. It ignores cost, latency, serving complexity, and the specifics of the workload. It was just a sensible starting point.

Why we didn’t compute a single accuracy number

The obvious move here is to report CER/WER against ground truth and crown a winner. We deliberately didn’t, for two reasons. First, that work already exists and is done well — the public benchmarks above are the right place for a rigorous, apples-to-apples score, and we’d only be reproducing them worse. Second, a single aggregate number hides exactly the failures that matter in production. A model can post an excellent average CER while occasionally and confidently rewriting a clause, and a quiet hallucination is far more dangerous downstream than uniformly fuzzy text.

So we used the public benchmarks to pick candidates, then judged the shortlist the way you’d actually judge it before shipping: by reading the output. We built a side-by-side viewer (last section of this post) and spent real time in it. The verdict and per-model recommendations live there, next to the evidence. If quality is mission-critical for your workload, do the same — pull a representative sample, run it through your top 2–3 candidates, and have someone who knows the domain eyeball the results.

Modal basics

Modal is a good fit for this work because it makes GPU-backed serving much less painful than doing the same thing on AWS/GCP/Azure from scratch. On a hyperscaler, serving a single GPU-backed model typically means: picking a GPU instance family with the right capacity and availability, wrestling CUDA drivers into the base image, wiring up autoscaling and cold-start handling, standing up a container registry, piping in secrets via IAM or a parameter store, and mounting some kind of network volume so you aren’t re-downloading 50+ GB of weights on every boot. Each of those is a half-day task in its own right, and most of them are about cloud plumbing rather than the model you actually want to serve.

Modal collapses most of that into a single Python file. The basic pattern is simple:

• stage model weights into a persistent Volume

• mount that Volume into a GPU-backed container

• start the serving engine during container startup

• expose the service over a normal HTTP interface

For batch-style setup work, like downloading Hugging Face weights into durable storage, we used one-off functions. For serving, we used a GPU-backed class that launched the engine as a subprocess during startup and exposed a web server once the container was ready.

Here’s the rough shape:

volume = modal.Volume.from_name("dots-ocr-1.5-assets", create_if_missing=True)

@app.function(
    image=image,
    secrets=[modal.Secret.from_name("hf-secret")],
    volumes={"/models": volume},
)
def upload_model_to_volume() -> None:
    snapshot_download(..., local_dir="/models/dots.ocr-1.5")
    volume.commit()

And for a long-lived service:

@app.cls(
    image=vllm_image,
    gpu="L4",
    volumes={"/models": model_volume, "/root/.cache/vllm": vllm_cache_volume},
    scaledown_window=300,
    min_containers=0,
)
class DotsOcrVLLM:
    @modal.enter()
    def startup(self) -> None:
        self.process = subprocess.Popen(["vllm", "serve", ...])

    @modal.web_server(8000, startup_timeout=900, label="dots-ocr-1-5-vllm")
    def serve(self) -> None:
        pass

A few details mattered. Each of these corresponds to a platform primitive that, if it didn’t exist, you’d have to build yourself on a generic cloud:

Volumes. A Modal Volume is a persistent, mountable filesystem shared across functions and containers. Model weights for a VLM run 20–200+ GB; downloading them on every container start would blow the cold-start budget and rack up egress from model stores like Hugging Face. Staging weights into a Volume once and mounting it read-only at runtime means each new container sees the weights as local disk in a fraction of a second.

Secrets. Modal Secrets are named, server-side key-value bundles you reference by name and inject as environment variables at runtime. We used them for things like HF_TOKEN so credentials never landed in the image, the repo, or local shell history. Rotating a secret is a single CLI call; no rebuild required.

Scaling controls. min_containers, max_containers, and scaledown_window let you trade idle cost against cold starts — a warm container eats GPU-time until it scales down, a cold one pays the model-load penalty on the next request. For benchmarking we set max_containers=1 so the system would stay on a single container rather than auto-scaling to absorb our synthetic load, which would have made throughput measurements unreadable. We also usually killed containers right after a run to avoid paying for idle time. In production, the same two knobs let you pre-warm for SLA-sensitive traffic or go fully scale-to-zero for batch windows.

That’s enough infrastructure to make the experiments reproducible without turning this post into a deployment manual.

Benchmark Methodology

The basic idea was straightforward: load a corpus of PDFs, send one OCR request per page until the sample run is complete, and measure throughput under different concurrency levels. Combined with measured pages-per-second-per-GPU and Modal’s on-demand pricing, that gave us a reasonable way to estimate cost and fleet size for a 100,000-page run before we accidentally emptied our bank accounts.

Concretely, we:

• ran the same models across multiple GPU types

• swept through concurrency levels to determine max GPU throughput

• evaluated the results quantitatively (speed, cost) and qualitatively (output fidelity)

One important setup detail: we pre-rasterized the PDF pages into images as a cheap one-off job before the main OCR benchmarks. That kept the benchmark focused on the part we actually cared about — model serving throughput and GPU cost — rather than letting PDF rendering dominate or add noise to the measurements. In a real production pipeline, you would still need to account for rendering, storage, and orchestration, but those costs are much smaller and easier to optimize than GPU-backed model inference.

We did not run every model/configuration as a literal 100,000-page end-to-end job. Instead, we used representative benchmark samples and combined the measured throughput with public pricing to estimate full-run cost. For planning and comparison, that was good enough. As we got closer to the final run, we moved from single-container spot checks to multi-container H100 runs of 5,000–20,000 pages so the extrapolation would reflect production fleet behavior rather than individual GPU performance. After those multi-container H100 runs gave us a configuration we trusted, we ran the full 100,000-page job with that setup.

Results

The rest of this section walks through these three axes in order: quality (is the output any good?), throughput and cost (can we afford it at scale?), and proprietary-API comparison (is it worth self-hosting at all?).

Assessing quality: the side-by-side viewer

Before any of the cost numbers mean anything, you have to answer the obvious question: does this model actually read the page right? So we built a small static viewer that lets you flip through individual pages and compare every model’s OCR output against the rendered original, side-by-side. Best on desktop.

Open the OCR Results Viewer in a new tab →

Our verdict after spending real time in the viewer:

• Reach for Chandra when fidelity is paramount. It’s our top pick on accuracy, and the lead is most visible on tables and complex formatting where the others start dropping cells or mangling structure. Good fit for compliance-sensitive documents, archive digitization, or anything downstream that assumes the text is trustworthy.

• Reach for dots.ocr-1.5 when throughput or cost is the binding constraint. It’s a very close second on quality and a runaway winner on cost-per-page — the right call when the downstream consumer is another model (RAG indexing, classification, summarization, agentic workflows) that can absorb a little OCR noise.

Both are genuinely strong; the pick is about which axis you’re optimizing. And don’t sleep on the Qwen variants in the viewer — they weren’t purpose-built for OCR and still hold up better than we expected. The viewer exists precisely so you don’t have to take our verdict, the public leaderboards, or the cost tables below on faith.

Self-hosted cost to process 100k pages

The headline run used a fleet of 60 H100s handling 200 concurrent requests. It submitted 100,056 pages, completed 98,159, and sustained 28.96 submitted pages/s — finishing in 56.5 minutes at $223.10 of measured GPU cost. Normalized over successful pages, that’s $2.27 per 1,000 pages, at a failure rate just under 2%.

We didn’t jump straight to 60 GPUs. Before committing to the full job, we ran a few smaller multi-container H100 sweeps with dots.ocr-1.5 to establish a throughput-and-cost baseline we trusted enough to extrapolate from:

The next table is a compact view of model x GPU sweeps. It does not list every concurrency setting we tested; instead, it normalizes each run down to the best per-GPU throughput and the estimated 100K-page cost that mattered for choosing the final profile.

These estimates used Modal’s pricing at the time of publishing and the formula:

Estimated run cost formula:

run cost ≈ (pages / throughput) × GPU_count × GPU_price_per_sec

Where "throughput" is measured in pages/sec/GPU.

Pricing assumptions used here:

• L4 = $0.000222/s

• L40S = $0.000542/s

• A100 (40GB) = $0.000583/s

• A100-80GB = $0.000694/s

• H100 = $0.001097/s

• H200 = $0.001261/s

• B200 = $0.001736/s

Proprietary API price-sheet comparison

We include proprietary API pricing as context, not as an argument for the APIs. The thing worth scrutinizing isn’t the per-page number; it’s the dependency you take on when one vendor owns both the model and the price sheet.

The shape of the comparison is a vise. The frontier models that match or beat our open-source picks on quality — Claude Opus 4.8, GPT-5.5 — cost roughly $4,000 and $6,000 for the same 100k-page run, 18–27× our self-hosted cost. The proprietary models that get close to our cost, like gpt-5.4-nano and gpt-4.1-mini, aren’t quality-competitive: in our review they were the ones most prone to quietly rewriting text. The example below is representative. The source sentence says researchers run registration and fusion concurrently; gpt-5.4-nano invented a different claim entirely, lifting the phrase “dynamic gradient sparsity property” from the next sentence and presenting it as the method. Nothing in the output signals that it’s wrong — which is exactly what makes it dangerous.

✓ Correct (matched by chandra-ocr-2 and dots.ocr-1.5)

To address the complexities of image registration task, certain researchers endeavor to execute registration and fusion processes concurrently.

✗ Hallucinated (gpt-5.4-nano)

To address the complexities of image registration task, certain researchers endeavor to implement a dynamic gradient sparsity property.

To sanity-check the earlier token-budget estimates, we also ran a sample of our own pages through several proprietary models and recorded the provider-reported costs. Extrapolating from those observed costs, this is what the same 100,000-page workload would look like.

Two things to take away from the table. First, read it by quality tier, not by price: the cheap rows ($120–$510) are not substitutes for dots.ocr-1.5 or Chandra, because they’re the ones that hallucinate. Second, once you filter to genuinely comparable quality, the proprietary cost sits an order of magnitude or more above our $223 self-hosted run. Batch APIs can narrow that gap for jobs you’re willing to defer, but a batch endpoint is a different product the moment you need predictable sub-hour completion.

What the API comparison misses

The broad takeaway is simpler than a head-to-head matrix: self-hosted VLMs on Modal are often competitive on run cost for comparable-quality OCR, and the dollar figure is the least of it. With the propietary APIs, the cheaper model you built around can get more expensive or retired out from under you. Self-hosting trades that exposure for stronger guarantees and choices you can make around:

• The deployment knobs. You choose the model, GPU, engine, concurrency, batching strategy, and fleet size. With an API, the highest-leverage knobs belong to the vendor.

• The deadline trade-off. Finishing faster is a visible capacity decision: add GPUs, pick a different GPU class, tune the server. With APIs, sub-hour throughput depends on account tiers, rate limits, and provider capacity.

• Where the data goes. For OCR workloads, the input is often the sensitive part. Running in your own Modal account changes the privacy and compliance conversation.

If OCR is becoming core infrastructure for your product or business, owning the serving path matters.

What we learned

1. Benchmark your workload, not somebody else’s

It’s tempting to read a leaderboard, take the top row, and move on. Sometimes that’s right. Sometimes it’s exactly wrong — two of our public-ranking favorites lost once we measured them on our own pages.

What actually decides the winner is the shape of your workload: document type, token mix, output length, preprocessing path, concurrency, and the precise quality threshold you need. OCR is a particularly nasty case because it pairs heavy image inputs with long text outputs, so a model that’s fast on chat-style traffic can fall over here. The leaderboard tells you who to test; your own pages tell you who to ship.

2. GPU price and SLA have to be evaluated together

On paper, an L4 or L40S is far cheaper per second than an H100. In practice that’s a trap, because per-second price and time-to-finish trade off against each other. Our measured dots.ocr-1.5 rows make it concrete: the L4 landed in roughly the same cost-per-page band as the H100 while running ~5–6× slower. So the cheap GPU buys you nothing on cost and costs you a one-to-two-hour job turning into an overnight one. The right pick is a function of your deadline: a loose SLA makes a low-end GPU perfectly sensible, a tight one justifies the H100 even at a higher sticker price. We chose H100s for the final run on that basis — best cost/time trade-off, not cheapest line item.

3. Quantization is worth taking seriously

We started out mildly skeptical that FP8 would hold quality without leaving performance on the table. The data didn’t support the skepticism. The clearest case from the later sweeps is Nemotron, holding everything else constant:

• BF16 on H100: ~0.10 pages/s, ~$1,100 per 100k pages.

• FP8 on H100: ~0.17 pages/s, ~$650 per 100k pages.

That’s a ~70% throughput gain and a ~40% cost cut from a single knob. It wasn’t enough to make Nemotron beat dots.ocr-1.5 for this workload, but that’s the narrow point. The broad one is that quantization is a real, near-free lever on both throughput and cost, and it belongs in the production search space rather than the “maybe later” pile.

Conclusion

We went in curious about two things: how good open-source OCR VLMs have actually gotten, and how much effort it really takes to run them at scale. The answers turned out to be “very” and “less than we expected.” 100,000 pages, 56 minutes, $223, on infrastructure we stood up in an afternoon.

Hosted APIs are the right default for prototypes and one-off product work, and we’ll keep using them there. But “default” isn’t the same as “neutral.” The moment you care about throughput, cost control, batch windows, data residency, or tuning the system around a specific workload, the calculus shifts toward owning the serving path. Self-hosting isn’t always the cheapest line on a price sheet, but for comparable-quality OCR at volume it was both cheaper and more controllable — and the operational know-how you build along the way is itself a moat that compounds.

That’s the part we’d underline for anyone weighing this: owning the model layer gives you direct control over the variables that decide whether something works in production — performance, cost, quality, and predictability — instead of renting them from a vendor whose incentives aren’t yours.

And we don’t think $223 is the floor. Newer serving engines, more aggressive quantization, and more careful tuning all point downward from here. We’ll keep tracking the state of the art; our guess is there’s still meaningful headroom left.

Useful references:

• Modal’s benchmarking guide

• guidellm

• Modal’s Pricing page

I’m ashamed to admit that when I get stuck, I’ll paste a screenshot into ChatGPT o3 and ask for help. It’s freakishly good. At one point, I noticed in the chain of thought that o3 often tries to reason through the entire puzzle. So I got curious: I started pasting screenshots of completely unsolved puzzles and asking o3 to work through the whole thing. Often, it would nail the solution perfectly.

That’s when it hit me: if these models can solve complex word puzzles, how do they actually compare? Which ones excel at this kind of deep, recursive reasoning?

Claude 4 Opus solving a recent Bracket City puzzle

Building the Benchmark

My first approach seemed obvious: feed screenshots to each model and see who wins.

It worked okay, but the inference endpoints timed out constantly. Models would get halfway through reasoning and just… stop. Even when they didn’t timeout, the visual parsing was inconsistent—some models would misread brackets or lose track of nested structures entirely.

The breakthrough came when I reimplemented Bracket City’s game logic as LLM tool calls:

const tools = {
  makeGuess: {
    description: "Make a guess for a specific bracket clue",
    parameters: {
      clue: "The clue text inside brackets (without the brackets)",
      guess: "Your guess for the answer to this clue",
    },
  },

  getHint: {
    description: "Get a hint (first letter) for a difficult clue",
    parameters: {
      clue: "The clue text inside brackets (without the brackets)",
    },
  },

  revealClue: {
    description: "Reveal the full answer for a clue (last resort)",
    parameters: {
      clue: "The clue text inside brackets (without the brackets)",
    },
  },
};

This approach worked beautifully. Models could now interact with puzzles programmatically, maintaining state while focusing purely on the reasoning challenge. I gave each model up to 50 steps to solve a puzzle, with the same scoring rules as human players: start at 100 points, lose 2 for wrong guesses, 5 for hints, and 15 for reveals.

The system prompt emphasized working from the innermost brackets outward—a key strategy for solving these puzzles efficiently. Models that understood this recursive pattern performed significantly better.

The Results

I tested 16 frontier models across 20 different Bracket City puzzles. The results revealed a fascinating tradeoff between accuracy and efficiency:

The Top Scorer: o3-high

• Average score: 92.11/100

• Success rate: 100%

• Average time per puzzle: 11 minutes

The Efficiency Champion: Claude 4 Opus

• Average score: 88.9/100

• Success rate: 100%

• Average time per puzzle: 3 minutes

While o3-high technically won with the highest score, it came at a steep cost—taking nearly 4x longer than Claude 4 Opus to achieve only marginally better results. This raises a crucial question: is a 3.2% improvement in accuracy worth quadrupling your inference time?

Here’s the complete leaderboard:

The most shocking results come from the bottom of the table. OpenAI’s “reasoning-optimized” mini models are disasters—o3-mini spent an absurd 27 minutes per puzzle to achieve a pathetic 39.13 average score. That’s nearly 9x longer than Claude 4 Opus for less than half the performance. Meanwhile, humble GPT-4.1 managed a respectable 45.75 score in just 33 seconds.

The Time-Performance Paradox

The benchmark reveals a critical insight about modern AI systems: more thinking time doesn’t necessarily mean better thinking. Look at these striking comparisons:

• Gemini 2.5 Pro (06-05) scored 70.3 but took nearly 20 minutes per puzzle

• Gemini 2.5 Pro (05-06) scored 62.8 but finished in just 1 minute

• o3-mini (high) spent 27 minutes thinking to score worse than GPT-4.1’s 33-second performance

This suggests that many models are stuck in inefficient reasoning loops rather than making meaningful progress. Claude’s models consistently demonstrate the best balance—they think efficiently, exploring productive paths rather than spinning their wheels.

Why This Matters

For real-world applications, the time-performance tradeoff is crucial. Consider these scenarios:

• Customer support: Would you rather wait 11 minutes for a 92% accurate response or 3 minutes for an 89% accurate one?

• Code debugging: Is a slightly better bug fix worth 4x the debugging time?

• Research assistance: Do you need the absolute best answer, or the best answer you can get in a reasonable timeframe?

In most cases, Claude 4 Opus represents the sweet spot—near-peak performance at practical speeds. The 3.2% accuracy gap between it and o3-high is negligible for most use cases, but the time difference is substantial.

The Surprising Winners and Losers

Winners:

• Claude models dominated the top spots with consistent speed and accuracy

• Grok-4 (86.15 score, 5.2 min) showed X’s model can compete with the best

• GPT-4.1 delivered respectable performance at lightning speed

Losers:

• o3-mini and o4-mini are embarrassments—slow AND inaccurate

• Gemini 2.5 Pro (06-05) took 20 minutes for middling results

• Qwen3-235b spent 25 minutes per puzzle for the worst performance

The failure of OpenAI’s mini models is particularly damning. These were supposedly optimized for reasoning tasks, yet they performed worse than general-purpose models while taking 30-50x longer. It’s a cautionary tale about the dangers of optimizing for the wrong metrics.

The Real Lesson

This benchmark teaches us that in AI, as in life, perfection is often the enemy of good. o3-high’s marginal victory comes at such a steep time cost that it’s rarely the right choice for practical applications. Claude 4 Opus emerges as the real winner—fast enough for interactive use, accurate enough for serious work.

The results also expose the hollowness of some “reasoning-optimized” claims. True reasoning capability isn’t about thinking longer—it’s about thinking better. Claude’s models demonstrate this perfectly, efficiently navigating solution spaces while others get lost in computational dead ends.

For developers building AI products, the message is clear: optimize for the full user experience, not just benchmark scores. A slightly less accurate model that responds in seconds will create more value than a marginally better one that makes users wait.

Want to test your own models or see the full results? The benchmark code is available at [github.com/redspringxyz/bracket-city-benchmark].