How a non-technical founder can define and measure”good” before a single line of code is written.
If you’re planning an AI product, there’s a piece of work you can do right now, before you hire an engineer or open a code editor, that will save you months of wandering: write your evals.
An eval is just a repeatable test for an AI system. You give it an input, it produces an output, and something — a rule, a rubric, or another model — judges whether the output is good. Simple, and doable for non-engineers. There’s a growing consensus among AI product leaders that evals are the key spec for AI products, rather than product requirement documents (PRDs). Traditional PRDs lean on words like “helpful,” “accurate,” and “concise,” which can’t really be tested. AI systems are probabilistic, meaning the same input can produce different outputs, and models change under your feet. An eval collapses the spec, the acceptance criteria, and the launch review into one runnable artifact that answers a single question: what does good look like, and how do I measure it?
One key benefit is that you don’t need an actual product to write evals. All you need is judgment about your domain. And if you’re the founder, you have more of that than anyone you’ll ever hire.
Step 1: Write down 20 real inputs
Every eval has three ingredients: a dataset (example inputs), a task (the AI doing the work), and scorers (how you judge the output). You can build the dataset and the scorers before you even have a product.
Before anything else, build the dataset. Write out twenty to thirty examples of what users will actually ask your product to do. Make them as natural as possible, based on language your target customers would use in real life. For instance, if you’re building a tool that summarizes legal contracts, go find twenty contracts. If the product is a customer support agent, write out twenty support tickets (make sure to include some “unhappy customer” ones and some “I don’t know what I actually need” ones).
Step 2: Define what good looks like in binary
Next, for each input you created, write what a great output would contain. Then turn your quality bar into yes or no questions. A few great examples: Did it cite the actual clause? Did it stay under 150 words? Did it avoid inventing numbers? Did it escalate when the customer mentioned a refund?
We’re often tempted to score things on a 1–10 scale, but resist that — it won’t be helpful in the long run. Pass or fail is hard enough, and it forces you to be brutally honest about your requirements. Most importantly, make the criteria fully yours. Avoid generic metrics and make your scorers specific to your product. High quality looks different for every product.
Step 3: Grade by hand first
Next, run your inputs through an AI model (Claude, ChatGPT, or whatever you use most often) using a rough prompt that describes what your product should do. Grade each output yourself: thumbs up or down, and include one sentence explaining why it was good or bad. That written justification is incredibly valuable: it becomes your failure taxonomy (”made up a policy,” “buried the answer,” or “wrong tone for a complaint”) and each failure mode becomes a scorer later on.
This stage is sometimes called vibe-checking. Don’t skip it. Never automate a judgment you haven’t first made yourself, repeatedly, on real examples. A spreadsheet with your labels is a completely legitimate eval system.
Step 4: Make it repeatable with Braintrust
Once your hand-graded spreadsheet exists, a tool like Braintrust turns it into something you can run over and over without writing code. You can paste your inputs and expected outputs into its dataset editor, write your prompt in the browser-based playground, and run it against the whole dataset at once, comparing prompts and models side by side. Its AI assistant, Loop, will even draft scorers from a plain-English description (”check that the summary never invents a dollar figure”). You describe your bar for quality in a sentence or two, then Loop will write the grading logic, and then you can verify if the logic agrees with your hand labels before using it. There’s a free tier, and the whole workflow is usable by a non-engineer. (Langfuse is a strong open-source alternative.)
Now, without hiring an engineer, you can change the prompt, rerun the eval, and know whether it got better.
Evals build a strong foundation
By the time you engage engineers, whether that’s a technical co-founder or a team like ours, you’ll be able to hand them a runnable definition of success instead of a prose wishlist. They can test three models against your dataset in an afternoon and tell you which one clears your bar (and what it costs). Every future change will get measured against the same benchmark, so improvements build on each other.
There are two warnings before you dive in. First, don’t get set on a perfect pass rate. If everything passes, your eval is too easy, and you’re optimizing for your own comfort rather than quality. Second, don’t let perfectionism stall you: an imperfect set of twenty examples you can come up with in a week beats “perfect” dataset that takes months to be designed by a committee. Once your product launches, real user data will reshape the dataset anyway. Production failures quickly become new test cases, and the eval will grow with the product.
There’s no secret access to better AI models. Everyone has the same ones. The secret to success is defining “good” outputs precisely and early on — and using those benchmarks going forward. You can start that today. All you need is a spreadsheet.
Redspring is a boutique AI and software consultancy in New York. Your founding engineering team, on demand. If you’ve got a dataset and a dream, we’d love to talk.