With traditional engineering, math, or even rocket science, there's almost always a right way, a right formula, and the most important of them all - a Eureka moment! Think of the exact speed to catch a falling rocket between two metal arms, or the provably shortest route for the traveling salesman. However hard the problem, there's a way to deterministically confirm you got it right.

Generative AI flips that dynamic: even when you can write down a hypothesis, the system’s behavior is probabilistic and the ‘right’ output is often subjective, which makes proving correctness much harder. In this blog, we are going over exactly how our teams at Zip tackle this. We will break down how we run evaluations for our AI systems, how we leverage LLMs as judges to scale our grading, and how we build continuous feedback loops to ensure our agents get smarter with every run.

The Raw Material: Where Do Tests Come From?

Before you can grade an agent, you need a robust set of scenarios to test it against. You need inputs that accurately reflect situations your agent will face in the wild. When it comes to datasets, there’s two simple kinds:

1. Online Data: This is sampled, real-world customer traffic - based on customer training approvals and anonymization to remove PII (Personally Identifiable Information). It is incredibly valuable because it represents true user behavior with built-in signals (e.g. manually correcting an AI's extraction), though extremely tricky to use since it comes with a lot of legal/compliance restrictions on what you can use this information for.
2. Offline Data: Data that is customer-looking but not actually belonging to the customers. You own, control, and curate the data. This is further of two types:‍
   
   • Human-Curated Data: This has been the industry standard for years. You have some manually crafted set of inputs and scenarios that you evaluate against. Can yield extremely high-quality, nuanced test cases, but is extremely time intensive and heavy human labor.‍
   • Synthetic Data: Use AI to generate massive batches of inputs and scenarios, and make it as realistic as you can get it to be.

When should you use what dataset?

1. Building a new application for a niche problem: At Zip, when we worked on AI Contract Orchestration, we were building agentic workflows for legal teams - often including flagging legal risks specific to in-house counsel and their playbooks. In this case, we relied heavily on human curated datasets as gold standard.
2. Building a new solution for a widely discussed problem: Working on Document Intelligence, we need to understand complex objects - PDFs, Excel, Image scans, tables, etc. Given this is a well charted problem in industry, we can find existing datasets from research labs, or generate gold quality truth from existing samples. A mix of synthetic/open source + human curated dataset is therefore ideal here.
3. Improving an existing application: Zip processes thousands of workflows a day. We have a massive well of data showing exactly what customers need when filling out requests. To add additional AI capabilities to it, we leverage anonymized online data (strictly adhering to compliance) to continuously evaluate and improve our AI systems.

Ultimately, there is no single correct approach here. The most effective data strategy is simply the one that best matches the maturity of the product and mimics the reality of the use case.

How We Grade: With an Answer Key, and Without

How you grade an agent shapes how you build its tests, so we settle it up front. Evaluation is based on one key question: does the output have a single right answer to check against, or not?

Grading with an answer key: deterministic checks

When an output has a concrete "right" answer, like a dollar amount or an invoice number, grading is easy as we check the AI's value against it and move on. No latency, low cost, and the same input scores the same way every time.

The catch is where that "correct" value comes from. Traditionally someone labels it by hand, which is slow and caps how much you can realistically test. A cheaper and more reliable alternative is to get the right answers from the users themselves.
Take our agentic prefill at user intake: once the AI auto-fills a request form, the user tweaks whatever looks off before submitting. This becomes the inspiration for source of truth, since it's exactly what the AI should have produced to save the user from fixing it at all.

Even with labels in hand, deterministic checks only go so far. The same value can be written many correct ways: a user picks "Acme, Inc." where the AI wrote "Acme Incorporated," and a literal match calls a right answer wrong. And most of what our agents produce isn't a single value at all. Once the output is a paragraph or a chain of actions, whether it's accurate, concise, and useful is a judgement call, not something you can look up and match exactly.

Grading without an answer key: trace-based judges

So we skip the answer key and let a model grade instead. Each case pairs the input with a rubric: not the exact answer we expect, but a description of what a good response looks like for this kind of request. That subtle distinction is what makes it scale. A single agent handles the same class of request in countless variations, and we can't write down the right answer for every one. What we can pin down is what "good" means for that class: which tools or data the agent should reach for, the logic it should follow, and what a strong final response includes. We write that once, per agent or per query type, and the judge applies it to each request on its own merits.

The unlock isn't the judge, it's what the judge reads. Rather than a stored answer, it reads the agent's execution trace, the actual record of the run: which tools fired, what came back, and what the agent decided at each step. It scores that trace against the rubric, so we grade the whole journey, not just the final line.

For chatbots, this trace spans an entire conversation. Given all the possible back-and-forth messages with the user, we evaluate the interaction at two levels: every individual turn of the conversation, and the conversation as a whole. These levels catch opposite failures. The AI can nail every single response but still leave the user stuck; conversely, it can get the overall arc right while completely fumbling a single response along the way.

What each run reports depends on the eval type:

• Single-turn runs return recall, precision, and helpfulness.
• Multi-turn adds conciseness to this and scores both per turn and overall.
• Tasks with ground-truth data, like field extraction, report accuracy and coverage.

These are just defaults. Every metric is backed by its own rubric, and since rubrics live as editable prompts rather than code, a team can retune what "helpful" means for their product, or add a brand-new dimension, without shipping a change.

A product onboards by registering three things: a dataset, a way to run it, and the set of evaluators it wants scored. The framework stays agnostic about the rest, running the target and tracking whatever those evaluators output each run.

Where These Tests Run: Offline, CI, and Online

The same scorers run in three distinct settings, each serving a different purpose:

• Offline: Runs against a fixed, curated dataset, so it's fully reproducible. With no dependency on live traffic, it can be run over and over on demand. An engineer makes a product or prompt change, reruns the same eval, and compares to baseline in a tight loop before anything ships. And because that loop is just run-test-compare, a coding agent can drive it entirely on its own, iterating change after change until the metric reaches the bar we set. And it's not only engineers: teams building custom agents lean on the same offline runs to test their agent as they develop it, well before it goes live.
• CI/CD: Runs the offline suite on a schedule (e.g., twice a day on staging). It features a threshold on the primary metric that alerts the team if quality drops. This is the standing guardrail.
• Online: Points the same rubric at sampled production traffic. Its job is to provide a snapshot of real-world quality, catching the messy, unexpected cases nobody thought to add to the dataset.

Offline tells you whether a change is safe to ship; online tells you how you're actually performing in front of customers. The reason to keep the two in sync isn't just that a match means your offline data is realistic. The real payoff is confidence: when offline and online track each other, a gain you earn offline is one you can trust will show up in production once you ship it.

Closing the Loop: From Signal to Action

A score only matters if it leads to a fix. We watch for four signs that the AI got something wrong:

• A quiet correction. A user edits the AI's answer before submitting. Whatever they changed it to is what the answer should have been.
• A thumbs-down on the AI response. Someone tells us directly that a response was lacking.
• A low eval score, from a test run offline or on live traffic.
• A drift in cost or speed, when responses get slower or more expensive than they should be.

These signals pile up fast, and reading them one by one is not scalable, so we don't. A triage step groups the signals by what they have in common and points at the likely culprit: a prompt that needs work, missing data, a wrong setting, a rubric that's grading unfairly, or a real gap in the product. The engineer is presented a targeted action item to focus on.

The loop closes on the last step. Whenever a real failure surfaces, we turn that exact situation into a new test case and add it to the golden set. The suite grows from the AI's own mistakes, so today's bug becomes a check that runs forever after. We don't get to make the same mistake twice.

Improving the System

Catching failures is only half the job. The other half is knowing whether the product is actually getting better, and for that you need a number you can watch. We pick a metric that matters, like accuracy or helpfulness, and track it run over run. Without that baseline you might know a tweak fixed today's bug, but not whether the product is any better than it was last month.

When the number slips, we work through three questions in order:

• Is the eval score even right? A bad score can either mean a failing agent or a judge that's grading unfairly. We check the judge against human opinion first and then a second LLM judge to evaluate the evaluating judge, because tuning the product to a broken rubric just chases a phantom problem.
• What actually broke? Match the fix to the cause. Bad wording gets fixed in the prompt, broken logic in the code, and a weak or outdated test set gets cleaned up directly.
• Does the fix hold up? An engineer tests it against production-like cases right from their dev setup, compares the new score to the baseline, and iterates on it. The cases we promoted from real failures are the final gate: a change can raise the average and still be blocked for breaking even one of them.

Takeaways + What’s next?

Building this system taught us things no tweet thread warned us about. A few were expensive to learn.

1. Calibrating the rubric became more important than ever. We assumed grading would be hard; instead the work shifted to writing and maintaining a rubric. A vague rubric yields confident but useless scores. And rubrics could drift as product capabilities evolve - what counted as “good” yesterday quietly stops being good if the judge starts rewarding the wrong thing.
2. Evaluating the evaluator is half the job. We audit our judges on past evaluation runs far more than expected. A low score is a hypothesis, not a verdict - it may signal a broken rubric as much as a broken agent. Optimizing against a miscalibrated judge is the costliest way to raise a number while quality drops.
3. The gap between offline and online is our most useful signal. We chase offline ≈ online not for matching numbers, but because divergence is information. When they pull apart, it usually means production is seeing cases our dataset never imagined—which are exactly the ones to add next.

Philosophers claim AGI is coming. Would we need evals then? Probably not for the engineering problems we consider problems today. Similar to how AI capabilities evolve, the shape of evals will evolve into something much different. Evals aren’t necessarily about whether the AI is technically right; they are about whether the customer actually agrees that it is right. And being "right" is highly subjective- what an LLM considers the perfect answer doesn't matter if it fails to solve the user's actual problem. As long as we are building for humans, we will always need evals to bridge that gap between the model's logic and the customer's reality.