Evaluation-Driven Development in Legal AI

In legal AI, evaluation is rarely as simple as comparing model outputs to a fixed ground truth. The underlying tasks are subjective, domain-heavy, and closely tied to real workflows, which means that even experts can disagree on what “correct” looks like. In this talk, I’ll share how we approach evaluation-driven development at Filevine by combining three feedback loops: structured annotation from an in-house labeling team, rubric design with domain experts, and production signals from real user behavior. I’ll show how we break complex AI workflows into measurable steps, where automated metrics help, where they fail, and why production feedback often reveals issues that offline benchmarks miss. The result is a practical framework for building AI systems that are not just impressive in demos, but reliable in real-world use.