Evaluating AI Agents: Lessons from Code Security

Everyone building agentic systems faces the same question: how do you know your agent is doing what you think it's doing? The honest answer is deceptively simple, you look. But looking well turns out to be surprisingly hard. Agent evaluation borrows from software testing, ML benchmarking, and empirical research methods, but fits none of them cleanly. We're trying to evaluate inherently stochastic systems with tools designed for deterministic ones.