Online vs. Offline AI Evals: When to Use Each

aaldersondev about 4 hours ago 2 commentsRead Article on inngest.com

⚡ Community Insights

Discussion Sentiment

33% Positive

Analyzed from 148 words in the discussion.

Discussion (2 Comments)Read Original on HackerNews

aldersondev•about 4 hours ago

Not-so-hot take: you need to measure your agent's performance in production.

Maybe-a-little-hotter-take: you should be using production data for evaluation rather than synthetic events.

Enter Online and offline evals; one measures performance in real time. The other, off the execution path, either before deployment, or after the agent run has happened. When to use both is explained in our latest guide!

gravypod•about 2 hours ago

I've found a good balance in looking through sessions based on user feedback (ex: user thumbs down and submits it into a review queue) to build an understanding of usage patterns and failure modes. This is very boring but critical to understanding how the LLM is trying to navigate the problem. Then you use your insights to create offline evals. Then hill climb those. Then A/B test the changes with your uses.

I don't know if this is the end state, but I think this is pretty close to what I was seeing a year ago and what we will be seeing next year. Yea, maybe we use agents to do the looking through but you will still be shepherding that.