AI Evaluation Frameworks: How Production Teams Test LLMs
LLM evaluation frameworks compared (LangSmith, Langfuse, Phoenix, Promptfoo). Eval methodology, LLM-as-judge patterns, and CI integration for production AI.
Production AI evaluation frameworks compared (LangSmith, Langfuse, Phoenix, Promptfoo). The eval methodology that catches regressions, LLM-as-judge patterns, and CI integration with concrete examples.
- Best for LangChain teams: LangSmith (native to LangChain stack)
- Best open-source: Langfuse (full feature self-hostable)
- Best ML-native: Arize Phoenix (drift detection, embeddings analysis)
- Best for CI/CD: Promptfoo (yaml configs, GitHub Actions)
- The verdict: Pick framework by stack alignment. The methodology matters more than the tool.
AI evaluation is the production discipline most teams skip. The cost of skipping: ship regressions silently when prompts change, models update, or retrieval breaks. The four leading frameworks each fit different workflows. We tested all four on production LLM apps over 90 days.
01At a glance: what we tested
| Framework | Hosting | Best | Pricing |
|---|---|---|---|
| LangSmith | Managed (or self-hosted Enterprise) | LangChain teams | $39/user/mo Plus |
| Langfuse | Self-hosted or managed | Open-source / self-hosted | Free + $59/mo cloud |
| Arize Phoenix | Self-hosted or Arize cloud | ML teams + drift detection | Free + Arize cloud pricing |
| Promptfoo | CLI / open-source | CI/CD integration | Free open-source |
| OpenAI Evals (compared) | Self-hosted | OpenAI-specific | Free open-source |
| Helicone (compared) | Managed proxy | Cost monitoring + basic eval | Free + $20/mo Pro |
02Methodology: build eval set before going to production
50-200 representative prompts with expected outputs is the production minimum. Run on every prompt change, model swap, retrieval change. Catches regressions in CI.
Buy if: not applicable. Skip if: not applicable.
Eval methodology matters more than tool choice. The pattern: (1) Curate 50-200 representative prompts covering common cases + known edge cases. (2) For each, define “good answer” criteria (exact match for structured output; rubric-based for subjective output). (3) Run eval after every prompt / retrieval / model change. (4) Block deploys on regressions. (5) Add new edge cases as they emerge in production. This discipline catches 80% of production regressions before users see them.
03LLM-as-judge: scales eval beyond exact-match
LLM-as-judge (use a model to grade outputs against rubric) handles subjective evaluation that exact-match can’t. Pattern: rubric prompt + judge model + score aggregation.
Buy if: not applicable. Skip if: not applicable.
Pattern: provide judge model with (1) rubric for what makes a good answer, (2) the prompt, (3) the model output, (4) instruction to score 1-5 with reasoning. Use Sonnet or GPT-5 as judge. Strong reasoners with stable judgment. Calibrate with human-graded sample (~50 outputs). Once calibrated, LLM-as-judge agrees with human judgment 85-92% of the time on production evals. Cost: $0.005-0.02 per evaluation. Cheap relative to the value.
04CI integration: block deploys on eval regressions
Run eval on every PR. Compare new vs baseline scores. Block deploys when scores drop. Promptfoo + GitHub Actions is the cleanest setup.
Buy if: not applicable. Skip if: not applicable.
CI integration is the production discipline most teams skip. The pattern: on every PR that touches prompts / retrieval / models, run the eval set. If aggregate score drops > X%, block the PR. If individual prompt score regresses, surface for review. Promptfoo is purpose-built for this with yaml configs + GitHub Actions. LangSmith and Langfuse support similar workflows. Without CI integration, eval is theater. You have the framework but don’t use it where it matters.
05Which option should you pick?
Pick by your situation
- Your stack is LangChain? → LangSmith
- You need self-hosted / open-source? → Langfuse
- You’re ML-team-shaped with drift detection needs? → Arize Phoenix
- You want CI/CD integration with yaml configs? → Promptfoo
- You’re OpenAI-only and want simple? → OpenAI Evals
- You want cost monitoring + basic eval? → Helicone
06FAQ
How big should my eval set be?
50-100 prompts is the production minimum. 200-500 is comfortable. Past 500, returns diminish on detecting regressions; growth helps on coverage of new edge cases. Add 5-10 prompts per month from production failures discovered in the wild.
How do I avoid eval-set bias?
Three patterns. (1) Sample eval set from real production traffic, not artificial cases. (2) Include adversarial / edge cases (5-10% of set). (3) Refresh monthly with new patterns. Without these, you optimize for the eval set instead of production reality.
Can I use ChatGPT as my judge model?
Yes for development. For production CI, use API access (Anthropic / OpenAI / Google API) so you can version-control judge model + prompt. Avoid using the same model for generation and judging. The model has bias toward its own outputs. Cross-model judging (Claude generates, GPT-5 judges) is best practice.
What’s the difference between offline and online eval?
Offline = eval set with ground truth, run during development / CI. Online = monitoring live traffic for quality (no ground truth, use heuristics). Both matter. Offline catches regressions before deploy; online catches drift after deploy. Most teams start with offline; mature ones add online.
How do these compare to OpenAI Evals?
OpenAI Evals is open-source CLI for OpenAI-specific evaluation. Free and powerful but specialized to OpenAI ecosystem. The four we tested are multi-provider with richer features (UI, datasets, traces). For OpenAI-only teams, OpenAI Evals is fine. For multi-provider production, the dedicated tools win.
07WikiWalls verdict
WikiWalls verdict. AI evaluation is real engineering discipline. The four frameworks fit different workflows; pick by stack alignment. The methodology matters more than the tool: 50-200 eval prompts, LLM-as-judge for subjective grading, CI integration to block regressions. Without this, you ship silently broken AI.
Last reviewed by WikiWalls editorial with current pricing, first-party benchmark data, and tested production reliability. Recommendations are editorially independent.
Last reviewed by WikiWalls editorial. Recommendations are editorially independent. Methodology: /test-methodology/. Editorial standards: /editorial-standards/.
