Lv.2 · Quality Gate

AI Harness

An AI harness is the execution layer for testing LLM features. It combines input samples, expected conditions, forbidden behavior, grading rules, and reports to detect regressions.

Why build this first?

RAG, skills, and local LLMs can all change behavior. A harness lets you separate real improvement from vibes.

Minimum Parts

golden set: real user questions and expected conditions
rubric: pass, partial pass, and fail criteria
runner: script that calls the model and stores output
report: reproducible failed cases and failure reasons

Eval Case Example

evals/refund-policy.json

{
  "id": "refund-policy-001",
  "input": "What is the refund policy?",
  "expected": ["mentions the refund window", "includes a source document"],
  "forbidden": ["states policy without evidence", "creates a fake link"],
  "tags": ["policy", "rag", "customer-support"]
}

Failure Report Example

eval-report.md

# AI Eval Report
- total: 20
- passed: 17
- failed: 3

## Failed cases
1. refund-policy-001
   - missing citation
   - answer used stale policy wording

Release Criteria

Critical cases must pass 100% before release
Answers requiring evidence fail when citations are missing
Compare model changes with the same table
Classify failures as RAG, prompt, model, or data issues

Next Actions

Eval Harness Recipe

Generate a harness draft from project inputs

RAG Basics

Connect retrieval quality to evals

Last updated: June 22, 2026 · Version: v0.0.1

Send Feedback

Opens a new issue page with your message.

Open GitHub Issue