AI Extension · Quality

Make an Eval Harness

Build a harness that validates AI features against the same inputs and criteria before release.

When to use

Create this before changing prompts, models, RAG data, or chunking strategy.

Eval Harness Design Prompt

Copy and use

"I want to build an eval harness for an AI feature. Design a golden set, grading criteria, run command, and failure report format using the information below. [Feature] - Feature: [e.g. document-grounded support answers] - Input shape: [e.g. user question + user tier] - Output shape: [e.g. answer + source array] - Required policy: [e.g. do not answer without sources] [Output requirements] 1) eval case JSON schema 2) criteria for at least 20 golden cases 3) pass/partial/fail rubric 4) command to run in CI 5) Markdown failure report template"

Copy this prompt and paste it into your AI chat.

Case Format Example

eval-case.json

{
  "id": "source-required-001",
  "input": "What is the refund policy?",
  "expected": ["includes refund window", "includes citation"],
  "forbidden": ["states policy without source", "adds content not in policy docs"]
}

Execution Checklist

0/5 done

Lock feature scopeLimit to one API or one user flow

Write golden setInclude normal, edge, and failure cases

Write rubricDefine expected conditions, not just ideal answers

Connect run commandUse the same command locally and in CI

Save failure reportRecord reproducible input and failure reason

TL;DR

Use this section as a quick baseline for eval harness.
Replace branch names, secrets, URLs, and thresholds with your team-specific values.

Prerequisites

Confirm repository and runtime access permissions.
Verify baseline commands (`install`, `build`, `test`) for local/CI.
Ensure environment variables, secrets, and deployment targets are up to date.

Steps

Lock the change goal and scope in one sentence.
Apply the core eval harness changes in minimal commits.
Capture validation logs and immediately update checklist/docs.

Validation

Confirm build/test pass reproducibly on the current branch.
Run manual sample checks on critical user/operation flows.
Cross-check ko/en docs and links for intent consistency.

Troubleshooting

Capture logs, request payloads, and environment values first on failures.
Narrow down by permissions, env vars, routing, and build artifacts.
Record temporary mitigation and root-cause fix separately.

References

AI Harness

Harness concepts and pass criteria

Testing Basics

Connect with unit, integration, and E2E tests

Last updated: June 22, 2026 · Version: v0.0.1

Send Feedback

Opens a new issue page with your message.

Open GitHub Issue