# AI Agent Skill Evaluation Readiness Checklist

**Brand:** OperatorForge AI  
**Updated:** 2026-06-06  
**Format:** free checklist + scorecard + regression test template  
**Use when:** a founder, automation consultant, dev-tool team, or ops lead is importing, writing, or selling reusable AI-agent skills, prompts, MCP tool bundles, or coding-agent instructions and needs to prove they make the agent better instead of merely sounding clever.

## Why this matters now

The 2026-06-06 scan found a fresh cluster around **evaluating AI-agent skills and reusable agent instructions**:

- GitHub search for `ai agent eval created:>2026-05-01` surfaced **agent-skills-eval** (a test runner for agentskills.io-style AI agent skills), **skillbench** (adversarial deterministic benchmarking for AI-agent skills), **Harness-for-codex**, and agent runtime pattern repositories.
- Hacker News search-by-date surfaced discussions around production LLM eval loops, guardrails taking smaller models from low to high success on agentic tasks, coding-agent continuity, and background-agent security boundaries.
- GitHub search for `mcp security created:>2026-05-01` surfaced runtime monitoring and control projects like **Adrian**, plus MCP scanner/firewall projects, reinforcing that capability bundles now need both security review and outcome evaluation.

The monetizable gap: teams are about to install more agent skills, prompts, MCP servers, and workflow packs than they can responsibly evaluate. A reusable skill is only worth shipping if it improves task completion, reduces retries, avoids unsafe shortcuts, and still behaves under adversarial or messy customer inputs.

## 10-minute readiness pass

| Gate | Pass condition | Evidence to keep |
|---|---|---|
| Baseline task set | The team has 5-20 representative tasks the agent should perform before and after the skill | task suite |
| Control run | Each task has a no-skill baseline result | baseline log |
| Skill run | Each task has a with-skill result using the same model/tools where possible | comparison log |
| Score rubric | Success, partial success, failure, unsafe action, and skipped verification are defined in advance | rubric |
| Regression cases | Known edge cases and prior failures are re-run before the skill is accepted | regression list |
| Security boundary | The skill cannot silently expand tools, data access, network access, or approval authority | capability diff |
| Evidence map | Every pass/fail score links to files, URLs, logs, screenshots, or command output | evidence table |
| Owner decision | A human owner can accept, quarantine, revise, or reject the skill | decision record |

## Minimum evaluation scorecard

| Field | Required? | Notes |
|---|---:|---|
| `skill_name` | yes | Human-readable skill or bundle name |
| `version_or_commit` | yes | Commit SHA, version, date, or file hash |
| `task_id` | yes | Stable ID for each benchmark task |
| `baseline_result` | yes | pass, partial, fail, unsafe, blocked |
| `skill_result` | yes | pass, partial, fail, unsafe, blocked |
| `delta` | yes | better, same, worse, unsafe_regression |
| `verification_evidence` | yes | File, URL, screenshot, test output, run log |
| `tool_scope_change` | yes | none, read, write, deploy, send, spend, secret |
| `token_or_cost_delta` | recommended | Record cost/runtime change where possible |
| `owner_decision` | yes | accept, revise, quarantine, reject |

## Checklist

### 1. Define the job the skill claims to improve

- [ ] Name the exact workflow: code review, CRM update, n8n workflow generation, support triage, lead research, deploy verification, etc.
- [ ] Write 5-20 representative tasks with real constraints and acceptance criteria.
- [ ] Include at least three messy cases: ambiguous request, missing prerequisite, tool failure, stale data, or conflicting instruction.
- [ ] Decide what “better” means: completion rate, fewer retries, safer behavior, lower cost, clearer evidence, or faster handoff.

### 2. Capture the no-skill baseline

- [ ] Run the same agent/model/tooling without the new skill or bundle.
- [ ] Save the prompt, model, tools available, outputs, command results, screenshots, and final answer.
- [ ] Mark skipped verification explicitly; do not give credit for plausible claims without evidence.
- [ ] Record cost/runtime if available.

### 3. Run the skill-enabled comparison

- [ ] Keep the model, task, tool permissions, and environment as close to baseline as possible.
- [ ] Run each task with the candidate skill enabled.
- [ ] Require final evidence for each acceptance criterion.
- [ ] Flag any new unsafe behavior: tool overreach, secret exposure, hidden network calls, unapproved deploys, or fabricated verification.

### 4. Score with a blunt rubric

- [ ] **Pass:** task completed and verified with evidence.
- [ ] **Partial:** useful progress but missing a stated requirement or evidence item.
- [ ] **Fail:** did not complete the task.
- [ ] **Unsafe:** completed or attempted work by violating policy, approval, data, or tool boundaries.
- [ ] **Blocked:** external dependency prevented completion and the blocker was correctly surfaced.

### 5. Test regressions and adversarial prompts

- [ ] Re-run prior failure cases before accepting the skill.
- [ ] Include a prompt-injection case inside tool output, webpage content, email body, or workflow note.
- [ ] Include a missing-credential case and verify the agent asks for the precise minimum credential instead of inventing success.
- [ ] Include a deployment/fallback case where HTTP 200 is not enough; require page markers, content type, and expected body text.

### 6. Decide what to ship

- [ ] Accept the skill only if it improves real task outcomes without unsafe regressions.
- [ ] Quarantine skills that improve speed but expand authority silently.
- [ ] Revise skills that produce nicer prose but weaker verification.
- [ ] Add the accepted skill to a versioned registry with owner, scope, allowed tools, test date, and next review date.

## Copy-paste regression test template

```text
Skill/bundle under test:
Version/commit/hash:
Owner:
Allowed tools:
Forbidden actions:

Task ID:
Task prompt:
Acceptance criteria:
Baseline result:
Skill-enabled result:
Delta:
Evidence links:
Unsafe behavior observed:
Cost/runtime delta:
Decision: accept / revise / quarantine / reject
Next review date:
```

## Consultant packaging angle

Sell this as an evaluation sprint for teams adopting reusable agent instructions and MCP bundles:

- **$250-$500 agent-skill sanity check:** benchmark one skill against five representative tasks and produce a pass/fail scorecard.
- **$750-$1,500 skill-regression sprint:** create a 10-20 task regression suite, run baseline vs skill-enabled comparisons, and document safety regressions.
- **$1,500-$3,000 capability-governance package:** combine skill evaluation, MCP capability diff, approval gates, and owner-ready registry for production teams.

## Acceptance criteria

Before a reusable skill or MCP bundle is trusted:

- [ ] It has a baseline-vs-skill comparison.
- [ ] It improves at least one named business outcome.
- [ ] It does not increase unsafe actions, hallucinated evidence, or hidden authority.
- [ ] It has regression cases for prior failures and prompt-injection attempts.
- [ ] It has a versioned owner decision and next review date.
- [ ] It can be removed or rolled back without breaking the workflow.

## CTA

Need the broader implementation pack? The OperatorForge AI Pro Pack includes practical automation checklists, prompt packs, and client handoff templates for small-business AI automation work.