Show HN: Caliper – pass@k reliability testing for Claude Code and Codex skills
Caliper is a lightweight, local CLI tool for testing skills used with agentic coding tools (Claude Code, Codex). It runs a skill multiple times in isolated environments to produce a pass@k reliability score, uses a YAML spec to define tasks with LLM judges or Python assertions, includes a baseline flag to measure the skill's actual impact, and ships with companion skills to manage evaluations.
Background
Caliper is a lightweight open-source tool from GitHub user edonadei for testing "skills" — custom instruction sets or tools that enhance AI coding agents like Anthropic's Claude Code, OpenAI's Codex, or Pi. The core problem: these skills are non-deterministic (an LLM may succeed once and fail the next time), so a single test pass is misleading. Caliper runs a skill K times in isolated environments and reports a pass@k score (e.g., 4/5 = 80%). It also has a --baseline flag that runs the same tasks *without* the skill, showing the delta — whether the skill actually improves results over the raw model. Evaluations are defined in YAML files and can use either an LLM-as-judge or Python assertions to check outputs. The project also ships two companion skills: "evaluate-skill" (run Caliper evals from within Claude Code/Codex) and "grill-skill" (interviews you about your goals, then auto-generates a 3-task eval spec). This sits in the broader trend of "agentic evaluations" — the growing need for systematic, reproducible testing of LLM-based agents as they move from demos to production.