Skill Creator (Anthropic, official)

tl;dr: a meta-skill for the skills system itself. Same shape as the four document skills (it’s a routing layer, not a tool), but instead of routing to a Python/NPM stack, it routes to a workflow: capture intent → interview → draft SKILL.md → write test prompts → run with-skill + baseline in parallel → grade → ship qualitative + quantitative outputs to a reviewer → revise → repeat. There’s a separate description optimization loop that uses a held-out eval split to avoid overfitting.

What it actually does

Per the frontmatter, this skill activates when the user wants to create a skill from scratch, edit or optimize an existing one, run evals to test a skill, benchmark performance with variance analysis, or optimize a description for better triggering accuracy.

The core loop (verbatim from SKILL.md):

• Decide what the skill does and roughly how
• Write a draft
• Create a few test prompts, run claude-with-access-to-the-skill on them
• Help the user evaluate qualitatively and quantitatively — while runs are in the background, draft evals.json assertions and use eval-viewer/generate_review.py to show the user the outputs alongside metrics
• Rewrite based on feedback and quantitative benchmarks
• Repeat until satisfied
• Expand the test set, try again at larger scale

The skill’s job is to figure out where the user is in this loop and jump in. “I want to make a skill for X” → narrow it down, draft, write test cases, run, grade. “I already have a draft” → straight to eval/iterate.

Anatomy of a Skill (SKILL.md contract)

skill-name/
├── SKILL.md (required)
│   ├── YAML frontmatter (name, description required)
│   └── Markdown instructions
└── Bundled Resources (optional)
    ├── scripts/    - Executable code for deterministic/repetitive tasks
    ├── references/ - Docs loaded into context as needed
    └── assets/     - Files used in output (templates, icons, fonts)

Three-level loading: metadata (~100 words, always in context), body (<500 lines ideal, loaded when triggered), bundled resources (loaded as needed). Keep SKILL.md under 500 lines; if approaching the limit, add a layer of references and point at them.

The iteration loop (the meat)

Step 1 — Spawn all runs in the same turn

For each test case, spawn two subagents at once — one with the skill, one without. Don’t sequence them; launch everything so it finishes around the same time.

With-skill run:

Execute this task:
- Skill path: <path-to-skill>
- Task: <eval prompt>
- Input files: <eval files if any, or "none">
- Save outputs to: <workspace>/iteration-<N>/eval-<ID>/with_skill/outputs/

Baseline run:

• Creating new skill: no skill at all → without_skill/outputs/
• Improving existing skill: snapshot the old skill to <workspace>/skill-snapshot/ and point baseline at it → old_skill/outputs/

Per-eval eval_metadata.json (assertions empty initially):

{
  "eval_id": 0,
  "eval_name": "descriptive-name-here",
  "prompt": "The user's task prompt",
  "assertions": []
}

Step 2 — Draft assertions while runs are in progress

Don’t wait. Write objectively verifiable assertions with descriptive names so they read clearly in the benchmark viewer. Subjective skills (writing style, design quality) → qualitative review, don’t force assertions. Update eval_metadata.json and evals.json.

Step 3 — Capture timing data

Each subagent task notification carries total_tokens and duration_ms. Save immediately to timing.json in the run directory — that’s the only chance; the data doesn’t persist anywhere else.

Step 4 — Grade, aggregate, viewer

1. Grade each run — spawn a grader subagent (or grade inline) reading agents/grader.md. Save grading.json per run. Use text / passed / evidence fields — the viewer depends on those exact names.
2. Aggregate: python -m scripts.aggregate_benchmark <workspace>/iteration-N --skill-name <name> → benchmark.json + benchmark.md (pass_rate, time, tokens, mean ± stddev, delta). With-skill versions before baseline counterparts.
3. Analyst pass — read the benchmark data; surface what the aggregate hides (always-pass non-discriminators, high-variance flaky evals, time/token tradeoffs). See agents/analyzer.md.
4. Launch the viewer (must use generate_review.py, not custom HTML):

nohup python <skill-creator-path>/eval-viewer/generate_review.py \
  <workspace>/iteration-N \
  --skill-name "my-skill" \
  --benchmark <workspace>/iteration-N/benchmark.json \
  > /dev/null 2>&1 &

For iteration 2+: add --previous-workspace <workspace>/iteration-<N-1>. Cowork / headless: --static <output_path> writes a standalone HTML; feedback comes back as a downloaded feedback.json.

Step 5 — Read feedback.json

{
  "reviews": [
    {"run_id": "eval-0-with_skill", "feedback": "the chart is missing axis labels", "timestamp": "..."},
    {"run_id": "eval-1-with_skill", "feedback": "", "timestamp": "..."}
  ],
  "status": "complete"
}

Empty feedback = fine. Focus revisions where the user had specific complaints. Kill the viewer server when done.

Improving the skill (the part that decides quality)

The skill is explicit about four patterns:

1. Generalize from the feedback. A skill used “a million times” must not be tuned to three test cases. If a stubborn issue persists, try a different metaphor or pattern of working — not a stricter MUST.
2. Keep the prompt lean. Cut sections that aren’t pulling their weight. Read transcripts, not just outputs.
3. Explain the why. LLMs are smart; theory of mind beats heavy-handed MUST. Yellow flag: if you find yourself writing ALWAYS or NEVER in all caps, reframe.
4. Look for repeated work across test cases. If all three test cases independently wrote create_docx.py, that’s a strong signal — write it once, put it in scripts/, tell the skill to use it.

Description Optimization (separate loop)

Triggering depends on the description field — it’s what Claude reads to decide whether to consult the skill. Run this after the skill is otherwise done:

1. Generate ~20 eval queries — mix of should-trigger and should-not-trigger. Save as JSON with {query, should_trigger} per item.
2. Review with user via assets/eval_review.html (template-driven, not custom HTML).
3. Run the optimization in the background: python -m scripts.run_loop --eval-set <path> --skill-path <path> --model <model-id> --max-iterations 5 --verbose. Splits 60/40 train/test, evaluates the current description 3× per query, calls Claude to propose improvements, re-evaluates on both splits. Returns best_description selected by test score (not train) to avoid overfitting.
4. Update SKILL.md frontmatter with best_description.

The skill is blunt about why this matters: Claude has a tendency to undertrigger skills — to not consult them when they would have helped. Compensate by writing descriptions that are a little “pushy.” Compare:

• ❌ “How to build a simple fast dashboard to display internal Anthropic data.”
• ✅ “How to build a simple fast dashboard to display internal Anthropic data. Make sure to use this skill whenever the user mentions dashboards, data visualization, internal metrics, or wants to display any kind of company data, even if they don’t explicitly ask for a ‘dashboard.’”

Advanced: blind comparison

agents/comparator.md describes a blind A/B for two skill versions when you need to know whether the new version actually wins. Independent agent judges without knowing which is which. Optional; most users don’t need it.

Gotchas

• Don’t sequence with-skill and baseline runs. Spawn them in the same turn — otherwise the wall-clock skews and the variance analysis loses meaning.
• feedback.json field names matter. The viewer depends on text / passed / evidence, not name / met / details. Don’t substitute.
• generate_review.py is the only sanctioned HTML path. Don’t write custom review UI — the skill explicitly says so, twice.
• Test selection beats volume. 20 well-chosen eval queries (8–10 should-trigger + 8–10 tricky should-not-trigger near-misses) outperform 60 vague ones. The skill warns: don’t make should-not-trigger queries obviously irrelevant (“write fibonacci” as a PDF-skill negative is too easy).
• Description optimization picks by test score, not train score. That’s the protection against overfitting to the queries you wrote. Don’t override.
• Claude undertriggers skills. The skill says this in two places. Push the description.
• Claude.ai specifics: no subagents → run test cases inline (one at a time), skip baselines, skip the browser viewer. Claude Code claude -p is required for the description optimization loop — skip on Claude.ai.
• Cowork specifics: no display → pass --static <output_path> to generate_review.py. Feedback downloads as a file, then you read it.
• Updating an existing skill: preserve the directory name and name frontmatter verbatim (research-helper stays research-helper, not research-helper-v2). Installed paths may be read-only — copy to /tmp/<skill>/, edit, package from the copy.
• run_loop.py description optimization uses claude -p via subprocess — needs Claude Code, doesn’t work on Claude.ai.
• present_files is optional. Skip packaging step if the tool isn’t available.

When this skill is not the right call

The skill is candid about its own scope: it’s for objectively verifiable outputs (file transforms, data extraction, code generation, fixed workflow steps). For subjective work (writing style, art), drop the evals/benchmarking machinery and just iterate conversationally — “vibe with me” mode is explicitly endorsed.

Install

# Claude Code (marketplace)
/plugin marketplace add anthropics/skills
/plugin install example-skills@anthropic-agent-skills   # installs skill-creator + the other example skills

For the description-optimization sub-loop you also need the claude CLI on PATH (it’s a claude -p subprocess call inside scripts/run_loop.py).

# Programmatic usage — the skill is just an instruction set; load its SKILL.md into context
# and route through Claude as you would any other skill.

Related

• Writing Skills (obra/superpowers) — same shelf, also a skill-authoring meta-skill; uses TDD (RED baseline / GREEN write / REFACTOR close loopholes) instead of eval/benchmarking
• Refactoring UI Audit — gold authored skill example on the same shelf, what “a real authored skill” looks like
• UI Skills catalog (Anthropic) — broader Anthropic skills surface
• Anthropic PDF (official skill) — sibling from the same Anthropic anthropics/skills repo (different plugin: document-skills, proprietary license)