Try It Out

What problem it addresses

When building agent skills, the hard part is usually not producing a first SKILL.md, but knowing whether it actually helps: Does it trigger reliably? Does it produce more stable output than a baseline? Are token usage and runtime acceptable? After several iterations, which assertions are meaningful and which just re-check the same behavior? Try It Out turns this ad hoc debugging loop into a repeatable workflow.

How it works

It starts by clarifying intent: what the skill should do, when it should trigger, what output format is expected, and whether test cases are useful. If the current conversation already contains a workflow, it can extract the relevant tool calls, step order, input/output formats, and user corrections before asking for missing details.

The main loop then moves from draft to evaluation:

• Write a SKILL.md draft, putting trigger conditions in description and using progressive disclosure or domain-specific reference files when needed.
• Create 2 to 3 realistic prompts that a real user might say, stored in evals/evals.json.
• Launch with-skill and baseline runs in the same turn so they finish around the same time.
• Draft objectively verifiable assertions while runs are in progress, then update eval_metadata.json.
• Capture timing data, grade assertions, aggregate benchmark.json and benchmark.md, and open a review viewer for both qualitative outputs and quantitative metrics.

It also performs an analyst pass to surface patterns hidden by aggregate stats, such as assertions that always pass, high-variance evaluations, and token/runtime tradeoffs. This helps decide whether the next iteration should focus on the description, the skill body, or a broader test set.

Boundaries and notes

The workflow is most useful for skills with verifiable outputs or stable procedural steps, such as file transforms, data extraction, code generation, or fixed workflows. For subjective outcomes like writing style or design quality, the source material favors qualitative review over forced quantitative assertions. Expect meaningful runtime overhead from multiple subagents, grading steps, and result files. In headless environments, a static review file can replace an interactive viewer.