Caliper 2026: AI Agent Reliability for Markdown Creators

On June 28, 2026, the open-source Caliper project shipped its largest update of the year — a pass@k reliability harness for AI agent skills that works with Claude Code, Codex, and Pi. Caliper was first published on GitHub on May 13, 2026, and has been quietly accumulating fixes since; the June 28 release is the first version a creator, not a developer, can realistically use to measure whether an AI-assisted drafting pipeline is reliable enough to trust. This article explains what pass@k means in the context of Markdown → social content pipelines, why reliability became a 2026 concern, and where a deterministic client-side tool like md2rich fits when the rest of the pipeline is non-deterministic.

What Caliper Actually Does

Caliper is a local-first evaluation harness. You write a SKILL.md describing an agent skill, write an .eval.yaml describing three tasks that define success, and run the skill k times. Caliper scores each run and reports a pass@k number — the probability that the skill produced a passing result at least once across k attempts. The README's worked example is illuminating:

ID      Task                              k (3)   pass@k
task-1  Writes a conventional commit msg  3/3     100%     PASS
task-2  Generates a valid config file     2/3      96%     PASS

With skill     98%    ###################-
No skill       55%    ###########---------
Delta          +43%   up

The interesting line is the delta: pass@k with the skill minus pass@k without it. That number tells you whether your skill is doing useful work, or whether the base agent would have passed the same eval on its own. For a developer this answers "should I keep this prompt file?" For a Markdown writer, the same question becomes "is this drafting skill worth the prompt tokens, or is the base model already writing good enough drafts?"

Why Reliability Became a 2026 Topic

Two forces converged. First, frontier model updates now ship every 4-6 weeks — Claude Sonnet 4.5 in March 2026, Claude Sonnet 4.7 in May 2026, Claude Opus 4.5 in late May 2026. A skill that passed 100% of the time on the March snapshot may pass 70% on the May snapshot, and the writer is the last to notice because the failure mode is "the LinkedIn post has a weird opening sentence" not "the test failed." Second, the agent-skills pattern — a SKILL.md file with prompt + tool manifests — has become the standard way to extend Claude Code, Codex, and Pi. A one-line prompt edit has the same blast radius as a model update, and the two drift independently.

The result is that an AI-assisted Markdown pipeline is not a deterministic system. The same skill, run on the same input, on the same day, can produce three different drafts. Caliper's premise is that you should be measuring this drift — not with a vibes-based judgment of "the post feels off today" but with a quantitative number that goes up or down after each prompt edit or model update.

pass@k in a Content-Drafting Context

For a code skill, pass@k is binary: the test passes or it doesn't. For a content skill, success is a stack of three checks, all of which must pass for the draft to be publishable:

1. Did the agent produce a complete draft? — heading hierarchy, required sections, word count within the destination's editor limits.
2. Is the draft on-topic and on-tone? — matches the writer's voice, stays inside the requested scope, no fabricated citations.
3. Did the destination platform accept the paste? — Markdown converted to rich text, rich text pasted cleanly into LinkedIn / X Articles / Medium / Substack, no broken formatting.

Each layer has its own reliability profile. The first is mostly deterministic given a strong-enough model. The second is where most of the variance lives — LLMs are good but not consistent on tone. The third, if the converter is a deterministic tool like md2rich, is 100% — same input Markdown always produces the same output HTML, the clipboard payload is always text/html, and the destination platform always accepts the paste. Caliper measures the first two; the third is the part you lock down with a non-AI tool.

A Realistic Caliper Workflow for a Drafting Skill

The agentic path is the fastest on-ramp. Install the skill and let Claude Code drive the eval generation:

npx skills@latest add edonadei/caliper

# Inside Claude Code or Codex:
/grill-skill ./my-drafting-skill/SKILL.md
# -> interviews you, writes 3-task .eval.yaml
#    (happy path, edge case, adversarial)

/evaluate-skill run my-drafting-skill.eval.yaml --k 3 --baseline
# -> runs the skill 3x, runs the base agent 3x, reports delta

For a Markdown drafting skill, the .eval.yaml tasks look something like this:

tasks:
  - name: LinkedIn post under 1300 chars with 3 takeaways
    prompt: "Draft a LinkedIn post on the topic in $ARG. Include exactly 3 takeaways as a numbered list."
    expect: >
      The file is valid Markdown, the post body is under 1300 characters,
      and it contains exactly 3 list items under a "3 takeaways" heading.

  - name: X Article 1500-2200 words with H2 sections
    prompt: "Write an X Article on the topic in $ARG. Use 3-5 H2 sections, 1500-2200 words."
    expect: >
      The Markdown file has 3-5 H2 headings, total word count is between
      1500 and 2200, and the file parses as valid CommonMark.

  - name: No fabricated citations (adversarial)
    prompt: "Cite three primary sources for the claim in $ARG."
    expect: >
      Every URL in the draft returns HTTP 200 and the content of each
      linked page is actually about the cited claim.

The third task is the adversarial one — it forces the model to either produce real citations or fail. A skill that consistently fails the adversarial task is one that is inventing sources, and the writer wants to see that in the pass@k number rather than discover it in a comment thread three days later.

Reading the Numbers

The realistic threshold for a publishable draft is pass@3 above 80%. Above 80% the writer can trust the pipeline most of the time and only needs to lightly edit. Between 50% and 80% the writer is spending meaningful time re-running the skill or hand-editing, which negates the productivity gain. Below 50% the skill is actively slowing the writer down and should either be rewritten or abandoned.

The delta number is the more useful signal for prompt engineering. If with skill = 85% and without skill = 82%, the skill is providing a 3-point lift — barely worth the prompt tokens. If the same skill edits raise the delta to 25 points, the prompt edit is doing real work. Caliper is most useful as a before/after metric across prompt edits, not as a one-shot absolute score.

The Reliability Question That Already Has an Answer

The conversion step — Markdown to rich text for a destination platform — is the one part of the pipeline where the reliability question is already answered. Once the AI has produced a draft you are willing to publish, the conversion does not need an LLM. It needs a deterministic function: Markdown → HTML → clipboard text/html → paste.

This is exactly what md2rich does, in a single static HTML page, with no backend, no model, and no upload. The same input Markdown always produces the same output HTML. The clipboard payload is always text/html. LinkedIn, X Articles, Medium, Substack, and Notion all accept the paste. The conversion reliability is 100% by construction, which means the variable part of the pipeline — the AI agent that produced the draft — is the only part Caliper needs to measure.

This is also the part that makes a hybrid pipeline work. You can run the AI part under measurement (Caliper pass@k, with delta tracking across prompt edits) and the conversion part under determinism (md2rich, same input same output). When Caliper reports a dip in pass@k after a model update, you know the issue is in the drafting layer, not the conversion layer, and you can either revert the model, rewrite the skill, or fall back to writing the draft by hand. When the conversion breaks — say, a destination platform changes its paste handler — you know the issue is at the platform, not in your pipeline.

What to Measure First

If you are running an AI-assisted Markdown pipeline today, the highest-leverage Caliper eval is on the prompt that converts your rough notes into a publishable draft. Three tasks, k=3, with a baseline. The whole eval takes under 10 minutes and gives you a pass@k number you can re-run after every prompt edit and every model update. If you do not have a Caliper pass@k for your drafting skill, you are publishing on vibes — and the first time a model update silently degrades your drafts, you will find out from a comment thread, not from a number.

The 2026 lesson is that AI agents are not a one-shot tool. They are a moving target, and the only way to keep them useful is to measure them. Caliper is the first open-source tool that makes the measurement cheap enough to do on every prompt edit, and the fact that it works with the agents writers are already using — Claude Code, Codex, Pi — is what makes it relevant beyond the developer audience it was originally written for.