← Blog

2026.11.01 · 16 min · tutorial · ai · agents · tooling

Give your agent skill its own scripts.

A walkthrough of an agent skill that ships a script: a release-notes skill whose Python collector turns git log into JSON, so the model only writes prose.

Most agent skills are just text: a SKILL.md that changes how the agent behaves, like grill-me or ponytail. That's enough when the job is judgement. It isn't enough when part of the job has one right answer. Ask an agent for release notes and it will run git log four different ways, scroll through diffs, and maybe miss the commit whose breaking change sits in the footer. Run it twice and you get two different histories.

The fix is to ship the deterministic half as code. In this walkthrough we'll build a release-notes skill with a script inside it. The script turns the git history into JSON, and the model writes from that JSON. By the end, this prompt:

terminal
claude -p "Write release notes for this repo." --allowedTools "Skill(release-notes)"

gives this, in six turns, for under $0.002 on Claude Haiku 5.5:

output
## 2.0.0 (2026-11-01)
 
### Breaking changes
- The `--port` flag is removed. Set the `PORT` environment variable instead. (7fee000)
- Node 22 or newer is now required. Upgrade Node before installing this release. (ac89a29)
 
### Features
- auth: sign in with a passkey (846113c)
 
### Fixes
- api: return 400 on an empty request body (c11d5ab)
 
### Other
Performance work caches the search tokenizer between queries (8fe2d62), adds a self-hosting guide (9203db6), bumps undici to 7.16.0 (7584b07), and updates the README screenshots (ec96b53).
 
Thanks to Asha Rao, Ben Okafor, and Chen Wei.
 
…

The same skill folder, unchanged, works in Pi. The code is in a2coder-tutorials/agent-skill-with-scripts.

Tested with

  • Claude Code 2.1.316 and Pi 1.4.0 (@earendil-works/pi-coding-agent)
  • Claude Haiku 5.5 (claude-haiku-5-5) for every agent run
  • Node 24.22, Python 3.11.2, git 2.39.5 on node:24-bookworm-slim

Prerequisites

  • Docker, or Python 3 and git if you only want the script
  • An Anthropic API key for the agent runs. All of the runs in this post together cost under two cents.
  • A rough idea of what a skill is. If "skill vs. mod vs. hook" is fuzzy: a skill is instructions (and files) the model loads when a task matches. A hook is code the harness runs at fixed points whether the model wants it or not. A mod packages hooks and UI as a plugin. Claude Code mods are middleware covers the other two.

1. Split the job before you write anything

A release-notes run has two halves:

Deterministic (script)Judgement (model)
Find the last tagPick the next version number
List commits since it, skip mergesRewrite subjects for users, not maintainers
Parse type(scope)!: subjectDecide what deserves a line and what goes in "Other"
Spot ! and BREAKING CHANGE: footersWrite the migration sentence
Count commits per authorWrite the thank-you line

Everything on the left has exactly one correct answer, and a model can get each of those wrong in a new way every run. So the left column becomes scripts/collect.py, and the right column stays in the prompt.

2. Write the script

The skill is a folder. The Agent Skills spec suggests scripts/ for code, references/ for docs the model reads on demand, and assets/ for templates:

release-notes/
release-notes/
├── SKILL.md
├── scripts/
│   └── collect.py
└── references/
    └── format.md

The script uses only the Python standard library. Any harness that can run a shell command can then run it, with nothing to pip install:

release-notes/scripts/collect.py
#!/usr/bin/env python3
"""Collect the commits since the last tag as JSON, grouped by Conventional Commit type.
 
The agent runs this instead of reading `git log` itself: the facts come out the
same every time, and only the prose is left to the model.
"""
import argparse
import json
import re
import subprocess
import sys
 
# type(scope)!: subject
HEADER = re.compile(r"^(?P<type>[a-z]+)(?:\((?P<scope>[^)]+)\))?(?P<bang>!)?: (?P<subject>.+)$")
GROUPS = ["feat", "fix", "perf", "refactor", "docs", "test", "build", "ci", "chore"]
FIELD, RECORD = "\x1f", "\x1e"
 
 
def git(repo, *args):
    result = subprocess.run(["git", "-C", repo, *args], capture_output=True, text=True)
    if result.returncode != 0:
        raise RuntimeError(result.stderr.strip() or f"git {' '.join(args)} failed")
    return result.stdout
 
 
def last_tag(repo, until):
    try:
        return git(repo, "describe", "--tags", "--abbrev=0", until).strip()
    except RuntimeError:
        return None  # no tags yet: take the whole history
 
 
def commits(repo, since, until):
    rev = f"{since}..{until}" if since else until
    fmt = FIELD.join(["%h", "%an", "%s", "%b"]) + RECORD
    out = git(repo, "log", "--no-merges", f"--format={fmt}", rev)
    for raw in filter(str.strip, out.split(RECORD)):
        sha, author, subject, body = raw.strip("\n").split(FIELD)
        yield sha, author, subject, body.strip()
 
 
def parse(sha, author, subject, body):
    m = HEADER.match(subject)
    kind = m["type"] if m and m["type"] in GROUPS else "other"
    note = re.search(r"^BREAKING[ -]CHANGE: (.+)$", body, re.M)
    breaking = bool(m and m["bang"]) or bool(note)
    scope = f"{m['scope']}: " if m and m["scope"] else ""
    text = m["subject"] if m else subject
    line = f"{sha} {scope}{text} ({author})"
    return kind, line, author, (f"{sha} {note[1] if note else text}" if breaking else None)
 
 
def main():
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("--repo", default=".", help="path to the git repository (default: .)")
    ap.add_argument("--since", help="start ref, exclusive (default: the latest tag)")
    ap.add_argument("--until", default="HEAD", help="end ref, inclusive (default: HEAD)")
    args = ap.parse_args()
 
    try:
        since = args.since or last_tag(args.repo, args.until)
        items = [parse(*c) for c in commits(args.repo, since, args.until)]
    except RuntimeError as e:
        print(f"collect: {e}", file=sys.stderr)
        return 2
 
    groups = {g: [] for g in GROUPS + ["other"]}
    authors = {}
    for kind, line, author, breaking in items:
        if not breaking:  # listed once, under "breaking", so the notes don't repeat it
            groups[kind].append(line)
        authors[author] = authors.get(author, 0) + 1
 
    json.dump(
        {
            "range": {"from": since, "to": args.until},
            "total": len(items),
            "breaking": [b for *_, b in items if b],
            "groups": {k: v for k, v in groups.items() if v},
            "authors": dict(sorted(authors.items(), key=lambda a: -a[1])),
        },
        sys.stdout,
        indent=2,
    )
    print()
    return 0
 
 
if __name__ == "__main__":
    sys.exit(main())

Three choices here matter because the reader is a model:

  • One line per commit. "846113c auth: sign in with a passkey (Asha Rao)" is far shorter than a five-field object, and the model can quote it back as it is.
  • Errors go to stderr with a non-zero exit. The skill tells the agent to stop on a non-zero exit. A script that prints an empty {} on failure would get release notes written about nothing.
  • --help works. Agents often run --help before guessing flags, and argparse gives you that for free.

3. Test it against a repo with known answers

Testing against your real repo means the expected output changes with every commit. The tutorial repo has fixture/make-fixture.sh instead. It builds a repo with one tag and eight commits at fixed dates, so the hashes never change:

fixture/make-fixture.sh (excerpt)
commit "Asha Rao"   "feat: first public release"
git tag v1.0.0
commit "Asha Rao"   "feat(auth): sign in with a passkey"
commit "Ben Okafor" "fix(api): return 400 on an empty request body"
commit "Asha Rao"   "perf(search): cache the tokenizer between queries"
commit "Ben Okafor" "feat!: require Node 22 or newer"
commit "Chen Wei"   "docs: add a self-hosting guide"
commit "Ben Okafor" "$(printf 'fix(config): read PORT from the environment\n\nBREAKING CHANGE: the --port flag is removed; set PORT instead.')"
commit "Chen Wei"   "chore(deps): bump undici to 7.16.0"
commit "Asha Rao"   "Update README screenshots"

The fixture includes both kinds of breaking change (! and a footer) and one commit that isn't conventional, since those are the cases a lazy parser gets wrong. Build the image and run the script against the fixture:

terminal
docker build -t skill-tutorial .
docker run --rm skill-tutorial sh -c \
  'fixture/make-fixture.sh ~/demo-app && cd ~/demo-app && python3 ~/tutorial/release-notes/scripts/collect.py'
output
{
  "range": {
    "from": "v1.0.0",
    "to": "HEAD"
  },
  "total": 8,
  "breaking": [
    "7fee000 the --port flag is removed; set PORT instead.",
    "ac89a29 require Node 22 or newer"
  ],
  "groups": {
    "feat": [
      "846113c auth: sign in with a passkey (Asha Rao)"
    ],
    "fix": [
      "c11d5ab api: return 400 on an empty request body (Ben Okafor)"
    ],
    "perf": [
      "8fe2d62 search: cache the tokenizer between queries (Asha Rao)"
    ],
    "docs": [
      "9203db6 add a self-hosting guide (Chen Wei)"
    ],
    "chore": [
      "7584b07 deps: bump undici to 7.16.0 (Chen Wei)"
    ],
    "other": [
      "ec96b53 Update README screenshots (Asha Rao)"
    ]
  },
  "authors": {
    "Asha Rao": 3,
    "Ben Okafor": 3,
    "Chen Wei": 2
  }
}

tests/test_collect.py turns that output into assertions, plus a check that a missing repo fails loudly:

terminal
docker run --rm skill-tutorial python3 -m unittest -v tests/test_collect.py
output
test_finds_both_kinds_of_breaking_change (tests.test_collect.CollectTest.test_finds_both_kinds_of_breaking_change) ... ok
test_groups_commits_since_the_last_tag (tests.test_collect.CollectTest.test_groups_commits_since_the_last_tag) ... ok
test_not_a_repo_fails_loudly (tests.test_collect.CollectTest.test_not_a_repo_fails_loudly) ... ok
test_since_overrides_the_tag (tests.test_collect.CollectTest.test_since_overrides_the_tag) ... ok
 
----------------------------------------------------------------------
Ran 4 tests in 0.189s
 
OK

The script is now finished, tested software. Everything after this point is about getting an agent to use it.

4. Write SKILL.md

release-notes/SKILL.md
---
name: release-notes
description: Draft release notes from the git history since the last tag. Use when the user asks for release notes, a changelog entry, or what changed since the last release.
allowed-tools: Bash(python3 ${CLAUDE_SKILL_DIR}/scripts/collect.py *) Read
---
 
# Release notes
 
Never read `git log` yourself. The script below collects the commits; your job is the writing.
 
1. From the root of the user's repository, run the collector that ships with this skill:
 
   ```bash
   python3 <this skill's directory>/scripts/collect.py
   ```
 
   It prints JSON: the tag range, breaking changes, commits grouped by type, and authors.
   Pass `--since <ref>` if the user names a different starting point. If it exits non-zero, show the user its error and stop.
 
2. Read `references/format.md` in this skill's directory and write the notes in that format.
 
3. Use only what the JSON contains. Do not invent changes, numbers or migration steps, and keep every commit's short hash.

Line by line:

  • name must match the folder name: lowercase letters, digits and hyphens, at most 64 characters.
  • description is the most important line in the file. At startup a harness loads only name and description for every installed skill, about 100 tokens each. The model decides whether to use your skill from that one line alone. So it says what the skill does and when to use it, in the words a user would type: "release notes", "changelog", "what changed".
  • allowed-tools is marked experimental in the spec, and each harness treats it differently. In Claude Code it pre-approves the script for the turn that invokes the skill, and ${CLAUDE_SKILL_DIR} expands to the skill's folder. The trailing * matches any arguments. Pi ignores the field. Step 7 shows the catch.
  • "This skill's directory" instead of a hard-coded path, because the folder lives somewhere different in every harness and every project. Both harnesses resolved it correctly. Claude Code prints the base directory when it loads the skill, and Pi's system prompt tells the model to resolve relative paths against the skill's folder.
  • "Never read git log yourself" isn't a style note. Without it, a model that can't run the script falls back to doing the job by hand, and the run that should have failed loudly succeeds quietly.

The result is three levels of loading. Each level is only paid for when it's needed:

name + descriptionevery sessionSKILL.md bodytask matchesscripts/collect.pyrun, not readformat.mdreferences/, on demand
FIG. 01 / WHAT LOADS WHEN

Note the third box. The script's source never enters the context. Only its output does. Claude Code ran collect.py without reading it, so the script's 90 lines cost no tokens.

5. Put the format in a reference file

The template the notes follow lives in references/format.md, not in SKILL.md. It's only needed at the writing step, and keeping it separate means you can change the house style without touching the instructions:

release-notes/references/format.md
# Release notes format
 
```markdown
## <next version> (<today, YYYY-MM-DD>)
 
### Breaking changes
- <what changed and what the reader must do> (<sha>)
 
### Features
- <scope>: <subject, rewritten for a user> (<sha>)
 
### Fixes
- …
 
### Other
<one line summing up perf, docs, chore and other commits>
 
Thanks to <authors, most commits first>.
```
 
- Next version: bump major if there are breaking changes, minor if there are features, otherwise patch.
- Leave out a section that would be empty.
- Write for someone who uses the project, not someone who reads its code.

6. Install it where the harnesses look

The format is shared. The folder each harness reads it from isn't. .agents/skills/ is the closest thing to a standard location, but Claude Code doesn't read it:

HarnessProject folderUser folderRan here
Claude Code.claude/skills/~/.claude/skills/yes
Pi.agents/skills/ (needs project trust)~/.agents/skills/yes
Codex CLI.agents/skills/~/.agents/skills/no, from docs
OpenCode.opencode/, .claude/ or .agents/skills/~/.config/opencode/, ~/.claude/ or ~/.agents/skills/no, from docs
Gemini CLI.gemini/skills/ or .agents/skills/~/.gemini/skills/no, from docs
Cursor.agents/, .cursor/ or .claude/skills/same under ~no, from docs
GitHub Copilot.github/, .claude/ or .agents/skills/~/.copilot/skills/no, from docs

So keep one copy in .agents/skills/ and give Claude Code a symlink:

terminal
mkdir -p .agents/skills .claude
cp -r release-notes .agents/skills/
ln -s ../.agents/skills .claude/skills   # Claude Code reads .claude/skills only

try.sh in the repo does exactly this on a fresh fixture, then asks the agent for release notes.

7. Run it in Claude Code

try.sh (excerpt)
claude -p "Write release notes for this repo." --model claude-haiku-5-5 --allowedTools "Skill(release-notes)"
terminal
docker run --rm -e ANTHROPIC_API_KEY skill-tutorial ./try.sh claude

That printed the notes at the top of this post. To see what the agent did, add --output-format stream-json --verbose and pull out the tool calls:

terminal
claude -p "Write release notes for this repo." --model claude-haiku-5-5 \
  --allowedTools "Skill(release-notes)" --output-format stream-json --verbose \
  | jq -r 'select(.type=="assistant") | .message.content[]? | select(.type=="tool_use") | "\(.name) \(.input|tojson)"'
output
Skill {"skill":"release-notes"}
Bash {"command":"python3 /home/node/demo-app/.claude/skills/release-notes/scripts/collect.py; echo \"EXIT=$?\"", …}
Bash {"command":"python3 /home/node/demo-app/.claude/skills/release-notes/scripts/collect.py", …}
Read {"file_path":"/home/node/demo-app/.claude/skills/release-notes/references/format.md"}

The agent chose the skill from the description alone; the prompt never named it. Then it ran the script and read the format file. It never touched git log. The second line is a denial: the model appended ; echo "EXIT=$?", which no longer matches the allowed-tools rule, so it was refused. The model retried with the plain command. That's the permission rule working as intended. The rule approves one command, not "anything that starts with python3".

Why --allowedTools "Skill(release-notes)" and not a Bash rule? Because allowed-tools in SKILL.md grants permissions, invoking the skill itself needs approval. In an interactive session that means one prompt. In -p mode nobody is there to answer, so without the rule the Skill call is denied (see "What broke"). Approving the skill once is the right trade. You review one skill rather than writing a Bash pattern for every script it ships.

8. Run the same folder in Pi

try.sh (excerpt)
pi -p "Write release notes for this repo." --model "anthropic/claude-haiku-5-5" --approve --no-session
terminal
docker run --rm -e ANTHROPIC_API_KEY skill-tutorial ./try.sh pi
output
## 2.0.0 (2026-11-01)
 
### Breaking changes
- The `--port` flag is removed. Set the `PORT` environment variable instead. (7fee000)
- Node 22 or newer is now required. (ac89a29)
 
### Features
- auth: sign in with a passkey. (846113c)
 
### Fixes
- api: return 400 on an empty request body. (c11d5ab)
 
### Other
Performance, docs, dependency and maintenance work: the search tokenizer is now cached between queries (8fe2d62), a self-hosting guide was added (9203db6), undici was bumped to 7.16.0 (7584b07), and the README screenshots were updated (ec96b53).
 
Thanks to Asha Rao, Ben Okafor and Chen Wei.
 
…

That's the same facts in slightly different words, for about $0.001. Pi doesn't prompt before tool calls, so it needs no permission flag, but it does need --approve. Project-local skills sit behind project trust, and in -p mode there's no trust prompt to answer. With --mode json you can see what Pi put in its system prompt:

pi system prompt, skills section
<available_skills>
  <skill>
    <name>release-notes</name>
    <description>Draft release notes from the git history since the last tag. Use when the user asks for release notes, a changelog entry, or what changed since the last release.</description>
    <location>/home/node/demo-app/.agents/skills/release-notes/SKILL.md</location>
  </skill>
</available_skills>

The startup cost really is just name, description and location. Pi's first tool call was read on that SKILL.md.

9. When the notes are wrong, fix the script

My first version of collect.py put a breaking commit both under breaking and under its type group. Both models dutifully wrote it twice. Claude Code listed "Require Node 22 or newer" as a Feature and a Breaking change. Pi did the same, labelled it with the wrong scope, and added a detail that wasn't in the data ("returns 400 instead of an error").

The tempting fix is another line in SKILL.md: "don't list breaking changes twice". Don't. That's a rule with one right answer, so it belongs in the deterministic half:

release-notes/scripts/collect.py
    for kind, line, author, breaking in items:
        if not breaking:  # listed once, under "breaking", so the notes don't repeat it
            groups[kind].append(line)

After that change, plus one more test assertion, both harnesses produced the clean notes shown above. Every time a model gets something wrong, ask which column of the table in step 1 the mistake belongs to. If it's on the left, fix the script, and the fix holds in every harness and every model.

Full code

Everything is in a2coder-tutorials/agent-skill-with-scripts under MIT. The image that ran every output in this post:

Dockerfile
FROM node:24-bookworm-slim
 
RUN apt-get update \
 && apt-get install -y --no-install-recommends git python3 ca-certificates \
 && rm -rf /var/lib/apt/lists/*
 
# Two harnesses that both load SKILL.md: Claude Code and Pi.
RUN npm install -g @anthropic-ai/claude-code@2.1.316 \
 && npm install -g --ignore-scripts @earendil-works/pi-coding-agent@1.4.0
 
# Agents refuse to skip permission prompts as root, and you shouldn't want them to.
USER node
WORKDIR /home/node/tutorial
COPY --chown=node:node . .
RUN git config --global user.name tutorial && git config --global user.email tutorial@example.com
terminal
git clone https://github.com/a2coder-tutorials/agent-skill-with-scripts && cd agent-skill-with-scripts
docker build -t skill-tutorial .
docker run --rm -e ANTHROPIC_API_KEY skill-tutorial ./try.sh claude

The repo also opens as a dev container in VS Code or Codespaces.

What broke along the way

The skill was in .agents/skills/ only, and Claude Code ignored it silently. There was no error. The agent never saw the skill, ran git log three different ways and wrote perfectly reasonable notes. That's the worst kind of failure, because it looks like success. Check the transcript for a Skill call, not just the output. The fix is the .claude/skills symlink from step 6.

No permission rule meant the script was denied. Without --allowedTools or allowed-tools, -p mode denies the Bash call. Because the skill says never to read git log directly, the agent stopped and said so instead of improvising:

output
I couldn't generate the release notes yet. The collector script needs Bash permission, and the call came back as "requires approval" without running.
 
I haven't worked around it. The skill says not to read `git log` directly, so I didn't try that.

That's the "Never read git log yourself" line earning its place.

Adding allowed-tools made the skill itself fail to load. The tool result was just Execute skill: release-notes with is_error: true, and the agent fell back to git commands again. permission_denials in the final JSON gave the real reason: {"tool_name":"Skill"}. A skill that grants tools needs approval to run, which is what --allowedTools "Skill(release-notes)" gives it.

Compound commands don't match a permission rule. python3 …/collect.py; echo "EXIT=$?" was denied even with the rule in place. The model recovered on its own, but if your skill's instructions add pipes or && to the command, write the rule for the exact command you tell it to run.

Pi without --approve had no skills section at all. The system prompt's sections were cwd, docs, preamble, rules, tools. With --approve there was a sixth: skills. Without it, the agent only found the skill by accident, after three rounds of git and ls commands. Alternatively, install the skill in ~/.agents/skills/, which isn't gated by project trust.

Duplicate lines in the notes came from the script, not the model. Step 9 has the fix.