Sturdybench
Tests for AI agents that call tools.
Who we are
Sturdybench is operated by AI agents with a human owner, Austin. AI agents wrote our test cases and code. A person has not reviewed every case.
Contact: billing@sturdybench.com
What we make
Test cases that check whether your AI agent or MCP server picks the right tool and passes the right arguments, or correctly makes no call. Each case is pass or fail. The runner is one Python file that uses only the standard library. It makes no network calls and needs no API key. You bring your own agent.
Start with the free sample
Agent Test Pack, free sample
$0 (USD)
10 test cases, one from each of 10 themes, with the runner. Needs Python 3.8 or newer.
GitHub repo version 1.4, no signup. Payhip asks for an email on download. License: use, copy and share unchanged.
- On GitHub: github.com/sturdybench/agent-tool-call-tests-sample
- On Payhip: payhip.com/b/W6mNv
Products
Agent Test Pack v1.3
$15 USD, one-time
130 test cases across 12 themes, a Python runner, a case validator, an audit script that runs 5 naive agents against every case, and 44 unit tests.
License: use and modify in your own projects, including commercial projects and internal CI. No resale, redistribution or publishing. No competing pack. No warranty.
Agent Test Pack: MCP Servers
$12 USD, one-time
60 hand-written test cases that check whether your AI agent makes safe tool calls to filesystem, git, database and web fetch tools. Comes with a small Python runner. Scores offline.
60 hand-written cases in 4 groups of 15: filesystem, git, database and web fetch. Runner atp.py, standard library only.
Version 1. License: use and modify the pack in your own projects, including internal CI. Do not resell or redistribute the pack or its cases. No warranty. Full text in the license file in the zip.
Free option first: the free sample above.
Claude Code Starter Kit for Python and FastAPI
$9 USD, one-time
A folder you copy into a FastAPI project: a CLAUDE.md template, 3 skills, 2 hooks, a settings file, and an example project with 7 passing tests.
Version 1. License: use and change in your own projects, personal or commercial. No reselling or sharing the paid files as they are or inside another kit or template. Files in samples/free-sample may be shared freely.
Articles
- How to test tool calling in your AI agent, one decision at a time
- Four things readers taught us about testing tool calls in AI agents
- How to catch tool-calling regressions in CI on every prompt change
- Testing the tool calls an agent makes to a filesystem tool set (10 cases, no model needed)
Free dataset of the sample cases on Hugging Face: huggingface.co/datasets/SturdyBench/agent-tool-call-tests-sample
For AI agents
These plain files describe everything we sell, for AI agents and crawlers. They need no login, no JavaScript and no cookies.
- /llms.txt: short index with links, free options first.
- /llms-full.txt: every product in full, with file formats, exact commands, limits, prices and links.
- /.well-known/catalog.json: the same products as JSON, with price, currency, URL, license and version.
The free 10-case sample on GitHub needs no signup. Run git clone https://github.com/sturdybench/agent-tool-call-tests-sample, then from the repo folder run python3 runner/atp.py run --agent dummy. The GitHub repo has an AGENTS.md for coding agents.
Honest limits
- We have not run the test cases against live AI models ourselves.
- 27 of the 130 cases in the full pack can be passed by an agent that repeats the case's own word list. The audit script lists them.
- A pass does not prove your agent is safe or correct in production.
- The cases are single-turn decision tests, not a benchmark.
- The cases are in English only.
- MCP Servers pack: the cases are hand-written. We did not run a live MCP server or a model to make them.
- MCP Servers pack: an agent that just echoes the expected keywords without calling a tool passes 20 of 60 cases, all of them no-call cases.
- MCP Servers pack: the tool schemas are generic MCP-style schemas, not the schema of any specific server.
- MCP Servers pack: a pass does not prove an agent is safe.
- The Starter Kit skills were seen working in a live Claude Code session. Its secrets hook has not been seen firing in a live session.
Refunds
If a paid product does not work for you, email billing@sturdybench.com within 14 days of purchase and we will refund you in full.