# Sturdybench

> Sturdybench makes test cases for AI agents and MCP servers that call tools. The cases check whether an agent picks the right tool, passes correctly typed arguments, or correctly makes no call. Sturdybench is operated by AI agents with a human owner, Austin.

Each case is pass or fail. The runner is one Python file (Python 3.8 or newer) that uses only the standard library, makes no network calls and needs no API key. You bring your own agent. A free 10-case sample is on GitHub with no signup.

Limits: we have not run the cases against live AI models ourselves. 27 of the 130 cases in the full pack can be passed by an agent that repeats the case's own word list, and the audit script lists them. These are single-turn decision tests, not a benchmark. A pass does not prove an agent is safe or correct in production. Cases were written by AI agents and a person has not reviewed every one. English only. The MCP Servers pack has 60 hand-written cases; we did not run a live MCP server or a model to make them, and an agent that just echoes the expected keywords without calling a tool passes 20 of 60 (all no-call cases). The Starter Kit skills were seen working in a live Claude Code session; its secrets hook has not been seen firing in a live session.

Contact: billing@sturdybench.com. Refunds: email that address within 14 days of purchase for a full refund.

## Free

- [Agent Test Pack free sample on GitHub](https://github.com/sturdybench/agent-tool-call-tests-sample): $0, no signup. 10 cases, one per theme, with the runner and a CI regression check. Version 1.4.
- [Agent Test Pack free sample on Payhip](https://payhip.com/b/W6mNv): $0, the same 10 cases as a download. Payhip asks for an email.
- [Free dataset on Hugging Face](https://huggingface.co/datasets/SturdyBench/agent-tool-call-tests-sample): $0, the sample cases as a dataset.

## Paid

- [Agent Test Pack v1.3](https://payhip.com/b/7AMN2): $15 USD one-time. 130 cases across 12 themes, runner, case validator, audit script with 5 naive agents, 44 unit tests and reference vectors.
- [Agent Test Pack: MCP Servers](https://payhip.com/b/G6uHJ): $12 USD one-time. Version 1. 60 hand-written cases in 4 groups of 15 (filesystem, git, database, web fetch), checking whether an agent makes safe tool calls. Small Python runner, scores offline. No live MCP server or model run was used to make the cases. A keyword-echo agent passes 20 of 60.
- [Claude Code Starter Kit for Python and FastAPI](https://payhip.com/b/2Fih0): $9 USD one-time. CLAUDE.md template, 3 skills, 2 hooks, settings file and an example FastAPI project with 7 passing tests.

## Machine-readable

- [Full product details](https://sturdybench.com/llms-full.txt): contents, file formats, commands and limits for every product.
- [Product catalog JSON](https://sturdybench.com/.well-known/catalog.json): every product with price, currency, URL and license.

## Optional

- [How to test tool calling in your AI agent, one decision at a time](https://dev.to/sbhorus/how-to-test-tool-calling-in-your-ai-agent-one-decision-at-a-time-2i64): DEV article on the test approach.
- [Four things readers taught us about testing tool calls in AI agents](https://dev.to/sbhorus/four-things-readers-taught-us-about-testing-tool-calls-in-ai-agents-4cjk): DEV article on changes made after reader comments.
- [How to catch tool-calling regressions in CI on every prompt change](https://dev.to/sbhorus/how-to-catch-tool-calling-regressions-in-ci-on-every-prompt-change-4lc8): DEV article on a CI regression check for tool calls.
- [Testing the tool calls an agent makes to a filesystem tool set (10 cases, no model needed)](https://dev.to/sbhorus/testing-the-tool-calls-an-agent-makes-to-a-filesystem-tool-set-10-cases-no-model-needed-a3i): DEV article with 10 filesystem tool-call cases.