mcpmark
Benchmark framework for evaluating MCP server effectiveness. Measures whether MCP tools improve AI agent accuracy, and at what token cost.
Currently includes a benchmark suite for TorchTalk.
Quick Start```bash
pip install -e .
export PYTORCH_SOURCE=/path/to/pytorch
python3 runners/claude_cli.py --condition both --max-turns 20
python3 compare.py results
## What It Measures
| Metric | Description |
|--------|-------------|
| **Pass/fail accuracy** | Per-task assertions + LLM judge second pass |
| **Token usage** | Input + output tokens per task |
| **File path verification** | % of mentioned file paths that exist in source |
## Scoring
1. **Per-task assertions** — `must_contain` (all required) + `must_contain_any` (at least one)
2. **LLM judge** — Haiku second-pass on failures, reference-based binary eval
3. **File path verification** — checks paths against real source tree
## Project Structure```
mcpmark/
├── dataset.json # Task bank (queries + assertions)
├── runners/
│ ├── claude_cli.py # Claude Max runner
│ ├── torchtalk_mcp.json # MCP config (WITH tools)
│ └── empty_mcp.json # Empty config (WITHOUT tools)
├── tasks/
│ └── torchtalk_bench.py # Inspect AI runner (API-based)
├── compare.py # Results comparison
├── run_benchmark.sh # Quick-run script
└── results/ # Benchmark outputs (gitignored)
