coder_eval

MCP开发工具

提供方 UiPath · 包名 coder_eval

开发工具 stdio · 本地进程 未标注 社区 / 第三方 Claude DesktopCursorClineClaude Code
查看接入方式

Test that your Claude Code skills, MCP servers, and CLIs actually work when an agent uses them — sandboxed YAML suites, activation checks, A/B experiments, CI gates.

一句话选型结论

按资料字段整理

coder_eval 属于开发工具类 MCP 服务器,stdio · 本地进程,鉴权方式未标注,可直接接入 Claude Desktop、Cursor、Cline 等,有可直接执行的安装命令。接入前先确认权限范围;涉及写入、删除或付款的操作,建议先在测试环境验证。

分类开发工具 传输方式stdio · 本地进程 鉴权未标注 部署方式本地部署
GitHub 星标 116 仓库热度,无公开仓库时显示未标注
工具数 未标注 暴露给模型的工具数量
资源 / 提示词 0 / 0 资源与提示词模板数量
维护方 社区 / 第三方 核验等级 仅资料核对

工具与能力

开发工具
数量来自来源目录的标注,具体工具清单以官方文档为准
工具数量未标注暴露给模型可调用的工具个数
资源数量0可读取的资源条目
提示词数量0内置提示词模板数量

能力标签

agent-evaluation agent-skills agent-testing anthropic claude

支持客户端

适用场景

agent-evaluationagent-skillsagent-testing

安装与配置

stdio · 本地进程 未标注
客户端在本机拉起子进程,数据不出本机
pip install coder_eval

这条命令只是入口,具体启动参数请按官方文档补齐,不要直接照搬到别的服务器。

  1. 1
    准备运行环境

    stdio 服务在本地起进程,需要先装好 Cross-platform,再按安装命令拉取依赖。

  2. 2
    安装服务

    复制上面的安装命令执行;命令里带占位符的位置,换成自己的密钥或路径。

  3. 3
    写入客户端配置

    在客户端的 MCP 配置里填好启动命令和参数,保存后完全退出客户端再打开,让新配置生效。

  4. 4
    验证是否可用

    连接成功后随便调一个只读工具,确认能返回结果,再接入正式流程。

典型使用场景

3 类
按分类与标签整理的常见用法,具体以官方文档为准
代码与仓库操作

在对话里直接查代码、提 issue、走 PR,省掉来回切工具。

本地命令与脚本

把构建、测试、格式化这类重复命令交给模型执行。

接口联调与测试

让模型调你的接口、跑一遍用例,快速定位问题。

选择建议

选之前先看这几条
按条目字段给出的判断,不替你决定用不用

值得选的地方

  • 有现成的安装命令,不用自己找入口,照着跑就能起服务。
  • 已适配 Claude Desktop、Cursor、Cline,换客户端不用重新研究配置。

先想清楚的地方

  • stdio 方式要求本地装好运行时(如 Node.js),换机器要重新装一遍。
  • 资料没标注工具数量,实际能力要装好后确认。
  • 版本更新与行为细节以官方仓库为准。

更适合

  • 需要在开发工具场景里让模型直接调用外部能力的人。
  • 已经在用 Claude Desktop、Cursor 的用户,接入路径最短。
  • 想先快速试一下、能接受在本地或自有环境跑服务的人。

不太适合

  • 完全不想在本地装环境的人。
  • 需要厂商级 SLA 和技术支持的生产环境。

同类对比

6 个同类
同分类里挑最接近的几条,字段以各自详情页为准
MCP 服务器工具数传输方式部署方式安装命令最后核验
coder_eval当前 未标注 stdio · 本地进程 本地部署 有 2026-08-23
Git 838e2ec1 12 stdio · 本地进程 本地部署 有 2026-05-17
Markitdown MCP 2c653a5f 1 stdio · 本地进程 本地部署 有 2026-05-17
JavaGuide 未标注 stdio · 本地进程 本地部署 有 2026-04-11
awesome-claude-skills 未标注 stdio · 本地进程 本地部署 有 2026-04-26
Context7 831acf40 未标注 stdio · 本地进程 本地部署 有 2026-05-17

服务器档案

开发工具
字段来自项目仓库与公开索引,中文说明由本站标注
提供方UiPath上游标注的维护方 UiPath
分类开发工具本站按上游分类展示
连接方式stdio · 本地进程客户端在本机拉起子进程,数据不出本机
鉴权方式未标注远程服务建议用最小权限凭据
维护方社区 / 第三方核验等级 仅资料核对
包名 / 运行时coder_eval · Cross-platform本地安装时用到的包标识与运行时
部署类型本地部署上游给出的部署形态
最近核验2026-08-23上游最后核验时间

详细介绍

16.2 KB
来自项目仓库与上游目录的原文,未做删改

Coder Eval — evaluate and benchmark AI coding agents and their skills

Playwright for coding agents — one declarative test file, any agent
runtime, a real sandbox, and a pass/fail gate in CI.

Coder Eval (pip install coder-eval / uv tool install coder-eval) is an
open-source, agent-agnostic framework for evaluating and benchmarking AI coding
agents and their skills
— built for benchmark authors, CLI builders, and skill
builders — with sandboxing, reproducibility, and data-driven analysis. It runs a real
agent — Claude Code, OpenAI Codex, Google Antigravity (Gemini),
OpenCode, or Pi — in a sandbox against declarative YAML tasks, then scores the files and
commands it actually produced. Changing harness is one field (agent.type); the
tasks, criteria, scoring, telemetry, and reports stay the same.

Reach for it when you want to benchmark agents on your own domain tasks,
test whether a skill triggers in the agent you ship for, A/B-test Claude Code
vs. Codex vs. Gemini vs. OpenCode vs. Pi
(or model vs. model, prompt vs. prompt), or
gate CI on coding-agent quality. It is not a fixed leaderboard: unlike
SWE-bench or SkillsBench, which rank models on a shared task set, you bring the tasks
and you bring the scoring — weighted 0.0–1.0 criteria, a skill_triggered activation
check, an A/B experiment layer, and per-tool cost telemetry, over whatever work you
care about. See How it compares.
📚 Full docs: coder-eval.com/docs.

  • Declarative YAML tasks with pinned dependencies and clear success criteria
  • Sandboxed execution in isolated environments with resource limits
  • Weighted, continuous scoring (0.0–1.0) with fractional credit and thresholds
  • Many criterion types — from file checks to code similarity and LLM-graded rubrics
  • Agent-agnostic by design — Claude Code, OpenAI Codex, Antigravity (Gemini), OpenCode, and Pi today; add your own harness through the plugin SPI
  • Experiment layer — A/B agent configs (models, tools, prompts) side-by-side
  • Full telemetry — every tool call, token counts, and cost, with real-time streaming

Watch the intro

🔊 Turn the sound on — GitHub's inline player always starts muted.

▶ Also on YouTube: Coder Eval: UiPath open-source framework to test AI Coding Agents
— what the framework does, and how a run works end to end.

What you can do with it

  • Benchmark coding agents — score an agent across a suite of tasks with weighted scoring and pass/fail thresholds
  • Compare models & configs — A/B-test Claude vs. Codex vs. Gemini vs. OpenCode vs. Pi, model vs. model, tool-on vs. tool-off, prompt vs. prompt
  • Evaluate skills — verify an agent actually engages a target skill (skill_triggered) and score skill-driven suites (SkillsBench-style), on whichever harness your users run
  • Keep skills up to date in CI — re-validate your skills on every change or on a schedule; catch silent regressions when models, prompts, or the skills themselves drift
  • Gate CI on agent quality — run the suite in GitHub Actions and fail the build on regressions
  • Bring your own dataset — fan one task out over many rows for larger benchmark suites

> Keeping skills fresh? Run Coder Eval as a scheduled GitHub Actions job so your
> skills are continuously re-evaluated against the latest model — a skill that quietly
> stops triggering surfaces as a failing criterion before your users hit it. See
> Tutorial 02 — Running Coder Eval in CI.

Quick Start

Prerequisites: Python 3.13+, uv 0.8+, and the
runtime of at least one coding agent
— plus that agent's own model credentials.
Pick the agent you want to evaluate; two of the four ship with a Coder Eval extra,
the other two are separate CLIs you install yourself:

Agent agent.type Runtime Guide
Claude Code (default) claude-code brew install claude — separate CLI Claude Code
OpenAI Codex codex uv sync --extra codex — the extra ships the Codex SDK + CLI Codex
Google Antigravity (Gemini) antigravity uv sync --extra antigravity — the extra ships the harness binary Antigravity
OpenCode (open-weight models) opencode npm install -g opencode-ai — separate CLI OpenCode

The examples below use the default claude-code agent. Developed on macOS; CI runs on
Linux.```bash
git clone https://github.com/UiPath/coder_eval.git
cd coder_eval

uv sync # install the framework (add --extra codex /
# --extra antigravity for those runtimes)
cp .env.example .env # then set ANTHROPIC_API_KEY — or skip that: an
# existing Claude Code login (claude login) is
# picked up automatically

uv run coder-eval plan tasks/hello_date.yaml # validate (no tokens spent)
uv run coder-eval run tasks/hello_date.yaml # run your first evaluation
uv run coder-eval report runs/latest # view the result


New here? Follow **Tutorial 01 — Your First Evaluation**.

Two more extras you only need on purpose: `--extra dev` adds the contributor
toolchain (pytest, ruff, pyright, pre-commit — see
CONTRIBUTING.md), and `--extra uipath` adds the in-host `uipath`
SDK for local sandbox parity (public PyPI, no credentials). Without either, the
framework still runs end-to-end.

**Just want the CLI, without cloning?** Install the published package — this is also
what a CI job or another repo does:```bash
uv tool install coder-eval    # puts the `coder-eval` CLI on your PATH,
                              # in its own isolated environment

uv tool install "coder-eval[codex,antigravity]"   # same, with agent extras
coder-eval --version                              # verify the install

To add it as a project dependency instead: uv add coder-eval or
pip install coder-eval. In a real CI gate, pin to a specific released version
so a harness upgrade can't silently move your results. (The example tasks/
live in this repo — clone it or point the CLI at your own task files.) See
Tutorial 02 — Running Coder Eval in CI for
the full setup.

Use inside Claude Code

Coder Eval evaluates any of the supported agents, and it also ships an authoring
front-end for one of them: this repo is a Claude Code plugin marketplace, so the
whole loop — scaffold a suite, author a task, check whether a skill triggers, read the
results — runs inside Claude Code. The suites you author this way run on every harness:```
/plugin marketplace add UiPath/coder_eval
/plugin install coder-eval@coder-eval


That adds six slash commands: `/coder-eval:init`, `/coder-eval:check-skill`,
`/coder-eval:task`, `/coder-eval:lint-tasks`, `/coder-eval:analyze` and
`/coder-eval:ci`. They drive the `coder-eval` CLI, so install it too
(`uv tool install coder-eval`). See Claude Code Plugin.

## Use as a GitHub Action

A composite action — on the Marketplace as
**coder_eval** — runs
`coder-eval` as a CI gate. It installs the pinned CLI, runs your tasks, writes a
JUnit XML report, reports where its artifacts landed, and fails the step on any
task failure:```yaml
- uses: actions/setup-node@v4      # the claude-code agent needs the Claude CLI…
  with: { node-version: '20' }
- run: npm install -g @anthropic-ai/claude-code

- uses: UiPath/coder_eval@v0       # …then run the gate (@v1 once 1.0.0 ships; @vX.Y.Z pins exactly)
  id: eval
  with:
    args: |
      tests/tasks/**/*.yaml
      --model
      claude-sonnet-5
    env: |
      ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}

Eight inputs, and none of them is a coder-eval run flag. The CLI has 21;
GitHub silently ignores an input the referenced tag does not define, so a
forwarding input that is mistyped or newer than your pin yields a run that
measured something else and still exits 0. A wrong CLI flag is a hard error. So
flags and task globs all go through args, and an input exists only where the
action does something with the value besides pass it along.

Input Default Purpose
args — Task paths/globs and every flag for coder-eval run, one argument per line, verbatim
version pinned release PyPI version, or local to install from the checkout
extras — coder-eval extras, composed into the install requirement (codex, antigravity,litellm)
extra-packages — Extra requirements installed into coder-eval's environment (--with), one per line
install-flags — Flags for uv tool install, one per line (--prerelease=allow, --extra-index-url …)
env — Credentials/backend passthrough: newline-separated NAME=VALUE pairs, exported for the run step only
working-directory . Directory every step of the action runs in
run-dir runs/ci Run directory; also where the reports are written

Outputs: run-dir, junit-path (/junit.xml) and run-md-path
(/run.md). The action writes nothing to the job summary — a consumer
that has to redact the report first cannot undo a write that already happened:```yaml
- if: always()
run: cat "${{ steps.eval.outputs.run-md-path }}" >> "$GITHUB_STEP_SUMMARY"
- uses: mikepenz/action-junit-report@v5
if: always()
with:
report_paths: ${{ steps.eval.outputs.junit-path }}


**Credentials and backend config** are the sole responsibility of `env` — a
passthrough exported for the run step only (never written to `$GITHUB_ENV`, so
it can't leak into later steps). Set whatever the run needs, Anthropic or not:```yaml
- uses: UiPath/coder_eval@v0
  with:
    args: tests/tasks/**/*.yaml
    env: |
      API_BACKEND=bedrock
      AWS_BEARER_TOKEN_BEDROCK=${{ secrets.BEDROCK_TOKEN }}

The step's exit code is coder-eval's own: non-zero on any failed task.

> Agent runtime is the caller's responsibility. The action is agent-agnostic —
> it installs coder-eval but no coding-agent runtime, which is why the example
> above starts with actions/setup-node +
> npm install -g @anthropic-ai/claude-code: the default claude-code agent
> needs the claude CLI on PATH before the action runs. Swap those steps for
> your own agent's runtime as needed.

> Security. Evaluated tasks execute agent-generated code. Do not run this
> action under pull_request_target with secrets exposed to untrusted fork PRs —
> use pull_request and gate on the same-repo condition, as this repo's own
> dogfood job does.

Telemetry

> 📊 Usage telemetry is on by default. coder-eval sends anonymous usage
> telemetry (command names, outcomes, counts, durations, an anonymous install id,
> platform info) to help improve the tool. It never captures prompts, file
> contents, or repo paths, and prints a one-time notice on first run. To disable
> it, set TELEMETRY_ENABLED=false
in your .env or environment. See
> Usage Telemetry for details and how to route
> it to your own resource.

Documentation

Guide What's in it
Tutorials Step-by-step walkthroughs — start here
User Guide Full CLI, configuration, output, and environment-variable reference
Task Definition Guide The task-file schema — all criterion types, scoring, templates
Claude Code Configuring and running the default Claude Code agent
Codex Running the OpenAI Codex agent
Antigravity (Gemini) Running the Google Antigravity / Gemini agent
OpenCode Running the OpenCode agent on open-weight models
Pi Running the Pi agent on open-weight models
Delegate Running UiPath Autopilot's Delegate agent
Run-Limit Parity What each run_limits field means on every harness
A/B Experiments Compare models / tools / prompts across the same tasks
Bring Your Own Dataset Fan a single task out over a dataset
Dialog Mode Evaluate agents in multi-turn conversation via a simulated user
Docker Isolation The container sandbox driver, with custom images
CI Gate & GitHub Action Run Coder Eval as a CI gate — the Marketplace Action, JUnit output, run reports
Claude Code Plugin Install the Claude Code plugin — author, run, and analyze suites from inside the agent
Extending Coder Eval Author a custom agent, criterion, or model pricing via the plugin SPI
Report Schema Field-level reference for run.json / variant.json / task.json
How It Compares vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts
Repo doc What's in it
CLAUDE.md Architecture, key patterns, and extension points
CONTRIBUTING.md Dev setup, quality bar, and how to contribute

How it compares

  • vs. fixed benchmarks (SWE-bench, SkillsBench) — they score a canonical dataset;
    Coder Eval scores your tasks with continuous 0.0–1.0 weighted criteria (and can
    still wrap a fixed dataset via Bring Your Own Dataset).
  • vs. large-scale / RL harnesses (Harbor) — Harbor targets scale and RL rollouts;
    Coder Eval targets weighted, skill-aware suites gated in CI.
  • vs. model-output eval tools (OpenAI Evals) — they grade model text; Coder Eval
    runs a full agent in a sandbox and scores the files and commands it produced.
  • vs. hand-rolled scripts — reproducible sandboxes, weighted criteria,
    cost/token telemetry, A/B experiments, and CI-ready pass/fail gates out of the box.

See the full comparison — with sources.

Task Definition

A task is a YAML file: a prompt, the agent config, a sandbox, and success criteria.```yaml
task_id: "hello_world"
description: "Create a Python script that prints Hello, World!"
initial_prompt: "Create hello.py that prints 'Hello, World!'"

agent:
type: "claude-code"
permission_mode: "acceptEdits"
allowed_tools: ["Read", "Write", "Bash"]

sandbox:
driver: "tempdir"
python: {}

success_criteria:
- type: "file_exists"
path: "hello.py"
description: "hello.py must be created"
- type: "run_command"
command: "python hello.py"
timeout: 10
description: "Script must execute successfully"


`agent.type` is the only harness-specific line: swap it for `codex`, `antigravity`, or
`opencode` — or override it per run with `coder-eval run … -D agent.type=opencode` — and
the same criteria score the same way. Tasks can omit the `agent` section entirely —
defaults resolve from the experiment layer (`experiments/default.yaml`). For the full
schema and every criterion type, see the
Task Definition Guide; for what each `run_limits` field
means on each harness, see Run-Limit Parity.

> **Tip:** With the Claude Code plugin installed, use
> `/coder-eval:task` to scaffold a task from a natural-language description, and
> `/coder-eval:analyze runs/latest` to get improvement suggestions from a completed run.

## Development```bash
make install    # package + dev + [uipath] deps + pre-commit hooks
make verify     # format + lint + typecheck + test + coverage (CI equivalent)

Run make verify before pushing — it mirrors CI (80% coverage threshold). See
CONTRIBUTING.md for the full workflow, commit conventions, and
extension points (new criteria, new agents).

Known limits & non-goals

  • Not a fixed benchmark or leaderboard — Coder Eval scores your tasks and ships
    example tasks, not a canonical scored dataset.
  • Tasks execute real code — run untrusted tasks only under the container driver
    (see Docker Isolation); the tempdir driver is not a
    security boundary.
  • Bring your own agent runtime and model credentials — Coder Eval never supplies
    model access, and it only ships an agent runtime where an extra says so (codex,
    antigravity); Claude Code and OpenCode are separate CLIs. Supply the runtime (see
    Quick Start) and the keys it needs — Anthropic, Bedrock, OpenAI,
    Gemini, or an OpenRouter key for open-weight models via OpenCode.
  • Python 3.13+ only.

Support & security

  • Security vulnerabilities — report privately via SECURITY.md; never open a public issue.
  • Bugs & questions — open a GitHub issue.
  • Everything else — reach the maintainers privately at coder-eval@uipath.com.

License

© 2026 UiPath. Licensed under the Apache License, Version 2.0 — see
LICENSE and NOTICE.

Acknowledgments

Built with Pydantic, Typer,
and Rich, on top of the harnesses it drives — the
Claude Agent SDK, the
Codex SDK, Google Antigravity,
and OpenCode.

常见问题

8 问
答案由本站依据条目字段整理,不引用上游原文
coder_eval 怎么安装?

在客户端按下面的安装命令接入:pip install coder_eval。对应的包名是 coder_eval。装完在客户端配置里引用同一个包或端点。

coder_eval 怎么在 Claude Desktop、Cursor、Cline 里配置?

它的传输方式是stdio · 本地进程,在客户端的 MCP 配置里填好启动命令和参数,保存后完全退出客户端再打开,配置才会生效。

coder_eval 收费吗?

本站不代收费用,也没有标注具体价格。是否收费取决于服务本身和背后的接口额度,正式使用前按官方说明确认。

coder_eval 需要 API Key 吗?

资料没有标注鉴权方式,是否需要密钥取决于服务本身,接入前按官方文档确认。

coder_eval 支持哪些客户端?

资料标注支持 Claude Desktop、Cursor、Cline、Claude Code;其他客户端只要能配置 MCP Server,也能用同样的命令或地址接入。

coder_eval 能提供哪些能力?

资料显示能力标签包括 agent-evaluation、agent-skills、agent-testing、anthropic、claude。具体工具清单以官方文档为准。

coder_eval 需要什么运行环境?

运行时为 Cross-platform;属于本地进程,需要在本机装好运行环境。

coder_eval 是本地还是远程?

它是本地进程(stdio),由客户端在本机拉起。传输方式为stdio · 本地进程。

热门专题

按标签挑服务器
全部专题 →

服务器信息以作者仓库与官方文档为准;分类、传输方式与鉴权方式的中文说明为中文整理,最近核验 2026-08-23。