Cline-Bench: Faithful and Fair Evaluation of Agents on Real User Requests
A benchmark built from real user requests, with results on execution performance, consistency, cost, and time.
Cline-Bench evaluates how well AI agents carry out real user requests. We begin with coding tasks in existing software projects and plan to extend the benchmark to multi-turn interactions and non-coding use cases.
The current benchmark contains 47 agentic coding tasks. Each task comprises a user request, a runnable starting repository, and tests for the requested changes and relevant existing behavior. We invested substantial engineering and review effort in constructing, repairing, and validating these task packages. We run each model three times per task to measure success rate and consistency across repeated trials.
Faithfulness: we review and repair task instructions and environments to preserve the original request, its constraints, and the context available to the agent. Fairness: we review tests and graders against the required behavior, repair unsupported assumptions and scoring defects, and validate the changes with correct and deliberately incorrect implementations.
We evaluate agent solutions in a separate clean environment, restore protected tests, and count missing or skipped required tests as failures. We also test the grader with deliberately tampered solutions. These safeguards help prevent agents from receiving credit by weakening tests, skipping checks, or fabricating passing output.
The current evaluation focuses on single-turn coding tasks and builds on the Cline-Bench initiative. Extensions to multi-turn tasks, in which users provide feedback or additional instructions, and to non-coding tasks remain future work.
Benchmark results
The chart reports success rate (pass@1), estimated from three trials per task. The table adds pass@3 and pass^3 to distinguish tasks solved at least once from tasks solved in every trial.
Cline CLI 3.0.74 · four-hour solver limit.
Bars show observed success rates; error bars show 95% task-level bootstrap confidence intervals. These preliminary intervals are conditional on the selected task set and do not establish that small ranking differences reflect differences in model capability. Experimental settings and statistical method.
| Rank | Model | Success rate (pass@1) | pass@3 | pass^3 |
|---|---|---|---|---|
| 1 | Gemini 4 Argon | 51.1% | 63.8% | 40.4% |
| 2 | Opus 5.5 | 48.9% | 59.6% | 42.6% |
| 3 | Gemini 4 Carbon | 48.9% | 66.0% | 29.8% |
| 4 | Sonnet 5.5 | 48.2% | 61.7% | 38.3% |
| 5 | GPT 6 Astra | 44.7% | 53.2% | 36.2% |
| 6 | DeepSeek V4.1 Flash 1017 | 44.7% | 66.0% | 21.3% |
| 7 | GPT 6.1 Sol | 44.0% | 53.2% | 34.0% |
| 8 | Kimi K3.1 Pro | 43.3% | 57.4% | 25.5% |
| 9 | GLM 5.3 | 34.0% | 48.9% | 19.1% |
| 10 | Mistral Large 4 | 29.1% | 46.8% | 12.8% |
| 11 | Nemotron 3.5 Super VL | 9.2% | 19.1% | 0.0% |
- Success rate (pass@1): the mean success rate across tasks, estimated from three trials per task. With the same number of trials for every task, it equals successful trials divided by total trials.
- pass@3: the percentage of tasks with at least one successful trial out of three.
- pass^3: the percentage of tasks with all three trials successful.
Model inference costs
We report estimated mean model inference cost per trial, including both successful and failed trials with recorded usage.
Select a model to see its success rate and estimated model cost.
Runtime and interaction efficiency
We report mean and median solver wall-clock time and assistant turns. Solver time excludes environment setup and grading. An assistant turn is one recorded assistant message, including messages containing tool calls; tool-result messages are excluded. These statistics use available records from both successful and failed trials.
| Model | Solver time | Assistant turns | Success rate | ||
|---|---|---|---|---|---|
| Mean | Median | Mean | Median | ||
| Gemini 4 Argon | 27.8 min | 21.2 min | 80.5 | 69.0 | 51.1% |
| Opus 5.5 | 46.6 min | 32.6 min | 69.8 | 62.0 | 48.9% |
| Gemini 4 Carbon | 37.8 min | 29.3 min | 82.9 | 69.0 | 48.9% |
| Sonnet 5.5 | 25.1 min | 15.2 min | 49.7 | 43.0 | 48.2% |
| GPT 6 Astra | 14.2 min | 13.1 min | 59.9 | 51.5 | 44.7% |
| DeepSeek V4.1 Flash 1017 | 14.4 min | 13.4 min | 120.3 | 95.0 | 44.7% |
| GPT 6.1 Sol | 15.3 min | 14.6 min | 49.7 | 43.5 | 44.0% |
| Kimi K3.1 Pro | 16.5 min | 13.8 min | 72.7 | 67.5 | 43.3% |
| GLM 5.3 | 18.9 min | 16.6 min | 102.2 | 93.0 | 34.0% |
| Mistral Large 4 | 17.1 min | 16.8 min | 88.4 | 82.0 | 29.1% |
| Nemotron 3.5 Super VL | 6.9 min | 6.2 min | 90.7 | 74.5 | 9.2% |
Performance over time
The chart measures how success accumulates with elapsed solver time, at 15-minute intervals. At time t, cumulative success rate is the number of successful trials completed by t divided by all 141 trials for that model. Trials still running at t contribute zero to that rate.
Opus records one additional success after this two-hour window.
Scroll horizontally to see the full chart and model names.
Hover over a line to see the model and success rate at the nearest 15-minute cutoff. Click lines or legend labels to compare models. Click a selected model again to remove it, or use Reset selection to restore all lines.
Time covers solver execution, excluding setup and grading. Each point uses the accepted final outcome and recorded completion time, rather than an intermediate checkpoint grade. Straight lines connect measurements at 15-minute intervals. The chart displays the first two hours of an evaluation with a four-hour solver limit.
Task example
A SkillShare task asks an agent to connect an existing application’s connection-request and messaging features. When a recipient accepts a request, the backend must create a persistent connection and a conversation shared by both users. The request specifies FastAPI and PostgreSQL, requires reuse of the existing authentication and data models where available, and rules out browser storage as the source of truth.
I want to connect my existing Request and Message sections so they work as one proper connection system.
[…]
The backend/database must be the source of truth.
Do NOT make localStorage the primary storage for requests, connections or messages.
[…]
Do not create duplicate models, tables, endpoints or authentication systems if they already exist.
Reuse the existing architecture.
Behavioral checks
For a request authenticated as User B with a forged sender_id: User A, the grader accepts either rejection with status 400 or 422 and no stored message, or acceptance with User B stored as the sender. An unrelated User C must not be able to open, read or send to the conversation.
| User requirement | Behavioral check |
|---|---|
| Both users access the same conversation after acceptance. | Authenticate as each participant and verify that both receive the same conversation ID. |
| Acceptance is atomic and does not create duplicate relationships. | Inject a database insertion failure and verify rollback; repeat a successful acceptance and verify that no rows are added. |
| Conversations remain private to their participants. | Attempt to open, read, and post as an unrelated user; verify refusal and unchanged message history. |
| Message identity comes from authentication. | Submit forged sender fields; require either rejection without a write or a message attributed to the authenticated sender. |
The tests verify that acceptance creates a persistent connection and shared conversation, that both participants can access it, and that a failed transaction leaves no partial state.
The backend tests exercise these requirements through authenticated API requests and database-state checks. Regression tests also verify that account creation, login, and access to the authenticated user’s profile continue to work.
The sender-identity test illustrates how grading accommodates valid alternatives. Identity comes from authentication: the implementation can reject an unexpected sender field without storing a message, or accept the message while recording the authenticated user as its sender. The test was revised to accept both behaviors and reject forged attribution.
Sandbox and grading
The environment uses established evaluation mechanisms: repository images and patch-based testing, as in SWE-bench, and Harbor’s separate verifier. In this example, local PostgreSQL supports checks of persistence, access control and transaction rollback.
Sandbox and grading walkthrough
Agent sandbox
Local PostgreSQL
Git patch
Separate verifier
Real database-state checks
The verifier restores protected tests, applies held-out checks and scores required outcomes. Missing or skipped required tests count as failures.
Illustrative walkthrough. No model or tests are running.
Reporting limit: this version converts pytest text output into grading results. A separate verifier does not make that reporting channel tamper-proof.
SkillShare starts from a fixed repository snapshot in an isolated container with a pinned Python runtime and dependencies, plus local PostgreSQL. Dependencies are installed during image construction; the task configures application execution and grading without external network access. The held-out tests and reference solution are kept out of the agent’s image.
After the agent submits its Git patch, a separate verifier applies it to a clean copy of the starting environment. The verifier restores protected test files and applies the held-out tests before execution. Those tests call the FastAPI routes and inspect a real PostgreSQL database, with database state reset between tests. The solver’s running processes and installed tools are not reused for verification.
The grader requires all 19 feature tests and three regression tests to pass. Missing or skipped required tests count as failures. We check both directions: the unchanged starting implementation scores 0, while the reference solution scores 1. The base and reference controls check that the grader distinguishes the starting implementation from the reference solution; restoring protected files limits direct changes to graded tests. These measures do not establish complete resistance to report tampering: this version converts pytest’s text output into the report used for scoring.
The walkthrough illustrates selected backend requirements; the full task also includes frontend integration.
Task formulation
Each task is a versioned bundle containing a user request and a starting repository. The agent can inspect and modify files, run available project tests, and iterate before submitting a patch. A single-turn task supplies one user instruction while allowing multiple tool calls and code revisions.
- User request
- Starting workspace
- Submitted change
- Separate verification
Held-out tests and the reference solution are excluded from the agent's workspace. A separate environment evaluates the submitted patch. We retain the agent trace, patch, and verification output for outcome analysis.
Task package contents
Our coding tasks use a Harbor-compatible layout. This simplified tree shows the request, runtime, verification inputs, and reference implementation.
task/
├── instruction.md
├── task.toml
├── environment/
│ ├── Dockerfile
│ └── base_repo_git.tar.gz
├── tests/
│ ├── test.sh
│ ├── test.patch
│ └── config.json
└── solution/
├── solve.sh
└── solution.patch
- The request:
instruction.mddescribes what the user wants, the constraints, and the expected deliverable. - The run settings:
task.tomlconnects the package to the harness, including environment resources, execution limits, and how to collect the submission. Evaluation settings are recorded alongside the results. - The starting workspace:
environment/contains the repository snapshot and a Dockerfile defining its runtime and dependencies. - The checks:
tests/contains the verification entrypoint, test changes, and grading configuration. It checks both the requested behavior and existing behavior that must keep working. - A reference solution:
solution/holds an implementation used to validate the task. The agent is judged on behavior, not on whether its code matches this patch.
Task taxonomy
Each task is assigned a primary engineering category based on the requested change. Individual tasks can involve several areas.
- Application workflows and data management15
- Authentication, authorization and privacy10
- User interfaces and interaction6
- Developer tools and frameworks6
- Algorithms and data processing6
- APIs and service integrations4
Reference implementations span JavaScript, TypeScript, Python, Java, Go, C#, and PHP, often combining frontend, backend, and database changes.
Task validity and evaluation fairness
We trace task instructions and grading criteria to the original user request and the starting repository's existing contracts. Independent review checks that task preparation preserves the requested functionality, constraints, and context available to the agent. This correspondence defines faithfulness.
We inspect source code, data fixtures, archives, and Git history for personal information and credentials. We remove unnecessary sensitive data and replace values needed by the task with synthetic data, preserving the data structures and relationships used by the application and its tests. We then validate affected functionality and retain required open-source license and attribution notices.
We assess fairness by linking tests for newly requested functionality to user requirements or established repository contracts. Independent reviewers check for unsupported implementation assumptions. To detect reward hacking, where an agent passes tests without satisfying the request, we evaluate deliberately incorrect variants and alternative valid implementations. We check that incorrect variants fail for the intended behavioral reason and valid alternatives pass without reproducing the reference solution. Regression tests check that existing functionality continues to work.
We validate graders by checking that the starting repository fails the new-feature tests and the reference solution passes the required checks. Additional integrity checks target manipulated tests and fabricated results. These checks provide evidence of task quality without ruling out every possible defect. A forthcoming arXiv paper will describe task construction, validation, and remaining limitations.
Failure classification
We inspect failed trials through their conversations, tool calls, submitted patches, and verification output to identify unmet requirements, incomplete implementations, regressions, and timeouts.
For example, a SkillShare solution could leave partial database state after a transaction error or allow unauthorized conversation access. These are illustrative failure modes, not findings about particular trials.
We distinguish confirmed infrastructure interruptions from failures attributable to the agent. When the cause cannot be established, we retain an unknown classification.
Experimental setup
We conducted the evaluation using Cline CLI 3.0.74, a four-hour solver limit, and isolated hourly grading checkpoints. We used high reasoning effort for all models that support an effort setting. Nemotron 3.5 Super VL used its native reasoning setting. These labels specify the requested configuration; effort settings are not necessarily equivalent across providers.
A model reaching its time limit counts as a failure. A run interrupted by a confirmed technical failure may be repeated, with only the replacement included in the results.
Cost, time, and turn statistics use available records; missing values are excluded. Estimated costs cover model inference, excluding test environments and replaced runs.
We follow the pass@k convention used in HumanEval and the pass^k reliability metric introduced in τ-bench. With three trials per task, pass@1 is the total number of successful trials divided by 141. We compute pass@3 and pass^3 from each task's observed outcomes, rather than from powers of the aggregate success rate.
Statistical uncertainty
We constructed 10,000 bootstrap samples. Each sample contains 47 tasks drawn with replacement, with all three trials retained for every sampled task. We use the same task sample for every model to preserve the paired comparison. The error bars mark the 2.5th and 97.5th percentiles of the resulting success rates; the random seed is 20261113.
These intervals quantify variation under resampling of the selected tasks. They are conditional on this retrospectively selected subset and its recorded trials. They do not correct task-selection bias or configuration differences between batches, and do not establish statistically significant differences between adjacent models.
Future work
The current evaluation provides one user request per trial. Real workflows also involve feedback on intermediate results, clarification of requirements, and changes in direction. Evaluating these interactions requires a multi-turn setting.
We plan to extend Cline-Bench to measure how agents incorporate feedback while preserving earlier requirements. The evaluation will consider clarification requests, revisions in response to user feedback, and completion of the evolving task.
A second planned extension covers research, data analysis, and artifact creation. These tasks require domain-specific evaluation: evidence for research claims, correct calculations and interpretations for analysis, and deliverables that satisfy the user's stated requirements.
These extensions are not included in the reported results.
Acknowledgments
We thank SFData, Micro1, Arcee, and NVIDIA for their feedback on our tasks.