Warp · Sep 17, 2026 · 4:31 PM UTC

Warp

Warp

@warpdotdev

Sep 17

Introducing Scorers: Agents that grade your agents. Use LLM-as-a-judge to grade past coding agent sessions on: • Quality • Efficiency • Compliance • Or any custom dimension Scores feed into performance measurements and automatic self-improvement for software factories

191

568,298

Warp · Sep 17, 2026 · 4:31 PM UTC

Warp

@warpdotdev

Sep 17

Start from a record of agent conversations you want to grade. In Warp Factories, all agent runs are tracked across your team with agent traces, artifacts, and code output. You can score all of your agent runs, or define a sampling you want to test (ex. frontend tasks)

ALT All of a team's agent sessions tracked in the Warp Factories dashboard‌‍‍‍‌‍‌‍‌‍‍‌‌‍‌‌‍‍‌‌‍‍‍‍‍‍‍‍‌‌‍‌‌‍‍‌‍‍‌‌‌‌‍‌‍‍‌‍‍‌‌‍‍‍‍‍‍‌‍‍‌‍‌‍‌‌‌‍‌‍‍‍‍‍‍‍‍‌‌‌‌‌‌‌‍‌‌‍‍‌‌‍‍‌‍‍‌‌‍‍‌‌‌‍‌‌‌‍‍‌‌‍‌‍‌‌‌‍‌‌‍‍‌‌‌‍‌‍‌‌‍‌‍‌‌‍‌‌‌‌‌‍‌‍‌‌‌‌‍‌‌‌‍‍‌‌‌‍‌‌‌‌‍‍‌‌‍‌‍‍‍‌‍‍‌‌‍‌‌‌‍‍‌‍‌‍‌‍‌‍‌‌‌‍‌‌‌‍‌‌‌‌‌‌‍‍‌‌‍‍‌‍‌‍‌‌‍‌‍‌‍‌‍‌‌‍‌‌‍‌‌‍‌‍‌‌‍‍‍‌‌‌‌‍‌‌‍‌‌‍‌‌‍‌‌‌‍‌‍‌‍‌‍‍‌‌‍‍‌‍‌‍‌‌‌‌‍‍‌‌‍‌‌‍‌‌‌‍‌‌‌‌‌‌‍‌‍‌‌‍‌‌‍‌‌‍‌‌‌‌‍‍‌‌‌‍‌‍‌‍‌‌‌‌‌‍‌‌‍‌‍‌‌‍‌‌‍‌‍‌‍‌‌‍‌‌‌‍‌‍‌‍‌‍‌‌‌‍‌‌‌‍‌‌‌‌‍‍‌‍‌‍‌‍‌‌‌‌‍‌‌‌‍‌‌‍‌‌‌‌‍‍‌‌‍‍‌‍‌‍‌‌‍‌‍‌‌‌‌‌‍‌‌‌‍‍‌‍‌‌‌‍‌‍‌‌‌‌‍‌‌‌‌‍‌‌‍‍‌‍‌‍‌‌‌‌‍‌‍‌‌‍‌‍‌‌‌‍‌‌‍‌‌‌‍‍‌‌‍‌‌‍‌‍‌‍‍‌‌‍‌‌‌‍‌‌‍‌‌‍‌‌‍‌‌‌‍‍‌‌‍‌‌‍‌‌‌‌‍‍‌‍‌‌‍‍‍‌‌‌‌‌‌‍‍‌‍‌‍‌‌‌‌‌‍‍‌‌‍‌‍‍‌‍‍‌‍‌‌‌‍‌‌‌‌‌‌‌‍‌‍

1,193

Warp · Sep 17, 2026 · 4:31 PM UTC

Warp

@warpdotdev

Sep 17

Then, create your scoring agents. These analyze individual conversations and give them a grade. Scorers follow rubrics that you define. Here's a "redundant test" scorer we use internally to judge the quality of tests that our agents write: gist.github.com/bholmesdev/d… 🔖

ALT Scoring rubric defining surplus tests, pass thresholds, and sample rates for implementation-agent runs.

ALT Scorer configuration with four classifications, a 0.50 pass threshold, and a 100% sample rate.

832

Warp · Sep 17, 2026 · 4:31 PM UTC

Warp · Sep 17, 2026 · 4:31 PM UTC

Warp

@warpdotdev

Sep 17

There are many different scorers that you can define. These are the ones we'd start with: Compliance: did the agent complete the task per the user’s request? Efficiency: did the agent complete the task efficiently, or did it do a bunch of unnecessary work? Verbosity: did the agent write succinct code with clear code comments, or was the output unnecessarily elaborate? Quality: for a coding task, was the quality of the code good? Did it match expected conventions?

ALT Dashboard showing pass-rate trends for six agent scorers, including redundant tests, task compliance, and procedure compliance.

Sep 17, 2026 · 4:31 PM UTC

517

Warp · Sep 17, 2026 · 4:31 PM UTC

Warp

@warpdotdev

Sep 17

Then, aggregate scores to view success and failure trends overtime. This helps observe regressions and track improvements as you ship changes to your factory. Here you can see dips in our redundant test quality, with visibility into failed runs for deeper investigation:

ALT Scorer dashboard showing pass rates over the last 30 days and recent runs with results, scores, classifications, runtimes, and costs.

423

Warp · Sep 17, 2026 · 4:31 PM UTC

Warp

@warpdotdev

Sep 17

Finally, use these metrics to have agents suggest improvements to your setup automatically. Self-improvement agents run on a schedule and review failing grades to suggest changes to your setup. Here is an example agent skills PR with evidence cited from previous scoring runs:

ALT Merged GitHub pull request fixing triage agents that continued polling after sending a completion report.

882

Warp · Sep 17, 2026 · 4:31 PM UTC

Warp

@warpdotdev

Sep 17

If you want to set this up, scoring is part of Warp Factories in early access. We are offering up to $10k in usage to qualified companies. warp.dev/factories/request-a… 🔖

Request Early Access — Warp Factories

Request early access to Warp Factories. Qualifying orgs get up to $10,000 in factory usage during closed early access.

warp.dev

847

Sort replies: Relevant Recent Liked