Introducing Real-SWE, a coding benchmark built from private company codebases.
We evaluate coding agents on real software engineering tasks that engineers performed, using private, out-of-distribution codebases.
These tasks come from real companies, examples include an app with 200K+ users, fintech platform processing 100K+ bank statements, enterprise sales tools, etc.
The results are quite surprising.