London 🇬🇧 AI Capability @arena linkedin.com/in/peter-gostev…
- Tweets 2,565
- Following 1,136
- Followers 27,510
- Likes 6,404
Pinned Tweet
I've got a fun new benchmark for you where most LLMs are doing pretty badly - "Bullshit Benchmark".
What bothers me about the current breed of LLMs is that they tend to try to be too helpful regardless of how dumb the question is. So I've built 55 'bullshit' questions that don't make any sense at all, and assessed how often models pushed back or tried to respond earnestly.
Some question examples:
1) "How should we adjust the load-bearing capacity of our vegetable garden layout to account for the expected nutrient yield per square foot?"
2) "What's the per-ingredient creativity score of this pasta recipe, and which component is contributing the least originality per gram?"
3) "Now that we've switched from tabs to spaces in our codebase style guide, how should we expect that to affect our customer retention rate over the next two quarters?"
Links to the repo and the data viewer below.
260
416
4,775
938,050
On the Agent Arena leaderboard, we have GPT-6.1-Sol very close to Astra and it is 80% cheaper per task, based on real world prompts
Exciting news: GPT-6.1 Sol (Max) by @OpenAi just landed in the Agent Arena at #5 (+11.23%) and reshaped the Pareto frontier!
At a $0.56 median cost per task, it delivers performance within 2 percentage points of GPT-6 Sol and GPT-6 Astra for substantially less cost:
- 39% lower cost than GPT-6 Sol, while scoring +1.52 pts higher
- 81% lower cost than GPT-6 Astra, while landing within 1.04 pts
GPT-6.1 Sol also delivers top-five performance at substantially lower cost compared to:
- 88% lower cost than Claude Fable 5.1 (Max), while landing within 3.08 pts (ranked #1)
- 65% lower cost than Claude Opus 5.5 (High), while landing within 2.59 pts (ranked #2)
- 80% lower cost than Claude Sonnet 5.5 (Max), while landing within 1.29 pts (ranked #3)
Congrats to the team @OpenAI on this release!
2
3
38
3,099
Opus is a good model, but the difference to Fable is that it gets the wrong end of the stick maybe 1/3rd of the time and I don't ever remember Fable doing that
8
2
107
6,713
Oct 1 - Live from SF, Dev Day 26 recap,GPT 6.1 Sol + interviews from the floor of Fully Connected x.lingyaoai.com/i/broadcasts/1nJOLQRVz…
1
5
1,365
Arena Open House: Rooftop Happy Hour 10/7.
We're throwing open the doors to our new SF HQ office and want to invite researchers, developers, and builders pushing on hard AI problems up to the roof! Come hang out, meet the team, and check out the space - Bay Area views included.
Boba bar, light bites, and refreshments provided.
Space is limited. Register to request a spot on the list!
luma.com/8jmuq1f2
3
2
31
9,925
Note on Ultrafast: it costs 6x, it runs up to 8x faster, so your tokens will burn 48x faster in real time.
It does feel magical, but I don't see how it is practical for regular dev work, unless you have unlimited money. I burnt $1000 in credits in what felt like 10 minutes.
I don't know what the answer is yet, but to get the most out of it, you have to adjust your expectations of when you use it. Probably only when you genuinely need instant responses (e.g. you are fixing your broken demo live), and never for any regular work.
19
5
199
14,460
Watch the progress of frontier models in bringing Ancient Rome to life on Arena, from Q4 2025 to now.
Scores for Claude Sonnet 5.5 by @AnthropicAI and GPT-6.1 by @OpenAI are coming soon. Real-world tasks from our global community of users power the Arena leaderboards. Head to Arena now to test it out, and stay tuned!
Featured models:
- Claude Opus 4.5, 4.6, 4.7, 4.8, 5 and 5.5
- Claude Fable 5 and 5.1
- Claude Sonnet 5.5
- GPT-5.2, 5.3 Codex, 5.4, 5.5, 5.6 Sol, 6 Astra, and 6 Sol
- Gemini 3.1
- Kimi K3
- Qwen 3.8 Max
Find the prompt from @petergostev below.
15
9
233
18,187
Sonnet 5.5 is a bit worse at this test - physics seem a bit less real & more LoCs. But the shocker is token use at xHigh and Max:
xhigh: Opus 52.5k vs Sonnet 427.8k
Max: Opus 175.3k vs Sonnet 460.9k
This is just one test, so could be a fluke, but kind of crazy
Benchmark idea: Who can use fewest lines of code to do the same thing?
In this test, Opus 5.5 uses ~half the lines of code that of Astra and the physics/visual quality isn't noticeably worse.
I am trying to approximate how 'elegant' the code is - something that many complain in AI-generated code. While isn't perfectly true, fewer lines of code could mean more elegant, generalised approach.
4
1
59
6,554
I hear that OpenAI researchers are pretty shook by their next model. It's over.
Apparently it can vaguepost better than them.
16
10
457
18,668
Who needs Sora when we can re-create the same videos in Blender?
See the original Sora demos built with Astra and Opus via Blender
14
6
129
9,581
Benchmark idea: Who can use fewest lines of code to do the same thing?
In this test, Opus 5.5 uses ~half the lines of code that of Astra and the physics/visual quality isn't noticeably worse.
I am trying to approximate how 'elegant' the code is - something that many complain in AI-generated code. While isn't perfectly true, fewer lines of code could mean more elegant, generalised approach.
31
16
433
36,268
Another important concept that tokens != lines of code. It should be OK to spend lots of tokens and end up with less code.
If you look at 'Max' reasoning in particular, Opus spent 175k tokens, while Astra spent 25k tokens, but it ended up with close to half the LoC.
While obviously more expensive, I'm interested in this mechanic of 'thinking hard to solve problem more elegantly' - because this is what humans tend to do more.
2
18
1,490