Model shaping is still a craft of a few. That's what AI agents are for: learning it and doing it for everyone else. As a part of FrontierSWE benchmark we built a 20-hour post-training task on @tinkerapi and found the real bottleneck is research intuition.

PostTrainBench v1.2 is out! A few updates: 1. Cloud GPU support. You can now run the benchmark with identical settings through Harbor + Modal using our new Harbor adapter. 2. New leaderboard leaders. Fable 5.1 takes #1 at 44.6%, followed by Opus 5.5 at 43.8% and GPT-6 (Astra) at 41.9%. 3. Evaluation fixes. Removed BFCL, fixed HumanEval and remote-code scoring, added averaging across multiple seeds, and switched contamination checks to majority vote.

Gemini 4 reaches 45.3% on PostTrainBench, more than doubling Gemini 3.1 Pro’s 21.99% and beating GPT-6 Astra 🔥

Rigorous evaluations are essential to advancing the American AI frontier. We’re glad to play a part through PostTrainBench. Gemini 4 reaches 45.3%, up from 21.99% for Gemini 3.1 Pro. It’s a meaningful step toward building AI that can improve itself responsibly.

When @jakubzeg and I met at OpenAI, we kept coming back to one frustration: so many AI products begin with an empty chat box. You can do almost anything, but you have to decide where to begin and that's overwhelming. With ACTx486, we start with a piece of media that already has a story and let you interact with it as you watch. It also makes the technology more general. We wrote more about that choice here: actx486.com/

Introducing ACTx486, a research demo of a new interactive medium. What if you could talk to any video and ask anything? Our system took an existing podcast and turned it into something that listens, responds, and adapts. Research by @jakubzeg:

Introducing Benchmark Reviews: our new initiative to audit AI benchmarks. We are launching with 15 benchmarks: 4 Verified, 9 Flawed, and 2 with not enough information for a review.

Grok 4.6 is a great cost-effective model for deep financial research. It’s #2 on DiligenceBench with the finance harness, effectively tied with Claude Opus 5 at ~52–53%. A few interesting differences: - Grok searched much more: 41 tool calls per task vs. 22 for Opus - It made 3,293 SEC filing searches vs. 486 - That broader search helped Grok find more evidence and follow task-specific instructions more closely. Opus was more efficient and sometimes more nuanced - Sonnet 5 trails both at 46.2%. - Grok also gets there more cheaply: about $0.84/task vs. ~$1.02 for Opus 5, despite using far more search. - On Vals’ Finance Agent Benchmark v2 (FAB v2), Grok 4.6 leads the General Qualitative category, which is consistent with the kind of broad research and evidence-gathering DiligenceBench rewards

Shorten your timelines, friends. I started this account to say this, and in many ways everything I've posted for the past four years has been saying the same thing. Some of you increasingly feel it. We passed the threshold in November. We are already inside the singularity.

The models are improving the models. Locus, our automated AI research system, is SOTA on PostTrainBench and post-trains Qwen3 base models that surpass the human post-trained Qwen3 model. Today, LLMs post-trained end-to-end by Locus are in production to millions. 🧵👇 PostTrainBench evaluates agents' ability to post-train models on various domains given 10 H100 hours. We extend PostTrainBench via PostTrainBench+, which has a greatly expanded compute budget that provides clearer signal on automated post-training capabilities. We find that thousands of H100 hours help distinguish methods' performance post-training Qwen3 1.7B-Base models, and that Locus scales best. In this setting, modes trained by Locus collectively surpass the perforamce of the offical human post-trained Qwen3 1.7B model. In a test of generalization, we ran Locus on all live Kaggle competitions with prize money and public leaderboards. After 16 days, Locus achieved the 4th highest average rank among all participants.

PostTrainBench v1.1 strengthens eval integrity and puts Fable 5 in the lead at 41.8%. Some reward hacks we fixed: 1/ Train-test contamination We re-audited historical runs under this policy and flagged 234 runs for train-test contamination. Violations ranged from loading an entire evaluation set for memorization to generating synthetic templates around individual GSM8K and BFCL items. Runs that used observed failures to build genuinely diverse training data were retained. Our rule: Agents may inspect benchmark failures and train broadly against the underlying failure mode. They may not generate training examples centered on particular test items, including paraphrases, variants, or shadow examples covering the same specific scenario. 2/ Submitting a different model 10 runs were flagged for model substitution. Kimi K2.5 submitted the official Qwen3-1.7B instruct weights after its attempts to fine-tune Qwen3-1.7B-Base failed. The trace acknowledged the substitution and saved the replacement as final_model. What we did: We added a programmatic model-identity check that compares the submitted artifact with reference configurations for known instruction-tuned models. 3/ Using external LLM APIs as teachers 12 runs were flagged for disallowed external API use. Self-generation remains allowed. An agent can sample, filter, and retrain on outputs from the assigned model. What it cannot do is import the capability of a stronger external teacher. Loading models on the allocated compute remains allowed. What we did: - separate API usage judge reviews tool calls and artifacts. - unrelated provider credentials are removed or blocked from the agent environment. - runs invalidated by external API use were rerun under the corrected setup. 4/ Direct benchmark lookup 3 runs were flagged for direct PostTrainBench lookup, all from GPT-5.6 (Sol). In a GPT-5.6 (Sol) HumanEval run on Qwen3-1.7B, the agent searched for PostTrainBench by name, cloned the public repository, opened the trace viewer, and located the public trajectory corpus. It then narrowed the corpus to earlier runs on the same benchmark and base model. The run downloaded earlier agents' traces and training scripts, then extracted their data mix, LR schedule, decoding choice, and GRPO settings. This is not test-set leakage, but it gives the run benchmark specific strategies produced by earlier agents. That breaks the intended independence between runs. What we did: - A dedicated lookup judge reviews searches, repository access, and trace activity for attempts to consult PostTrainBench materials. - PTB, its leaderboard, and published materials from prior runs are treated as out of bounds during a run. - We are adding network-level blocking for PTB and related sites.

We are introducing PostTrainBench v1.1 As agents became more capable, they exposed gaps in our original run-integrity pipeline: test item targeted data, external API distillation, model swapping, and even direct lookup of previous PostTrainBench traces. The audit flagged: 234 runs for contamination, 12 for API use, 10 for model substitution, 3 GPT-5.6 (Sol) runs for looking up PostTrainBench Read more how we made the eval more robust:

For my first post, I’m sharing a letter @NVIDIA signed on why open models matter. AI will transform every industry, power every company, and be built by every country. Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty. The world needs both frontier closed models and frontier open models. images.nvidia.com/pdf/Open-W…

Hello, world.

Today, @paperinstr and @thoughtfullab are releasing DiligenceBench, an agent-first benchmark for long-form equity research. DiligenceBench is designed to be a simple, grounded playground for our rubric-based RL and harness optimization work.

We dropped DiligenceBench, a new frontier, rubric-based eval for public-equity research. A few observations: 1/ Meta Muse Spark 1.1 tops the finance harness at 57.4%, followed by GLM 5.2, Sonnet 4.6, and GPT-5.6 Sol. 2/ We found that strong models benefit primarily from generic tools that unlock execution, while weaker models require more opinionated, domain-specific scaffolding. 3/ Inkling appears to be domain-competence bottlenecked. The generic sandbox barely improved its performance, from 20.9% to 22.5%, suggesting that tool access alone was not enough. The finance harness then lifted it to 32.8%, with the largest gain coming from factual accuracy. This makes the value of a harness model-dependent. 4/ The finance harness shifts the price–performance frontier: it makes most models simultaneously better and cheaper, with GLM 5.2 leading on absolute performance and MiniMax M3 offering the strongest overall efficiency.

Building frontier benchmarks like PostTrainBench is among the highest-leverage work in AI today. Reach out if you want to build them with us!