I lead formal methods for @anduriltech; YC S24; CS PhD & GRFP; 10p blue belt; created bstn.cc in 2020
Boulder, CO
- Tweets 2,799
- Following 3,651
- Followers 1,213
- Likes 20,869
Pinned Tweet
I’m beyond excited about this. We are bringing together startups, academics, hackers, and random degenerate type theory friends of ours, to answer the following question totally empirically: *are current LLMs good enough to let normal SWEs without specialized training write meaningful, formally verified software*?
The answer may be a resounding no! In which case, we will analyze each failure point and suggest corresponding research directions for usable fm.
You should join us. It’s gonna be rad!
We’re organizing Vibecheck, a formal methods hackathon with some of the best folks in the space - @maxvonhippel, @qd_forall, @nolanlwin, @jessemhan, @emiyazono, and @workersio.
You’ll get a weekend to build real production software, and formally verify it.
We’ll have people who really know their stuff around to help, and you can come with a team, find one there, or just hack on something yourself.
We are glad to have a phenomenal group of sponsors, including @harmonicmath, @mathematics_inc, @theoremlabs, @thegp, @omnicom, @primeintellect, @astriolabs, @buildwithparty, @wearerandomlabs, and @lanyon_ai, as well as some more we will announce in the coming days!
If you’re a hacker, a formal methods person, or just someone who thinks “how do we know it works?” is an interesting question, we’d love to have you.
Signup Now: fmxai.org/vibecheck/
4
10
72
32,633
The equivalent of this in software engineering is not another agent reviewing the output and telling you it's good.
It's precise specifications, proofs, model checking and simulating real production scenarios as proof of work.
There is no way any of us are going to look at code the same way again.
We'll be spending a lot more time trying to understand the outputs of language models. A few thoughts, tips & tricks:
Writing. Something I've had success with: Ask your LLM to explain something in ASD-STE100, it's a controlled language specification originally developed for aerospace maintenance documentation. LLMs well-versed in this language and it comes with heavy constraints on clean writing style that I often find a lot more readable. Sometimes I've tried to soften it a bit e.g. ask for "80% of the way to ASD-STE100" because the spec is quite stringent. But even better:
Diagrams / images. Instead of writing, ask your LLM to create a diagram. These can be a lot easier to process, parse, and understand. But even better:
Web pages. Ask for output "in HTML" to get a beautiful, interactive webpage. LLMs are getting really good at frontend and can create beautiful experiences, animations, etc. But even better:
Explainer videos. The output format I am most bullish on is fully custom / bespoke explainer videos generated on any arbitrary topic. Experiment with things like "Create a 3b1b style video explainer on X. Use my ElevenLabs API key for audio narration". (you'd need an API key for the latter or you can ask your LLM to find you decent free alternatives that use your local compute). This is actually starting to work!
In summary:
- As LLMs get better, they will do more and more of the legwork autonomously, and a lot more of our work will rise up the abstractions into oversight and understanding.
- Luckily, LLMs can help here too because as intelligence and code are increasingly abundant, you can ask for large, custom, discardable software artifacts (e.g. web apps, video explainers) that would have never made sense to create before. Push the boundaries here and you'll be surprised.
1
2
10
1,122
RT @mweber_PU: Moving from individual proofs to large-scale autoformalization requires new tools. We introduce Choir, an open protocol for…
76
1
I asked someone at the foresight institute event last night what they do and they replied “I’m an expert at AI” and I thought, that might be the most concise possible way to inform me that you are not
7
2
62
1,615
Max von Hippel retweeted
the year is 2026
in order to make agi write like a normal human being you have to ask it to turn the knob 80% towards ASD-STE100 a language specification made for aerospace maintenance documentation in 1986
We'll be spending a lot more time trying to understand the outputs of language models. A few thoughts, tips & tricks:
Writing. Something I've had success with: Ask your LLM to explain something in ASD-STE100, it's a controlled language specification originally developed for aerospace maintenance documentation. LLMs well-versed in this language and it comes with heavy constraints on clean writing style that I often find a lot more readable. Sometimes I've tried to soften it a bit e.g. ask for "80% of the way to ASD-STE100" because the spec is quite stringent. But even better:
Diagrams / images. Instead of writing, ask your LLM to create a diagram. These can be a lot easier to process, parse, and understand. But even better:
Web pages. Ask for output "in HTML" to get a beautiful, interactive webpage. LLMs are getting really good at frontend and can create beautiful experiences, animations, etc. But even better:
Explainer videos. The output format I am most bullish on is fully custom / bespoke explainer videos generated on any arbitrary topic. Experiment with things like "Create a 3b1b style video explainer on X. Use my ElevenLabs API key for audio narration". (you'd need an API key for the latter or you can ask your LLM to find you decent free alternatives that use your local compute). This is actually starting to work!
In summary:
- As LLMs get better, they will do more and more of the legwork autonomously, and a lot more of our work will rise up the abstractions into oversight and understanding.
- Luckily, LLMs can help here too because as intelligence and code are increasingly abundant, you can ask for large, custom, discardable software artifacts (e.g. web apps, video explainers) that would have never made sense to create before. Push the boundaries here and you'll be surprised.
18
17
647
32,318
Generations of gender studies PhD dissertations will be written about the completely fucked second order effects of letting generative ai models produce, field, and a/b test video advertisements totally autonomously — effectively hill climbing on “how hot is too hot?”
3
17
690
Example:
The whole point of building Tavus has been simple: talking to a machine should feel as natural as talking to a friend or coworker.
It’s hard to describe all the tiny nuances that make a conversation feel human. The little expressions. Moving around in your chair. Knowing when to speak and when to listen. The dance of it all.
Griffin is by far the closest anyone has come to a model that can capture those nuances. The first time I saw it being used, I had no idea I was watching our model rather than just a normal video call.
I’m so incredibly proud of this team and what they’ve built.
1
99
People complain all the time about the slop that comes out of startups. But they’re also a glimpse into the future! Eg, the inspect element primitive is going to be a part of every normal UX designer’s toolbox now. Inspect -> edit -> refresh.
1
11
1,074
Max von Hippel retweeted
You know what would be great? A single point of failure for all of our business processes.
Today we’re launching LiteLLM Lens.
@LiteLLM is already the chokepoint for 100% of your enterprise’s AI traffic. We believe the next era of the gateway is using the data flowing through it to help your agents improve.
We’re building for a future where developers run agentic swarms generating 200K+ traces.
We’re focused on two things ⬇️
h/t @Mokhalil01 @yujonglee
5
1
38
4,603
Max von Hippel retweeted
Hey @GayaniFigma , I think in spirit, when I created MCP, I envisioned an open ecosystem. That to me feels core. Seeing restrictions like this is sad and I hope Figma can get to a point where it’s more open or at least make the process of getting into the allowlist very easy.
40
55
1,837
193,037
Max von Hippel retweeted
Models being good at writing is key for humans to stay in the loop. Also it's so much fun to play with
Say hello to Echo, the best writing model at style imitation. Echo beats frontier models at writing tasks ranging from fiction to technical explanations, despite costing less than $5K to train.
9
14
438
39,858
Max von Hippel retweeted
I got to try this model a little before release and it's very cool. Not many humans can imitate the style of Tom Robbins or Neal Stephenson, but Echo did very well!
if you give it a try send me cool screenshots please. I love literature
Say hello to Echo, the best writing model at style imitation. Echo beats frontier models at writing tasks ranging from fiction to technical explanations, despite costing less than $5K to train.
17
12
362
25,788
Max von Hippel retweeted
fun fact, the letters of OOPSLA rearrange to OSLOPA. this is a subtle nod to the fact that OOPSLA is filled with slop, and I think that's a beautiful thing
2
3
20
1,282
Max von Hippel retweeted
hi new followers, still hiring a formal methods person, slightly theory-heavier than most nonacademic JDs but still involves slinging code. DM me for JD
3
2
24
1,373
Max von Hippel retweeted
You've heard lots about the promise formal methods, you aren't yourself a formal methods expert.
Come judge for yourself!
Knowing who is organising it, I know this is gonna be a super fun event.
We’re organizing Vibecheck, a formal methods hackathon with some of the best folks in the space - @maxvonhippel, @qd_forall, @nolanlwin, @jessemhan, @emiyazono, and @workersio.
You’ll get a weekend to build real production software, and formally verify it.
We’ll have people who really know their stuff around to help, and you can come with a team, find one there, or just hack on something yourself.
We are glad to have a phenomenal group of sponsors, including @harmonicmath, @mathematics_inc, @theoremlabs, @thegp, @omnicom, @primeintellect, @astriolabs, @buildwithparty, @wearerandomlabs, and @lanyon_ai, as well as some more we will announce in the coming days!
If you’re a hacker, a formal methods person, or just someone who thinks “how do we know it works?” is an interesting question, we’d love to have you.
Signup Now: fmxai.org/vibecheck/
2
2
14
1,177
