Designing computer-use agents @yutori_ai
- Tweets 7,511
- Following 997
- Followers 1,711
- Likes 32,440
Pinned Tweet
Excited to have my Mousepower talk live on @aiDotEngineer's YT channel (link below), but I have retrospective thoughts on this topic & here is what I'd change if I gave the talk again today:
1. Thesis wasn't stated simply enough: value of agents is limited by our ability to measure them, otherwise we can't justify the ROI of their token cost
2. Need an explicit connection to the discourse on verifiable domains, which we need not be limited to today's set as understanding customer mental models unlocks new ways to verify value (e.g. horsepower)
3. After reading a nice post by @jon_stokes on verifiable domains, I'm specifically interested in the n-order F/X of their "optimization pressure"
4. Also rambled too much, perhaps a function of the topic being too underbaked... will see if GPT voice can coach me for next time
Follow up incoming (someday).
Excited to speak again at @aiDotEngineer World's Fair! This year I'm presenting "mousepower": on how agents break our measurement frameworks, which weren't built for systems that output tirelessly in parallel. If execution is now cheap & judgment is the new bottleneck, how do we measure value at the speed of compute?
1
2
6
3,035
Forgot I had this random Figma file w/ a bunch of photos from my travels that I wanted to use as design references. Might need to vibe code a web viewer so they're easier to parse + share.
1
1
193
I've never seen Overmono perform live & I now realize I must correct this mistake immediately...
1
125
A new personal behavior around this: within an IDE's agent view, I open relevant browser tabs directly in a thread. The pages sit next to the work they inform, within a worktree that I'll archive post-merge. Creates a nice hierarchy & flow.
Seems many, like Josh (who I respect & do agree w/ his take’s broad strokes), have missed a key detail: AI browsers aren’t dead, nor been bailed on, they are instead being folded into agent-first applications (e.g. Claude Code/Cowork, Codex/ChatGPT Work). The key innovation isn’t an AI-first browser, it’s an agent harness so good that we never need to open a browser ourselves.
1
417
All these live wallpaper examples circulating on X recently may look like fun toys, but I think they're emblematic of our desire for the next wave of agentic interfaces / a new window into our evolving machines.
286
Maximillian Piras retweeted
I know we are all focused on AI agents
But a good human agent (w/AI skills + taste + agency) can transform your business much more. There is unreasonable alpha in finding these human agents maybe now more than ever.
53
13
232
13,429
The idea to have a CUA leverage the a11y tree was quite clever, which I believe @skybysoftware pioneered. I'll be very happy if it ironically comes full circle by unlocking interfaces that make computers even more accessible!
OpenAI quietly solved the single problem that's been blocking computer-use agents from going mainstream: the cursor war.
Anthropic shipped computer use in October 2024. OpenAI shipped Operator. Google shipped Mariner. All three hit the same wall. When the agent moves the cursor, you can't. The screen is one resource, and two drivers cannot share it. Every demo ended the same way: "run this while I go get coffee."
That kills the ROI math. If the human has to stop working to let the agent work, the agent is functionally a replacement. Adoption of computer-use agents flatlined at power users and scripted demos for exactly this reason.
Here's what OpenAI actually shipped. Codex runs multiple agents against your real Mac apps, in parallel, with their own cursors, in the background, while you keep using your machine. The mechanism is OS-level sandboxing. OpenAI acquired Sky Applications last fall. That's the team that built Workflow, which became Apple Shortcuts. They spent a decade figuring out how to let third-party code drive other Mac apps without breaking the foreground experience. OpenAI didn't buy a team. They bought the only people on Earth who had solved this.
The economic unit of white-collar work just changed. Yesterday the ceiling was one human with one agent doing one task. Today it's one human supervising five agents working across five apps the human is simultaneously using. The bottleneck stops being compute. It becomes the human's ability to review output.
Codex has 3M weekly devs. The feature that matters is not the IDE. It's that OpenAI turned the Mac into a multiplayer instrument.
233
An early thesis we had at Yutori, which culminated in our first app called Scouts, is that our persistent need for timely information is still so underserved & thus we often make under-informed decisions across our daily lives. Agents may solve this.
haven’t seen it discussed this weekend yet on east coast, but killer use case for instinct is sifting through municipal updates on the nor’easter and giving me real-time updates on down power lines, roads flooding, etc.
1
5
522
I once gave a talk called The Bitter Layout, inspired by this same book, arguing that until models commoditize, interface design should primarily focus on absorbing the latest capabilities. Well, it may soon be time to dust off the “chat isn’t the future of AI UX” t-shirt...
No one is going to care about the underlying AI model soon, especially for consumers. The models are already good enough for most of what people need. What consumers want is for AI to be useful, easy to access, and FREE or bundled into something they already pay for.
Clayton Christensen and Michael Raynor described this shift in the figure below which is in their book The Innovator’s Solution. When a product isn’t good enough, performance wins. Once it is good enough, convenience and cost start to matter more.
The winning question may soon be less Whose model is best? and more Why would I pay separately for this?
1
263
the right way to use model capabilities is not to ship 10x more features to prod
it's to spend more time understanding your users, trying experiments, building prototypes, learning about things you don't understand so that you can ship things that actually work
276
618
8,242
390,897
Maximillian Piras retweeted
Every platform needs to service the developer community better than the hot launch period.
Apple’s success comes from courting all types of developers for years on end, not the top 7 icons recognizable by the X community at any moment.
Are normal people empowered to build for this platform? If not, this becomes another plugin directory with the same companies at the top.
Opening access for developers to build Muse connectors. You bring the API -- Muse brings the agent, the browser, and the context of what the person actually wants. People reach your service just by asking for it, and their agent takes it from there.
New connectors are live today. Come build with us. muse.ai/platform
5
1
18
5,892
Or we can just read the manual: docs.typesafe.ai/model-jagge…
I am going to build a jev app to classify which of all these demos are actually a good use case for jev.
2
383