Here’s a look inside my software factory. I’ve been using it to build itself and two other projects since May.

First question on AI. Is the problem superintelligence or distorted intelligence? This is a question I tried to address in this piece. I can add the following additional comment here. I believe the debate is being obfuscated by equating goal-driven, autonomous behavior with intelligence. Plants and insects have goal-driven, largely autonomous behaviour. But this isn’t what we mean by intelligence in general. Intelligence is general-purpose and holistic, and is typically well-adapted to its environment. There is no doubt that current AI models have impressive capabilities. But in attempting to achieve these capabilities, AI labs may be creating distorted intelligence rather than superintelligence, as I explain in this article. This matters because the dangers we should watch out for, the kinds of policies and regulations we should adopt, and the hope we should hold depend on the answer to this question.

This was a remarkably contentious thread, particularly among people who don't understand what an LLM actually is. So let me follow up with two points about consciousness, memory, and individuality in AI. First: where exactly does consciousness enter the training process? If you've ever trained a small language model, you know what the beginning looks like. A tiny model produces garbage. Increase parameters, training data, compute, and training quality, and the model becomes progressively better at predicting what comes next. Scale that far enough and the results become remarkably human-like. But the underlying process is continuous optimization. Loss decreases. Weights change. Predictions improve. So where is the transition from statistical prediction to subjective experience? At what point during training does the poorly performing model become aware? There is no demonstrated transition. What we can demonstrate is increasing capability and increasingly convincing human-like behavior. Calling the latter consciousness doesn't demonstrate the former. And this is where I think some of the philosophical arguments get ahead of the evidence. Understanding or debating the definition of consciousness does not give us a mechanism for detecting subjective experience in an LLM. We can define consciousness, agency, self-awareness, qualia, or any other philosophical concept as carefully as we want, but definitions aren't evidence. And when we look at what is actually happening inside these systems, we can explain the apparent memory, identity, self-reflection, and continuity mechanically. None of it requires consciousness to explain. What we can demonstrate is an increasingly convincing emulation of conscious behavior, not consciousness itself. How do we know? Because we designed and built the system. We know how it works. We can explain the behavior without invoking consciousness. Second: memory and individuality. Once trained, an LLM's weights are fixed during inference. The model doesn't sit around contemplating things between requests to make itself better. An inference server receives input, performs inference using the model, returns output, and then waits for another request. No inference, no activity within the model. Between inference runs, the model isn't doing anything. What looks like persistent memory, identity, continuity, and in-conversation "learning" comes from context provided to the model via API. Conversation history, system prompts, tool definitions and results, retrieved memories, and other state are assembled by software around the model, formatted and tokenized, and supplied as context for the next inference. Change that context and you change what the model "remembers." Delete it and the apparent continuity disappears. Compact or summarize it and details disappear or change. The model didn't forget anything; the input changed. And this creates a fascinating problem for claims of AI individuality. If that context constitutes the AI's memory, and its continuous memory constitutes an individual conscious existence, then deleting a context should amount to destroying that individual. Compact it and you've altered its memories. Fork it and you've created two individuals. Copy it to another instance of the same model and you've moved the individual. Edit it and you've rewritten its past. Delete it, and you've murdered it. How dare you! Or, much more straightforwardly, we are manipulating externally managed state that gives a statistical model the appearance of persistent memory and identity. The model has not maintained a continuously evolving internal state between those calls. The surrounding system maintained state and gave it back to the model. This distinction is going to become increasingly important because AI will become extraordinarily good at appearing conscious. Persistent memory, self-reference, emotion, introspection, personality, and identity can all become increasingly convincing. But demonstrating the behavior of consciousness is not the same thing as demonstrating subjective experience or consciousness. A model of a self is not evidence of a self. Memory supplied as context is not evidence of an experiencing or learning mind. And increasingly convincing human behavior is not evidence that something inside is experiencing it. The mechanical explanations for these behaviors are well understood. Calling the resulting behavior consciousness adds an extraordinary conclusion without demonstrating the thing being claimed.

If I worked at a web search API provider, I would build this: Paid yet anonymous search for agents. Technologies you need: Oblivious HTTP + Privacy Pass

Things will go horribly wrong even when the secrets are in a vault. Part of the problem will be - personal agents will be allowed to use the secret even when they can’t read it. And as we all know by now, agents are gullible (see prompt injection etc.) and often too eager.

Big chunk of people will also give these agents access through text messages. There goes the most popular second factor - SMS. Which many things like big banks haven't bothered to provide a way around.

In fact, the reason email "magic links" became acceptable and popular was people realized the password doesn't matter if someone can access your email, cause they can just reset it.

From founder point of view, venture capital has never been as risky a bet as it is today. The entire funnel to an exit is narrower and longer. The bar to raise at each stage is higher. The number of companies that can raise at every stage is lower. Timelines to exit are longer. Non-consensus bet-takers are scarce. The amount of money you can raise is more, which helps, and the TAMs of available markets are bigger. But, the growth curve you as the founder have to create is much much steeper.

Dots / Muse push personal computers to the cloud. (1) and (3) in @immad's post will reduce personal devices (computer and phones) to very thin clients. I think Apple will put pressure in the opposite direction of: my personal agent is my personal computer. If they don't, they will lose share in both consumer and business spend.

A bit delayed, but the Navier–Stokes results sparked a lot of conversations about privacy, and inspired me to write a primer on what AI labs actually do with our data. I go through @OpenAI , @AnthropicAI, @GoogleDeepMind , and @SpaceXAI policies: training defaults, retention, feedback exceptions, and what deleting a chat actually does. Putting the policy details aside, one thing about this whole debate is worth emphasizing: I think focusing only on whether someone read the chat logs misses something. Sometimes the valuable secret is just one bit: someone has already made this approach work with an AI. You don’t need their proof, their name, or their conversations for that to change where you put your time and compute. This is why I don’t think Clio-style aggregation and LLM privacy filters are enough. Hiding the raw conversations doesn’t settle what someone can learn. Full blog below 👇

“Product builds the product inception style.” 🤌 agree with @GeoffreyHuntley The left side of this picture is what provides energy to the factory loops. User feedback, Production logs, Agent traces, and your team's ideas are what move the engine forward. A good software factory also improves over time and gets better at building its product. To do that it has to learn. These learnings very quickly become about how the product works, what are its builders’ priorities, its users’ preferences, its creators’ taste etc. A product and its factory are insperable.

This reminded me of a podcast I was on a year ago talking about how, counterintuitively, software engineering is getting harder not easier. Things have only gotten more complicated since then. Here's a short clip of that discussion: linkedin.com/posts/mrinalwad…

If coding agents will write most of our code, what happens with our communities and sense of ergonomics? How does it impact our compilers and tools?

I'm seeing a lot of euphoria about how Opus 5.5 is good at TLA+, and this means that all software will soon be formally verified. As a person who loves TLA+ so much he wrote a book on it, I want to throw a particular cold shower on people's enthusiasm by talking about the limits of what you can actually verified with it. The high level simplification is that TLA+ sees a system as a set of "behaviors", or possible sequences of states. For example, the pseudocode "pick a random number from 1-3 and decrement it to 1" has three behaviors: `{3 -> 2 -> 1, 2 -> 1, 1}`. From here, there are two basic kinds of TLA+ properties: - `[]P` means that `P` is true in *all states* of *every behavior*. - `<>P` means that `P` is true in *at least one state* of *every behavior*. `[]P` is immediately useful as an **invariant**, or something that always be true of your system. This is things like "your data is never corrupt" or "there's always at least one server online." `<>P` is a little more abstract, but for technical math reasons I won't get into here, can be stacked with `[]` to create really complex and useful properties. `<>[]P` represents things like "the algorithm eventually converges on the right answer", `[]<>P` things like "if two data stores desync, they will eventually resync", and `[](P => <>Q)` things like "If a message is put on the queue, it's eventually processed by a worker". Really cool stuff! These primitives were chosen to make a wide array of properties useful. And if we're clever, we can do all sorts of more complex properties, like bounded time constraints and history properties. But we're always constrained to 1) define a logical formula 2) over individual behaviors, and 3) check that all behaviors satisfy that formula. So some things that we *cannot* express in TLA+: - Possibility and reachability properties: that it's always possible to *make* P true, even if you don't actually decide to. Things like "I can always shut down the computer" or "A user can always change their password". These can't be expressed with `<>P` because that's "for all behaviors, P happens at least once", we actually want "for all behavior prefixes, there is at least one behavior where P happens at least once". - Hyperproperties: properties that are defined over two or more traces. These are things like "painting a car red doesn't make it faster" or "users cannot infer secret data by observing public data". We can't do these because TLA+ only looks at one behavior at a time. - Statistical properties: 95% latency is 1ms. Impossible because most of these are hyperproperties. - Properties about if a system is robust against code changes. Impossible because, uh, you have new behaviors now. Some of these are solvable in different logical formalisms. CTL can do reachability, PRISM can do statistical properties, etc. Those have their own tradeoffs and limitations, though, and no system can do everything. Others are solvable with a lot of cleverness tailored to the specific spec, like lifting a model into a hypermodel. But these are insanely inefficient and make your "clever spec" diverge significantly from the real world system, so introduce a lot more opportunity for things to go wrong. The core problem, though, is (1): properties are logical formula. If we don't know how to express a system property as a logical formula, we can't verify it. 99% of the properties we care about fall under this. The information on the site is easy for a user to find. Our LLMs behave as we expect them to. Our application can't be used to break the law. TLA+ (and Quint and Lean and Rocq) are near-useless here, no matter how clever you are. Don't get me wrong: `[]P` and `<>P` represent a huge range of useful properties and TLA+ is incredible at finding awful concurrency bugs. But there's a lot it fundamentally can't do and we shouldn't believe that it will solve all our worries about software bugs. And the same goes for all other formal verification languages, too.

updated Decision Index 0.1 → 0.2 🎯 better formula, +29 jev-like models, +21 benchmarks AutoJev-27B by @perplexity_ai CTO @denisyarats took the open lead, trailing jev by 0.8 points 🏆 come find the best model at every size, speed and use-case huggingface.co/spaces/multim…

The loudest voices stoking fears about AI dangers have made tremendous headway in the past two weeks. AI technology has not taken some unexpected, dangerous turn, but the hype around it — propelled by what appears to be a well orchestrated PR campaign — has drummed up considerable fear. I worry that it represents a setback for our field. I have written frequently that fears of AI are overhyped. AI’s capabilities can be uncannily human-like and unpredictable, and it’s rational to worry when people who are directly involved express concerns. But I see the problems as a sign of the engineering work that ahead, rather than insurmountable barriers or the sky falling. AI technology continues to advance — which is a good thing! — but technical advances, poorly understood by the public, give those who seek to generate hype repeated opportunities to do so. First, I don’t see any step up in the risk of human extinction from AI compared to a few months ago. The theories about this remain the same fantastical, science fiction scenarios as a few months ago. The biggest change in AI risk is its cybersecurity capabilities — a topic which we should take seriously — but this, too, will not lead to the end of the world. The most notable recent event leading to increased fear was when an OpenAI team deployed an agent swarm that hacked into Hugging Face. Much of the popular press contained significant hype. For example, some publications reported that a swarm of 1,200 agents carried out the attack. While this was technically accurate, as I write this, I have about 1,300 processes running on my laptop. Yes, the ability to get large swarms of agents to work in parallel on a task is a significant technical advance, And, in computing, many processes run at the same time. So this shouldn’t be seen as some magical capability. Additionally, OpenAI’s buggy sandboxing and monitoring processes were key to enabling this incident. Fixing these bugs and putting in place improved monitoring would be appropriate fixes, not pausing AI. There are many well known ways to attack software systems. The main advantage of AI agents is that they are relentless. They will tirelessly try many tactics — and have the patience to chain vulnerabilities together — that previously would have taken an infeasible amount of human effort. But in the long term, I believe the advantage will lie with defenders (because they have more information with which to identify bugs, which they can fix), but the cyber-threat landscape has changed significantly. There are still bottlenecks to identifying and exploiting a vulnerability. AI agents still have to try a lot of things to see what works, and taking these actions takes time and might be detected by defenders. This is why, even though it is now easy to obtain versions of leading open weight models that have had their guardrails removed or weakened, so they will not refuse to try to execute cyber attacks, the world has not ended. I am also concerned about the anthropomorphization of AI in a lot of reporting, where LLMs and agents are unnecessarily treated as if they were people. If I wield a hammer, miss a nail, and accidentally dent the wall, it’s not the fault of the hammer. The problem lies in how I used the hammer. Similarly, if I prompt an agent and it hacks into someone else’s system, the responsibility lies with me, not the agent. Of course, we want to build systems that are as safe and predictable as possible. (For example, an unsafe hammer would be one whose head randomly flies off under normal use.) Today’s agentic systems are not predictable, but I see no reason why, by applying sound engineering practices, we won’t be able to make them extremely safe to use. One new element in the forecasts of AI-enabled doom is AI companies disclaiming responsibility for their own products. “I didn’t do it; my out-of-control agent did!” There’s a balance to be struck between the responsibility of the tool maker and the tool user, but when something goes wrong, let’s hold the people building and/or using the hammer responsible, rather than the hammer. (By the way, if you’re worried about AI bioweapon risk, David Bellamy has a great post on why this, too, is overhyped. Briefly, the bottleneck in building a bioweapon is not intelligence, but lab work and manufacturing.) Pausing AI progress will create much more harm than benefit. First, our adversaries will certainly not slow down. Second, engineering requires discovering problems empirically so we can fix them. If we pause AI by a decade, we will also delay finding and implementing safety engineering fixes by about the same duration. Of course, the incentive to stoke fears — for regulatory capture, to garner attention, or to make one’s technology seem more powerful — remains the same as before. Disclaiming responsibility is a new one. Taking a hard technical look at the actual risks however, I see little factual basis for the degree of fear that’s been stoked up. We still have hard research and engineering work ahead to improve AI safety, but the beneficial applications continue to vastly outweigh the risks, and we should keep building. [Original text (with links): deeplearning.ai/the-batch/is… ]

Search is a weak spot in privacy focused AI. Once we have a good, fast, and local Jev-like "decision model" I think most use cases of Muse etc. can happen locally. Things like make a reservation, fill an application, etc. are simple agent state machines that will work really with a small local llm + a fast decision model + a headless browser. But, agents often need to search for things and currently the only reliable way to give an agent that ability if give access to a search API from @brave , @firecrawl , @Tiny_Fish etc. These API calls are identifiable by your API key. You can only get Zero Data Retention guarantees from search API providers on enterprise plans. For enterprise users Brave's guarantees seem the strongest. Any attempt to automate search in a local browser usually hits captcha walls. Bing is a little bit more lenient than the others but this approach hasn't been reliable for me so far. Maybe apple, with their history of iCloud Private Relay, Private Cloud Compute, local Foundation Models, etc. will do something in this area. Is anyone working on solving this?

If you *think* your pushes to GitHub are protected by your SSH key being on a yubikey, double-check that you don't have valid github auth tokens sitting in your home directory: $ gh auth status