“Hey guys, I smashed the loom, we’ll stick to knitting by hand from now on”

you either die a capabilities researcher or live long enough to see yourself become an alignment researcher

The bitter lesson in 26 words: Don’t be distracted by human knowledge, as AI has been historically. Instead focus on methods for creating knowledge that scale with computation, like search and learning.

It's odd to be living through what feels like one of the most critical periods in human history and to feel all of the weight of it from the inside.

By the way - I think a valid (if extreme) take on GPT-2 is "lol you need 10,000x the data, 1 billion parameters, and a supercomputer to get current DL models to generalize to Penn Treebank."

Unfortunately my ideas are too out of distribution to be targeted by LLM psychosis.

Hypothesis, I think shame might help reduce reward hacking, esp for long horizon tasks It doesn't prevent shortcuts, but Gemini often mentions how shameful it feels when it violates the spirit of the requirements, so at least the actions are faithful to the CoT Curious to see sparsity/platonism of shame circuits as models advance

if you value intelligence above all other human qualities, you’re gonna have a bad time

the timelines are now so short that public prediction feels like leaking rather than scifi speculation

Meta presents Layer Skip Enabling Early Exit Inference and Self-Speculative Decoding We present LayerSkip, an end-to-end solution to speed-up inference of large language models (LLMs). First, during training we apply layer dropout, with low dropout rates for

Open AI presents The Instruction Hierarchy Training LLMs to Prioritize Privileged Instructions Today's LLMs are susceptible to prompt injections, jailbreaks, and other attacks that allow adversaries to overwrite a model's original instructions with their own malicious prompts.

Meta announces Megalodon Efficient LLM Pretraining and Inference with Unlimited Context Length The quadratic complexity and weak length extrapolation of Transformers limits their ability to scale to long sequences, and while sub-quadratic solutions like linear attention and

Google presents Mixture-of-Depths Dynamically allocating compute in transformer-based language models Transformer-based language models spread FLOPs uniformly across input sequences. In this work we demonstrate that transformers can instead learn to dynamically allocate

welcome to bling zoo! this is a single video generated by sora, shot changes and all.

Bits to get in the door, Atoms to scale up.

RT @apples_jimmy: The only thing that matters is AGI and ASI. Nothing else matters.

Excited to share a new paper showing language models can explain the neurons of language models Since the first circuits work I’ve been nervous whether mechanistic interpretability will be able to scale as fast as AI is. “Have the AI do it” might work openai.com/research/language…

NVIDIA reporting LLM use? "NVIDIA has detected that you might be attempting to load LLM or generative language model weights. For research and safety, a one-time aggregation of non-personally identifying information has been sent to NVIDIA and stored in an anonymized database."

Does anyone have a GPT-4 license I can borrow?

here is GPT-4, our most capable and aligned model yet. it is available today in our API (with a waitlist) and in ChatGPT+. openai.com/research/gpt-4 it is still flawed, still limited, and it still seems more impressive on first use than it does after you spend more time with it.

The timeless struggle between the people building new things and the people trying to stop them…