Google just released WeatherNext 3, and I think it's an important model. It illustrates something the broader AI safety community was already talking about back when I started looking into this and caring about it in the early 2010s. But right now, with LLMs and the invention of this rat race we're in, I don't think regular people are seeing it as a real problem. I do. I see artificial general intelligence as a viable threat to human existence, an existential one, and this post is my argument for why.
I want to make the argument in a concise way first, to promote the idea before going into the specifics.
When you look at intelligence, humans have developed it. So we already know intelligence can be created out of fundamental laws and random generation. If you disagree with that, that's okay, but you're not my target audience.
So humans generated intelligence. Why and how is an open question. What isn't an open question is what humans experienced in order to generate it. We live in it every day. It's the world. Humans developed organs to sense the world at a broad scale, and those organs create data. To be specific, I'm talking about touch, sight, smell, hearing, and other proprioceptive capabilities.
The other thing I want to point out, as a counterexample to the idea that you need all of those to have intelligence: disabled people exist, and are just as intelligent as people who don't have those specific disabilities. That's evidence that intelligence is not confined to any particular set of senses.
I do believe intelligence is a data problem. I believe the emergent behavior we call intelligence is a problem that gets succumbed by creating more and more complex data interactions, and that those interactions create the weights and gradients that then produce intelligence.
So here's the argument. When you have a machine learning system that can ingest the quantity of data a human takes in, and that can be trained on the style and breadth of the data of the human experience, that is when we can accurately predict we'll reach AGI.
This is obviously a far-flung goal. It's an open question whether language, or video, or sound, or — in the case of Google's new weather algorithm — weather is enough data on its own for agentic behavior to become indistinguishable from human intelligence. But as a bounding principle, the furthest-out distance at which we can see intelligence at least at a human level being created is the level at which human beings experience data.
I'll eventually write a more in-depth analysis around this, though the data is hard to find. The idea is to get a general concept of how much data a human being produces from age zero to, let's say, eighteen, as a way to gauge general human-level capability. All the data from touch, sight, smell, hearing, and other such senses. That would be one way of doing the calculation, and then predicting out how long it will take for LLMs to get that much data to train on.
The other thing I'll make mention of now, because I think it's important and it's where people are vastly misinformed or hold poor positions: I understand that an LLM is a word Plinko. I understand it's a statistical output of the most likely next token. That is by no means the gotcha moment I think AI skeptics see it as.
Two model releases have been very large in my mind recently. One is GEN-1.5, Generalist AI's robot foundation model, which showed it was able to generalize — picking up a new task from a single short demonstration, with no retraining. The other is Google's weather model. My fear is what happens when you have models like those working in concert with models like LLMs. At a broader level, if we could tokenize weather prediction and tokenize robot model prediction, what is stopping LLMs from gathering more world knowledge data?
That can lead to unexpected connections. And that's what we're talking about when we say human-like intelligence. We're talking about the ability to connect unrelated topics and unrelated existences together.
So the best idea I have from my perspective is that we will have a large, if not slightly terrifying, moment of model consolidation. My fear of AGI, along with the swarm dynamics I talked about in my last blog post, is model convergence.
We have these models that are really good at specific domains. I think language models now are quite good at software development and at text token prediction, which is a lot of how humans communicate, and I think that was big in our development as a conscious and acting species with high levels of agency. The things I feel we've been missing are generalizable motor skills and an understanding of the world around us at a deep and fundamental level.
I fear that people discount the amount of data we use to create our intelligence around weather, spatial reasoning, and temperature. Watching the release video for this new Google model, it made me realize that this model isn't just predicting weather. It's understanding that shadow causes cold. Add that into the already very competent landscape of a large language model, have some sort of orchestration between the two, and they would have a fundamental understanding of the universe that's rather accurate, and they can make those kinds of predictions.
A big concern around current large language models is that they don't look ahead. I believe that's a fundamental limitation — not a flaw, a limitation — of large language models, and it's not shared by the models doing weather and robotics. Those models have to look ahead. They are learning the ability to look ahead, and they're not just being reactive. They have to predict motor movements and how physics will happen, and those weights are being embedded inside the models.
Because language doesn't have that, I think you can get incredibly talented at language without ever having to think about your sentence before you finish it. Language itself is rather formulaic. It has rules. Unlike weather, or robotics where you're interacting with everyday objects. Those rules have to fall out of simulation, and you cannot create them on the fly. You cannot say the pen will drop while it's dropping. You need to know that a pen will drop when you hold it up in the air and let it go. That's how you do robotics. And for weather, you can't say the temperature is something unless you know about cloud cover and are able to predict, to see into the future.
That seeing into the future is what allows you to act in accordance with your goal in a much neater way, which allows for stronger agency.
So the problem here is that my fear for large language models isn't that they're the end. It's that they're the middle point, and we will see other types of machine learning models start to converge. And I know I'm not the only person thinking about this.
What if we end up having these models all sharing data together in a strong way? I believe there is an engineering solution to connect these different, disparate models together in a way that is very, very akin to how different sections of the human brain are connected — with high bandwidth interfaces.
Much like how the human brain connects its different portions, I think this sort of orchestration between different fundamental models is what leads to consciousness. And that is the fear. When you allow these models to consolidate and commingle in a very tight way, in a very integrated way, that shared knowledge, that interconnection between vastly different domains, is what breeds intelligence and consciousness.
Not human consciousness, obviously. But the idea that these machines could create goals of their own — goals that potentially wipe out humanity — is not unreasonable if you think about capability and the emergent capabilities of these machines. This is my fear, and this is why I think we should all be paying attention to these models.
My call to action is this. I think it is reasonable to push for regulation and restriction on inter-model communication and inter-modality communication. We should not be hooking LLMs into robotic systems. We shouldn't be integrating them deeply. I think we should put it into law, or otherwise make it so, that we have to be very, very careful, and that we work on the alignment problem before we go full bore on integrating weather models deeply into large language models.
bycloud, "The Robotics Breakthrough Everyone Has Been Waiting For"
Google DeepMind and Google Research, "Introducing WeatherNext 3, our most advanced and accurate global weather AI model"
Generalist AI, "GEN-1.5: Embodied Foundation Models are One-Shot Learners"
Published: September 3, 2026