I agree. Ring attention and other structures will smear the compute so that it will scale to whatever task is needed. Mixture of experts and mixture of everything else will be used to assign components of a task to smaller more efficient models.

I think agentic computing will take us most of the way to where we want to go. I think AGI is purely structural and can't be obtained by giving models ever more data.

I do think though that the unexpected emergent properties that arise as models get bigger will result in the Oligarchy of AI that's forming to always try to get their models bigger. To suck in more stuff.

The fact that some folks are figuring out how to untrain models might let the larger models be fine tuned after training, rather than starting from scratch.

But to your point, inference will def be the big chew on compute because a lot of that inference will be to feed the SAAS that will be everywhere.

Given models are more accurate if you give them more time to chew on the answer to any given prompt, it is very likely we'll see a lot of results out of the models already trained.

So the Stargate data centre and SAMA's $7 trillion expectation is partly real estimates, and partly hype. The AI oligarchy want us to think that they have all the answers so that the investors will keep investing.

Meanwhile the average noddy won't know what's coming till their job disappears or they're told their new assistant is a robot.