I think the key problem with LLMs is that they have no grounding in physical reality. They’re just trained a whole bunch of text data, and the topology of the network ends up being moulded to represent the patterns in that data. I suspect that what’s really needed is to train models on interactions with the physical world first, to create an internal representation of how it works, the same way children do. Once it develops an intuition for how the world works, then it could be taught language in that context.