Another example of what I find to be two very passionate AI communities taking past one another. The original assertion by Bender/Gebru/McMillan-Major/Mitchell holds: any internal structures we find inside a model are not grounded in the model's own perception and action and both rather are only learned from the patterns found in the linguistic forms that describe those worlds. As such, they reflect nothing more than what are the self-consistent patterns found in those linguistic forms. What Pierre and Natand others are talking about are compressed latent structures which are indeed present. But, using Plato's allegory of the cave, I would suggest that these artifacts are more like the flickering of the shadows from the light: there's a pattern, it's imprecise, and it's not directly grounded in what the world really is. And all that being said, transformer-based models remain next token predictors. The fact that they can yield surprisingly coherent results is splendid and useful. I'll return to something @aran_nayebi observed and so which I fully agree: alignment is important, because the shadow and the light are different things. But the open question remains as to if this is possible at all with our current architectures.