And this is part of where the EA/rationalist origin of the AI labs comes through. They're articulating a future that might have high "aggregate spreadsheet utility" but which isn't very relatable. And which is maybe is a bit wrong/naive about real human preferences.

It's not that hard to do it! In fact, if you ask the models, they do a pretty good job (e.g., much better health care, AI makes work less tedious, most people still work but for slightly fewer hours). The impression one gets from the labs sometimes is singularity/utopia or doom.

One political problem for AI labs is that they can't articulate an optimistic medium-term future that sounds broadly good for humanity without also sounding incredibly weird, having huge wealth/power disparities, or both.