[Long off the cuff post reacting to this posting + the discourse in comments and quote tweets] This is really thorny. On one hand, I think it's clear the people involved here are earnestly interested in making this kind of thing have very pro-social outcomes (i.e. I think everyone involved really wants to improve Claude's ability to tackle grand societal challenges, and they might succeed). (And caveat, I really like everything that the OP has put out and think it's great this line of work is being boosted by Anthropic; while I'm very worried about the overall impacts of AI industry at large on content ecosystem, incentives for knowledge work, and so on, I think an underrated mitigating effect is that the labs really are boosting some impactful research agendas) However, on the other hand, I think there's a strong argument (and one that leans on, in particular, perspectives from political scientists and economists re: social dilemmas, market failures, labor solidarity, capital concentration) that there will at some point be an inflection point where an individual's choice to contribute to something like this may *substantially* accelerate replacement in a sub-domain. So depending on one's own normative preferences and "judgment of social impact" -- it could be the case that replacement effects, and especially the notion of defection or "scabbing" that impacts one's close colleagues by eroding the leverage they have via their previously leverage-generating skills outweigh other positives. It's possible we've already hit this inflection point. (And yes, there's a big debate about it being pro-social to accelerate certain types of replacement -- if they're going to lose that leverage soon no matter what, rip the band-aid off, so every individual needs to reason about how likely it is that collective action / labor coordination is achieved at all) However, with replacement concerns fresh in everyone's minds (because of math progress, other "capability jumps", etc.), it's notable that the AI industry does not really seem to have an answer for this yet. No joke, perhaps one of the outcomes of programs like this that boost social scientists could help provide an answer! I think there is a world in which top social scientists create RL envs that make Claude better at helping AI employees form a union, or help data intermediaries operate, make data markets healthier, etc.