Joseph Low

Multi-agent alignment is basically just political philosophy, right?

I want to echo Nathan's comment here. Completely understand these decisions by many people I very much admire, and think that they will indeed continue to do great work. But it's just plainly obvious that working for a lab in this way involves a significant sacrifice of independence, for your ability both to publish critical results, and to select the research area that will be most challenging for your employer if it generates results. And frankly for your ability to even recognise those research areas as worth pursuing: every company (or community for that matter) has a culture and it invariably shapes what counts as a good research question. You can't avoid that, setting aside whatever other motivated reasoning might derive from one's personal financial interests being intertwined with the corporation's. I think most people now recognise this is a problem. The question is obviously what to do about it. First, while the labs have made a good start at putting out funding calls for public-interest research, I think they should do much more, and they should completely loosen the reins. They shouldn't direct how this funding is spent, beyond ensuring that it isn't wasted. Second, they need to make more of their data and models available to public-interest researchers. If what you really care about is the AGI transition going well, you shouldn't force such a hard trade-off between independence and access. Many researchers will go where the data is, leaving all other considerations aside. So, build the structures necessary to ensure that independent public interest researchers can get that kind of access, and then reduce gatekeeping to the minimum required to ensure that serious work is being done. Third, it's definitely not all on the labs. One reason this is happening is that universities have dropped the ball so badly on AGI. Universities suck at multidisciplinarity, are ideologically conformist to the point of monoculture, and have had their heads stuck firmly in the sand about the AGI transition. Many researchers look at the prospect of working in an intrinsically interdisciplinary community, with at least a different ideological mix, and with eyes firmly on the prize, as heavenly by contrast with academic life. And there's more. Tenure is great for academic freedom, but creates one opportunity after another for gatekeeping. Universities compare themselves with other universities, so want only to be the best *university* doing X, not the best research organisation full stop. There is *so much* bureaucracy, about everything. Competing on salary is hard even when the funds are available, due to spurious equity concerns. I think we should view this as a serious call to action. Some of us have been arguing for years that universities need to change in order to actually help navigate the transition to powerful AI. The hour is getting late. But we can still do a lot better. At JHU our group focused on Governing the AI Transition (with @ghadfield, @nickacaputo and, next year, @zhitzig) has already solved for multidisciplinarity, ideological diversity, and being squarely focused on powerful AI/AGI. Universities are cumbersome beasts, but in our new school of government and policy we are determined not to be held back by ossified bureaucratic constraints. We will build a group of both tenure-line faculty and senior research scientists who can steer an ambitious agenda across all of the societal impacts of AGI, with genuine independence. We want to contribute everything that a university can uniquely contribute to this space—in particular, both fundamental research in the public interest and *training*, which is another key public good that this kind of privatisation of research undermines. But we also want to work very closely with other orgs, whether universities or non-profits, who care about making the AGI transition go well. And while we will insist on total independence from the AI companies whose societal impacts we are investigating, we welcome collaborations that preserve that independence. So: this is both a call to researchers out there who are eyeing the labs because they think that universities can't do this kind of research: we're going to be hiring, soon, in both faculty and research scientist roles. Hold on for a couple more months, and you can have your cake and eat it too (and while we can't approach lab salaries and equity, we will definitely be responsive to market conditions—and you get to preserve your independence). It's also a call to the labs: show your commitment to the AGI transition going well by creating more pathways for funding and data/model access to independent research orgs. I think the current, project-based approach to funding this kind of work is both bad from a metascience perspective, and gives too much control to the funders. That should change. If there are specific institutional bottlenecks that are holding you back, let's talk about how to resolve them, and what kind of ecosystem-level infrastructure is needed. And it's a call to the other groups, at universities and non-profits, who care about public interest research on the governance of powerful AI. We need to start working together, cos if we present a united front then it'll be much harder for the labs to say "no". I am doing everything I can to meet everyone with a footprint in DC; I'll visit SF soon as well. Please reach out to me (or my JHU colleagues) to talk.