When Palantir is the only company earning positive rewards in your RL environments

I’ve worked on a number of risk and compliance implementations across financial services, food safety, automotive, aviation, oil and gas, heavy construction, and healthcare amongst others. I think it’s important to understand how they work, or not, a level down as they are referenced to justify current theatrical efforts. Risk is an excruciatingly difficult thing to manage. It cannot be done with theatre and hypotheticals. Insurance is a ~$10 trillion industry for a reason. The advocates for this theatre in AI are failing any objective evaluation in their business practices and culture today, perhaps because they are monetizing the actual levered methods they could edit to control risk with their current business practices, so they have to concoct new methods that ignore the low hanging fruit. You don’t get to choose convenient areas of risk if you have a rigorous safety culture. As an outsider it seems the advocates of risk reduction are choosing very convenient definitions of risk (and benefits), pushing them to be undefined and in the future. In the present, they are struggling to deliver two 9’s of uptime. They have paid billions of dollars in fines for the compromised integrity of their data supply chains already. They have been permissive in their enablement of distillation attacks through ineffective KYC controls. They have instituted paper thin controls in their offensive cyber testing environments. They have regularly been at odds with one time partners when they launch competitive products. That is clearly not a culture that takes the stairs one step at a time with a hand on the railing. The necessary but insufficient reason to rely on Sovereign AI is generalized controls will not protect individual use. Of course the better a system is controlled, the more valuable it is as well. Patterns that are consistent across all of the above industries: - Safety is cultural. Oil and gas and mining companies who have visceral understanding of the risk of the environments they operate in will open every meeting with a safety moment where a participant will share an anecdote like talking about their child falling on their bike. They will enforce norms like you must take the stairs one at a time and hold the railing with one hand at all times. They install cultural norms because they have learned that efficacious controls require consistency across all business practices, not convenient ones. - Risk outpaces controls. Risk is an adaptive adversary; either literally with the ingenuity of the criminal, or as a function of the entropy in any complex physical system. - Every general control trends to a 99%+ false positive rate. The simple way to think about it is static logic adjudicating a dynamic world means even hyper-tailored controls drift in their impact over time. Think how many times you have pulled from a TSA line for secondary screening. - General purpose controls that are at odds with environment specific dynamics will create substantial friction and that friction will lead to business practices that work around the control, leaving them uncontrolled. Think refusals for doctors doing biological research. - Compliance programs focus on what is knowable. This means they crave control of inputsand not just outputs. The tightness of the causal link between input controls and real world outcomes defines the efficacy versus cost of the program. - Provenance is a critical pillar in domains like financial services, aviation, and manufacturing. A bank has to be able to substantiate Source of Wealth and Source of Funds. An airplane has to have serialized tracking of a part in order to trace potential causes of an issue. A manufacturing company has to assert that none of its suppliers has used sweat shop labor. - Compliance is organizationally exhausting. Installing a process that is hyper attenuated to the thousandth straight false positive is a very difficult systems design challenge. - Human power dynamics introduce substantial entropy. The Korean Pilot case study extrapolates across many domains. Relative talent, power, and understanding have to be modeled both cooperatively and competitively which is hard. Why this matters for AI: - While the intention may be to control unsubstantiated future risk, any actual implementation of a program is going to be limited by what we can actually control for. The more this relies on creativity of potential risks, the wider the collateral impact on benign use will be. If it is untethered from what is measured, it will become a projection of a belief system that is unfalsifiable. Like wiping down your groceries during Covid. - Risk management will create safety theatre not safety outcomes the more it is done at a general level. The application of the technology in specifically controlled contexts will be dramatically more effective, measured only by success in controlling risks, than any generalized logic. There is both enormous deadweight loss in universal logic applied to specific outputs, and more importantly if you are singularly focused on risk reduction, enormous false positives that lead to exhaustion that lead to knowable bad outcomes. - The natural consequence of this for any motivated and empowered overseer is that the cost of controls will also move to the edge, i.e. specific customer use. - Provenance. The supply chain for AI is the data and RL environments. Opening the kimono and establishing rigorous controls on what inputs are used in models is the obvious place to start. I suspect this is the last place those advocating for controls will want to start, but it seems like the place with the highest impact with most control for a motivated overseer. - Banks have very strict KYC controls, weapons are subject to ITAR. If you were a startup bank that was facilitating terrorist financing or sanctions evasions before you could mature your compliance program, you would be sent to jail. There seems to be low hanging fruit if you are optimizing for risk reduction and I’m curious why programs that aren’t sufficiently onerous for the stated risk haven’t been implemented already. - The asymmetry in expertise will lead to psychological pressures, as this whole fiasco and the herding behind it is already clearly demonstrating. Let’s spend our time arguing about efficacy not “should or should not.”