Nick Stathas retweeted
I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4.
OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its training distribution. I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it's a core goal of our current research program.
enjoy unlimited yapping!
Nick Stathas retweeted
Super excited about our new research direction for aligning smarter-than-human AI:
We finetune large models to generalize from weak supervision—using small models instead of humans as weak supervisors.
Check out our new paper:
openai.com/research/weak-to-…
Nick Stathas retweeted
human simulator / imitation saliency problem explained: one very natural generalization is to just say ~what a human would say. if this is more natural than what the human would say if they knew what the AI knew, then it will systematically hide things humans can't understand










