Excited to share some core RL work for a change (e.g. not LLMs, yet)! We present: A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learning This paper is part review part synthesis of recent trends in model-based reinforcement learning. As you'll see in my forthcoming work, understanding modular RL approaches will be key to understanding RLHF. Onto the paper's content. What is objective mismatch? It is the erroneous presumption that two learning processes are numerically linked (the training of an accurate dynamics model and achieving a high reward on a task). In the paper, we identify the following groupings of research in MBRL: 1. Distribution Correction (policy and model shifts): shifting the learning of model or policy to state/actions that matter! 2. Control-as-Inference: While similar to distribution correction, this focuses more directly on return optimization as opposed to estimation as signified by the maximization of the likelihood of optimal trajectories 3. Differentiable Planning: Solving all control components with gradients directly! 4. Value-Equivalence (robust control and value-prediction): Robust control approaches imbue a pessimistic decision making process over dynamics models that are inherently optimistic, giving more calibrated results. Value-prediction tries to do distribution correction at control time. There is also substantial experimental analysis of the experimental approaches in recent work. The paper can be found here: arxiv.org/abs/2310.06253, credit to authors and friends Ran Wei, Anthony McDonald, Alfredo Garcia, and @RCalandra

You can now more easily save and share your MBRL agents & dynamics models in MBRL-Lib. I hope this helps build new research programs around re-using compute. Huge shoutout to @ShivaSujit for pushing this through.