Great work led by Jiale! NoRA builds on LoRA’s parameterization, with the down-projection matrix normalized. Even more surprisingly, applying this normalization only at initialization is already sufficient to obtain considerable performance gains. More details in spherelab.ai/NoRA/.

Our recent works, DiagDistill (ICLR’26) and XYZFlow (ICML’26), are built on a simple principle of progressively strengthening constraints. Less trajectory ambiguity → fewer-step generation. The key is to spend more compute early, when generation is uncertain. As context accumulates, later generation becomes easier and needs fewer steps. This simple idea is exactly leveraged across video chunks in DiagDistill, and across image patches / denoising trajectories in XYZFlow. 🔹DiagDistill: spherelab.ai/diagdistill/ 🔹XYZFlow: spherelab.ai/xyzflow/

🚀 Excited to introduce PEFT-Arena! This project grew directly out of the challenges we faced when evaluating existing PEFT methods. We argue that PEFT evaluation should not ask only what improves. It should also ask what is forgotten. For too long, parameter-efficient finetuning methods have been evaluated by downstream accuracy. But target performance alone is not enough. A method can adapt well while silently losing pretrained capabilities. PEFT-Arena evaluates existing popular PEFT methods through the stability-plasticity trade-off, i.e., how much the model learns on the target task, and how much general ability it still preserves after finetuning. A good PEFT method should adapt without forgetting and achieve a strong stability-plasticity trade-off. In our evaluation, we find that Orthogonal Finetuning achieves the strongest trade-off among all methods. Beyond task-driven benchmark scores, PEFT-Arena also provides geometry-based internal diagnostics in weight space, activation space, and interpolation paths, helping us understand not only which methods work, but why they forget. What PEFT-Arena has done: 🌟 Adaptation + preservation: evaluates both target gains and forgetting. 🌟 Trade-off frontier: reveals stability-plasticity patterns, with OFT often leading. 🌟 Geometry diagnosis: explains forgetting via weight spectra, activation distortion, and interpolation paths. Welcome to try PEFT-Arena and evaluate your own PEFT method! 🌐 Project page: spherelab.ai/PEFT-Arena 📝 Paper: arxiv.org/abs/2605.28819 💻 Code: github.com/Sphere-AI-Lab/PEF…

I want to share more about why we started building Orbit: spherelab.ai/orbit/ Orbit is an OFT-centric reinforcement learning pipeline designed for ultra-efficient post-training of large language models. In simple terms, Orbit allows us to post-train a trillion-parameter LLM on a single 8-GPU node. But more importantly, Orbit represents something much larger for us, as it connects our long-standing research to a practical system that can be used at frontier scale. One of SphereLab’s core missions is to develop a principled and unified full-stage training pipeline for large foundation models. We are especially interested in what we call the spectral scaling paradigm: the idea that the geometry and spectral structure of model weights should not be treated as incidental details, but as central objects in how we design training algorithms. This perspective has alreadyled us to develop a series of pretraining algorithms and systems, including POET, POET-X, and Pion: - POET: spherelab.ai/poet/ - POET-X: spherelab.ai/poetx/ - Pion: spherelab.ai/pion/ It has also guided our work on post-training and adaptation, including OFT, BOFT, OFTv2, PEFT-Arena, and OrthoMerge: - OFT: oft.wyliu.com/ - BOFT: boft.wyliu.com/ - OFTv2: spherelab.ai/oftv2/ - PEFT-Arena: spherelab.ai/PEFT-Arena/ - OrthoMerge: spherelab.ai/OrthoMerge/ Although these projects may appear to target different stages of the model lifecycle, they are all driven by the same underlying principle -- the spectrum and geometry of neural network weights matter. Preserving and controlling these structures can lead to more stable, efficient, and scalable training and adaptation. While creating these algorithms has been extremely rewarding, we take a step further to build practical systems from the ground up that can leverage the same principle to best scale our algorithms. That is why a small team like us still spent so much time and efforts to build Orbit. Orbit is only the starting point. We are excited to keep building systems that bring principled training algorithms to real-world foundation model development.

🚀 Excited to introduce POET-X, a scalable and highly memory-efficient algorithm for LLM pretraining. ✨ LoRA-level GPU memory, better-than-AdamW pretraining performance! POET-X finally marries training stability (from POET's spectrum preservation) and practical scalability (from our new implementation and CUDA kernels). POET-X can pretrain billion-parameter LLMs (eg., Llama-8B) on a single NVIDIA H100, where standard optimizers like AdamW run out of memory under the same settings. We carefully reimplemented every computation step of POET (arxiv.org/pdf/2506.08001). POET-X combines many small checkpointing and parallelization tricks. While each may appear incremental, together they dramatically improve scalability and reduce memory usage by over 70% compared to the original POET. The memory-efficiency of POET-X comes from the unique parameter-efficient reparameterization (where sparsity comes in) of the weight update rule. POET-X bridges this gap between parameter efficiency and memory efficiency. Code is now public. Feel free to try it! ➡️ paper: arxiv.org/pdf/2603.05500 💻 Code: github.com/Sphere-AI-Lab/poe… 🌐 Website: spherelab.ai/poetx #AI #LLM #MachineLearning #DeepLearning

Interesting work! Doing proper normalization is definitely important for training neural networks stably. We considered hyperball normalization for convolutional neural networks back in 2018, see arxiv.org/pdf/1804.08071. Besides hyperball normalization, we also proposed multiple other normalization methods for weight/activation. Quite surprisingly, we also did gradient normalizaiton in order to make it actually work. See Section 4 of the Decoupled Networks paper. I somehow got the impression that many many old ideas are worth revisiting for LLM pretraining, especially those that stablizes the training (but may slightly hurt the performance for conventional CNNs).

🤩 This is awesome. When we are doing the agentic design project (besiegefield.github.io) using the Besiege game environment, we have to hack the game to get as much feedback as possible to do RL and stuff. However, I start to think differently after seeing the Genshin agent. We humans don’t need that much feedback to learn to master the game, and visual feedback is already sufficient. I am wondering what will happen if the agent learns to master many games this way. Will it develop some universal skills for game playing? Will it see the world differently?🤔

🤯 Merging many finetuned LLMs into one model, effectively? Introducing Functional Dual Anchor (FDA), a new framework for model merging. 🚀 Current merging works poorly due to the underlying parameter conflicts. FDA shifts knowledge integration to the input-representation space for seamless merging. This "dual" perspective bridges the gap between post-hoc merging and joint multi-task training, reducing the knowledge conflicts. ✨ FDAs are synthetic anchors that precisely capture a finetuned model's functional shift. ✨ FDAs can complement existing model merging methods and achieves SOTA performance. ➡️ Paper: arxiv.org/abs/2510.21223 💻 Code: github.com/Sphere-AI-Lab/FDA 🌐 Project: spherelab.ai/fda #AI #LLM #MachineLearning #DeepLearning

The physics prior matters in molecular structures. We model potential energy between molecules for drug design. This happens to have a coincident yet interesting connection to my past work, hyperspherical energy (arxiv.org/abs/1805.09298), which considers potential energy between imaginary electrons (i.e. neurons in neural networks). But this time we are modeling real molecules for drug design. :) Excited that our new AI-for-science paper is finally online: "Manifold-Constrained Nucleus-Level Denoising Diffusion Model for Structure-Based Drug Design." Very glad to be part of the wonderful team. @Shengchao_Liu Caltech news: caltech.edu/about/news/new-a… Paper link: pnas.org/doi/10.1073/pnas.24… Project page: yanliang3612.github.io/Nucle…

This is almost a year-long project and led by @ItsTheZhen. My biggest takeaway is that physical simulation is very effective as a reward signal, and this efficient verification is crucial for unlocking LLMs’ design novelty. This conclusion is actually aligned with our previous work spherelab.ai/SGP-Gen, where the verification is done by a renderer.

🤖 Can LLMs learn to create? Introducing "Agentic Design of Compositional Machines" — a new frontier where AI builds functional machines from standardized parts. We present BesiegeField, a simulation testbed to benchmark LLMs on tasks like building cars & catapults. Key findings: 🔧 Compositional design is extremely challenging even for SOTA LLMs (human can easily do better) 🛠️ Multi-agent workflows + RLVR boost performance ⚙️ Physical simulation as a verifiable reward is effective for eliciting LLM’s design capabilities. 🧠 High-level planning ≠ precise execution — a core challenge Paper: arxiv.org/abs/2510.14980 Project: besiegefield.github.io #AI #Agents #LLM #GenerativeAI #AIGC #Simulation #RLVR