I recently completed a research project through a highly automated collaboration with Codex. The division of labor was simple: Codex maximized exploration, while I focused on judgment. In detail, Codex proposed hypotheses, implemented and debugged methods, ran experiments, analyzed results, and iterated from feedback. I set the research direction, chose the questions, inspected the evidence, and decided which ideas were worth pursuing. This is also a concrete example of the research workflow I discussed in my previous post. I believe it is particularly effective when a problem is close to purely logical and each hypothesis can be verified with modest compute. Agents can then explore at high throughput, while cheap verification enables rapid iteration. The result is my new paper, which asks a question: what limits length extrapolation in recurrent models, and how can we learn recurrent dynamics that remain effective beyond the training horizon? More specifically, recurrent models are coming back not only because they are efficient. Their shared update rule provides a structurally natural path to length extrapolation. Yet learning recurrent dynamics that remain reliable over substantially longer horizons is still an open problem. To study this problem, I build a controlled task suite that separates state tracking, memory retrieval, and their interaction. My diagnostics show that vanishing parameter gradients (previous focus) alone do not explain extrapolation failure. A fixed transition rule can still be learned from dense local supervision despite severe gradient decay. The harder failure occurs when future losses cannot transmit sufficient credit to earlier recurrent states that must preserve useful information. This suggests intervening before the parameter gradient is formed. I call the gradient of future losses with respect to an earlier recurrent state as state credit. It is the upstream learning signal: a future loss must first reach the state before it can train the recurrent update. This motivates Credit Stabilization through Time (CST) to act directly on this signal. During backward propagation, it restores the norm of state credit when contraction accumulates across sequence chunks, while preserving its direction and leaving the forward computation unchanged. The results support this hypothesis across two regimes: Synthetic tasks • CST significantly improves length extrapolation across all task families and evaluated settings, with gains verified at up to 128× the training length. Real language modeling (still toy) • I train a 123M-parameter recurrent language model at 4K and evaluate it up to 128K on both in-domain and out-of-domain data. • CST produces a modest improvement over all tokens, but a substantially larger gain on key tokens that require long-range context. • The benefit generally increases with extrapolation length. Learning Length-Extrapolatable Recurrent Models Paper: arxiv.org/abs/2609.09157 This is my first paper centered on model architecture design and language modeling, and also my first attempt at this kind of highly automated research workflow. So comments and corrections are genuinely welcome. Hope you find both the paper and workflow make sense :)

Interested in how frontier labs pre-train image/video generation models? We were too. Since those recipes are rarely made public in full, we started from the most mature pretraining playbook available in the open: how modern LLMs are built. Introducing Chimera: a visual generation model family that brings LLM-style hybrid linear attention and scaling co-design to visual generation. In large-scale pretraining, nearly every design choice eventually shows up. That means solving architecture design and scaling as one coupled problem: every architectural choice changes how the model scales, and scaling behavior determines which choices actually survive. Chimera approaches both jointly, building a model family that remains predictable as model size, compute budget, and data distribution change. Our key architectural observation is a simple division of labor: a single raster-ordered KDA stream carries long-range state, while modality-aware short convolutions preserve native local geometry. Together, they form an effective and elegant linear-attention backbone for multidimensional visual data, with periodic MLA providing direct global interaction and sparse MoE expanding capacity at controlled activated compute. This design comes with a useful effect: NoPE. In Chimera, position is represented by the computation itself. KDA’s ordered recurrence and learned state decay encode order and recency, while the short convolutions encode local spatial-temporal structure. Explicit positional embeddings are not needed in our design. Because these mechanisms are not tied to a fixed training grid or sequence length, Chimera shows strong zero-shot extrapolation in both space and time. Trained exclusively on 1K images and 5-second videos, it directly generates coherent 4K images with little visible quality degradation and 30-second videos with only 6.5% FID degradation over the final five seconds, all zero-shot, without resolution- or length-specific finetuning. But architecture alone is not a pretraining recipe unless it scales predictably. Thus, scaling should not be treated as an afterthought: fitting a curve over model sizes is easy; making that curve meaningful is much harder. If every model size is differently under-tuned, your scaling law may simply be measuring optimization error. We propose HeteroP to transfer proxy-tuned hyperparameters module by module across width and depth, giving us a consistently tuned model family. This allows us to fit Chinchilla-style laws over activated model size, training tokens, and the image-video data mixture. The laws not only provide the reference for compute-optimal model and data size, but also suggest that visual generation may be more model-hungry than we tend to assume. Under the same parametric loss-fitting method used in Chinchilla, compute-optimal model size grows as FLOPs^0.516 for images, compared with FLOPs^0.46 for language. Video is even more model-hungry, with the exponent rising from 0.516 for images to 0.544 for video. Guided by these laws, we trained an 11B-parameter Chimera that activates only 2B parameters per token, using ~600 H100 days. - It matches Wan-2.1 2B pretraining loss with 7.3x fewer FLOPs, and runs 2.14x faster than full attention at 255K tokens. - It matches FLUX.1-dev and Z-Image-Turbo on GenEval and outperforms both on DPG-Bench, using roughly 20× less training compute than Z-Image-Turbo. Ultimately, Chimera indicates that once you pretrain at scale, every decision shows up. And if the Kimi K3 recipe caught your attention, Chimera may look oddly familiar, except the tokens are pixels and frames with diffusion models. A team effort from team @ChongjianG30781 , me, @VisionSteve , Jiuxiang Gu, @Xu_Arthas , @chenziwee , @ShaotengLiu , @Jingorz , @YicongHong , @Zefan_Cai , @HaoTan5 ; supported by Hailin Jin and @kalyank_s at @Adobe @AdobeResearch