🚀 Session 4 for World Model from Scratch book is on Thursday, October 15, at 9AM PST. We will look into Chapter 5 — ​Architectures of Modern World Models -> Plan: cover DreamerV1, V2, V3, and V4 in one session, focusing on the architectural deltas and engineering motivations between versions luma.com/euoop1sm?tk=WwNpnZ Session Overview of the third session for the book via @Cohere_Labs 🎙️ Hosted by @nahidalam, a computer vision researcher and 4-year Cohere community member, currently working on a world model project for drones 🌐 The session walks through building a compact, transformer-based video world model inspired by the Cosmos architecture 🏗️ The world being modeled is an excavator arm performing sand mining — chosen for continuity with a previous chapter's video GPU & Model Size 📉 Original plan was a ~3B parameter model; revised to target 8GB and 16GB consumer GPUs so attendees can run training locally ⚙️ Two model configs are currently available; two more (for larger GPUs) will be added later Model Architecture Components 🗜️ VAE (3D Causal VAE from Wan 2.1): Compresses raw video into compact spatiotemporal latent representations before passing them to the transformer; our Cosmos-style implementation leverages Wan 2.1's pretrained Causal VAE 🎞️ Spatiotemporal compression works because consecutive frames and neighboring pixels share high temporal and spatial redundancy, allowing 8×8 spatial and 4× temporal downsampling without losing critical detail 🧩 Patchification: Latents are grouped into 2×2 patches to reduce the sequence length of tokens fed to the attention heads; given the VAE's 8×8 spatial downsampling, a 2×2 latent patch matches a standard ViT's 16×16 pixel receptive field ⚡ Larger patch → fewer tokens → lower compute footprint but coarser spatial-contextual representation 🔬 Smaller patch → more tokens → richer fine-grained dynamics learning but quadratically higher compute; careful trade-off needed 🎯 Attention Head / QKV: 🔍 Query (Q): what the current spatiotemporal token is looking for (e.g., the next kinematic state or position of the excavator arm) 🔑 Key (K): what each token offers (its own spatiotemporal feature representation) 📦 Value (V): the actual information payload routed forward once attention weights are computed 🧮 Self-attention scales as O(n²) across total token count n; 3D VAE compression + 2×2 patchification drastically reduce n to make local training tractable 🌊 Flow Matching Network (Denoising): 🌫️ Direct single-step L2/MSE latent prediction learns the conditional expectation (the blurry average of many plausible futures) — undesirable for multimodal, dynamic scenes ✨ Conditional Flow Matching starts from Gaussian noise and iteratively integrates a learned velocity field toward a clean target latent over multiple ODE steps 🔗 The Diffusion Transformer (attention backbone) directly parameterizes this velocity field, unifying spatiotemporal context conditioning with iterative generation 🎭 Prefix Mask: Tells the transformer which latent frames are observed conditioning context (input) vs. which future frames it must generate; under Wan 2.1's 1 + 4k → 1 + k causal temporal stride, 5 raw frames compress into exactly 2 latent frames ⏩ The model conditions on observed frames 0–4 (2 latent frames) and generates target frames 5–16 (3 latent frames) as an example 🛡️ Target frames are interpolated with noise for flow matching training, while observed prefix frames receive light conditioning-noise augmentation (or per-frame noise levels) so the network stays robust against compounding rollout errors Q&A Highlights 🧬 VAE source: Our implementation uses the open-weights 3D Causal VAE from Wan 2.1 rather than training a custom tokenizer from scratch or using NVIDIA's proprietary Cosmos Tokenizer ⚖️ Patch dimension choice: Driven by available compute and spatiotemporal redundancy trade-offs; strictly problem-specific with no universal rule 💸 Redundancy: Token redundancy is primarily a compute and memory budget concern (training/inference cost), not necessarily a representation quality bottleneck 🎯 Design trade-offs generally: Reaching 80% quality with 20% engineering effort is feasible; the final 20% of fidelity requires 80% of the effort and is typically domain-specific or an open research question gitbook: world-models-from-scratch.gi… github: github.com/nahidalam/world_m…

Session 2 for World Model from Scratch book is on Thursday Sept 17 at 9AM PST. We will look into an existing World Model architecture and evaluating it with existing benchmarks. luma.com/ymbl73af?tk=EoQlJ5 Summary of the first session for the book 📋 Session Overview & Logistics 🎙️ Hosted by @nahidalam, a computer vision researcher and 4-year Cohere community member, currently working on a world model project for drones 📅 This is a bi-weekly series of ~10 chapters covering world models from theory to hands-on implementation 🎓 Attendees who complete ≥75% of sessions will receive a certificate from @Cohere_Labs 📖 GitBook (open, interactive) will be published publicly after this session ends; GitHub repo is already public 📈 Chapters 4 and 5 are expected to generate the most interest; earlier chapters build foundational concepts 🧠 Chapter 1: Core Concepts 🌍 What is a "world"? A bounded environment — not necessarily visual or video-basedExamples: self-driving cars, drones, computer-use agents (clicking through Slack), Excel-operating agents in banking ⚙️ What is a world model? A learned system that models how an environment changes — predicting possible futures given current state and actionsKey distinction: world model ≠ video model; video is just one possible world type 🔑 Key vocabulary defined: 👁️ Observation — what the world looks like now (e.g., camera frames, sensor data) 📦 State — encoded representation of the current situation 🕹️ Action — input that drives the agent (e.g., braking, speed change) 🔄 Transition — one predicted step from current state to next 🛣️ Rollout — a full sequence of transitions representing one possible path to a goal 🛤️ Multiple futures: A model naturally produces multiple probable next states; the highest-probability outcome is typically selected, but all possible futures can be explored 🌪️ Dynamics are environment-specific (e.g., drone aerodynamics, car vehicle physics, biological processes) 🎯 Why world models? Core value is reasoning over possible futures — especially useful for rare/edge-case scenarios (e.g., a child unexpectedly crossing the road, or simulating impact of a new medicine on a geography) 🏗️ Simulators: Historically rule-based; increasingly integrated with learned world models and RL. Seen as a space ripe for disruption, especially for startups 💻 Chapter 2: Hands-On Demo (NVIDIA Cosmos Model) 🌌 The series will use the NVIDIA Cosmos ~2.5B parameter model for inference; a similar model will be built in Chapter 4 (capped at ~3B parameters) ⛏️ Demo showed a rollout explorer using a sand-mining input videoSteps: load video → encode frames → load Cosmos model → generate one or multiple rollouts using different seeds 🔍 A disagreement map can highlight where two rollouts diverge 🎛️ Key parameter: guidance scale — controls how strongly the prompt influences output (lower = less controlled) 💾 Pre-generated outputs are committed to the GitHub repo so attendees without GPUs can follow along 🚀 Tested on NVIDIA A40 (48 GB VRAM); inference may be possible on smaller GPUs; Mac GPU not tested and likely challenging 💡 Q&A Highlights 🧬 Is the world model generative? Yes — the series uses generative models similar to transformers 🔁 Can a rollout be categorized as continual learning? Unanswered — flagged as a good open question 🕸️ Is it related to graphs? Potentially — the tree-like rollout structure is graph-like in CS terms, but no direct graph-world-model work was cited 📉 World model vs. tabular RL? World models handle complex, high-variation environments; tabular/rule-based methods suffice for deterministic, low-variation scenarios 🎲 How is uncertainty represented? Through the probabilistic nature of the learned system (e.g., Markov models, LLM-based approaches); probability is the representation of uncertainty 🔗 Where is causality embedded? Likely in actions (direct events) and in environment-specific dynamics, possibly also trickling through the loss function — noted as an interpretability question ⚖️ World model vs. RL? RL is a type of learning; world model is a type of model — RL can be used to train a world model but is not required ☁️ Can inference run on Google Colab? Likely yes for inference (Chapters 2–3); training may be limited depending on Colab tier 💽 GPU Resources for Upcoming Hands-On Sessions ⚡ Modal — $30 free compute credits on the starter plan 🌩️ Lightning AI — previously offered free credits; current status unclear ☁️ Google Colab — works but may require workarounds for memory constraints ⏳ Hands-on session (Chapter 4 build) is approximately one month away; attendees encouraged to prepare GPU access in advance gitbook: world-models-from-scratch.gi… github: github.com/nahidalam/world_m…