PhD @ POSTECH, prev intern @ ByteDance Seed

Joined September 2021
Seungwook Kim retweeted
⛰️ 太惊艳了,微软的 TRELLIS.2 能把一张平面图直接变成能上手用的 3D 资产。 GitHub 上 1.2 万 star,一千四百多个 fork,背后有论文(arXiv 2512.14692),40 亿参数的模型权重和在线演示都放了出来。 做 3D 的朋友最清楚,以前拿一张参考图起模型,要先建基础形体,再手动缝拓扑、补 UV,最后一张张贴材质,一个道具磨一天不稀奇。TRELLIS.2 用的是一种叫 O-Voxel 的稀疏体素结构,开放表面、非 manifold 的几何、内部封闭结构都能直接生成,不用先做有损转换;出来的东西带完整 PBR 材质,粗糙度、金属度、透明度都在里面,渲染出来有透明效果也不用返工。 速度写得很实在:在 H100 上 512³ 分辨率约 3 秒,1024³ 约 17 秒,1536³ 约 60 秒。门槛也没藏着,目前只在 Linux 上测过,显卡至少要 24GB 显存,A100 和 H100 上验证过。 模型和代码按 MIT 许可放出来,这种大方程度不多见。 GitHub:github.com/microsoft/TRELLIS…
8
84
1
578
33,707
Seungwook Kim retweeted
🎉 Native, encoder-free VLMs are an important step toward more unified multimodal intelligence. We’re also pioneering this paradigm with our NEO & SenseNova-U series: - NEO: github.com/EvolvingLMMs-Lab/… - SN-U: github.com/OpenSenseNova/Sen… Excited to see the field move forward together!
We now have state-of-the-art pre-trained base models in the 3T-parameter weight class, built entirely in-house at MAI. Post-training results coming soon -- stay tuned! Lots of details in our technical blog post. microsoft.ai/news/frontier-p…
3
47
3
336
16,991
Seungwook Kim retweeted
Google just open-sourced Harness². (turn any harness into a self-improving system) a harness is everything wrapped around the model, like the system prompt, tools, memory, skills, and subagents. today a person designs it once, and the agent runs every new task with that same fixed setup. Harness² lets the agent refine that setup for each new task on its own, with the model weights left untouched. on every task, the agent proposes edits to its harness, runs a few candidate versions, and composes a better harness by comparing how those runs behaved. because it learns from the differences between runs, it needs no correct answer or grader for the task. that covers a single task. Harness² also carries what it learns from one task into the next. after each task, it saves the harness it settled on along with the lessons that held up across the runs. the next time it refines a harness, it starts from those notes instead of from scratch. this means every finished task makes the next round of edits more informed. Google tested this with Gemini 3.5 Flash and GLM-5 on legal, office, and professional work benchmarks. one round of edits was enough to beat their original harnesses, and giving the agent more rounds pushed scores higher. those extra rounds can go wide or deep. the agent either tries several candidate harnesses at once or keeps refining a single one step by step, using OpenCode or Codex to carry out the work. you can find the Harness² repo here: github.com/google-research/h… I wrote about a similar idea for agents already running in production, where the harness should repair itself after a failure instead of waiting for someone to debug the trace by hand. that piece walks through how @Cometml's Opik (open-source) closes this loop, going from a failing trace to a proposed fix to a regression test that stops the same failure from coming back. it's quoted below.
286
821
85
6,447
551,096
Seungwook Kim retweeted
CEO of Stripe's advice to young ambitious people:
66
907
63
7,306
231,911
Seungwook Kim retweeted
绝对是本周最值得读的文章之一,来着 Claude 团队的 Felix 的个人博客。 一篇帮助你理解如何更好地与 AI 协作,如何更聪明地使用 AI。 在这个能力每隔几周就突飞猛进的世界里,我们需要一些技巧,让我们能够做到以前从未想过或认为不可能的事情。 其中有个对我有启发的点: 关于如何把一件事情做到极致的指令或思路:坚持使用业内顶尖专家使用的工具;选择一些看似不切实际的目标,然后告诉模型这些目标非常合理;挑战你的认知,创造新知识,使用外部评判者。 甚至还搭配了一个提示词: 我想要的是一个更深入的版本,而不是一个更大的版本。在重写题目之前:(1)请说明该领域的顶尖专家会做哪些我没有要求的事情;(2)请列出他们会从哪些最原始的素材入手,而不是我已经整理好的素材;(3)请列出他们会使用的具体方法和工具;(4)除了你我之外,还有什么可以证明结果是正确的。然后,请修改我的题目,使其完全符合上述要求。
Hi, I'm Felix. I'm an eng lead working on Claude. I love watching people take flight and do things they never thought possible. We often describe that as "being more ambitious" when using AI, which is tricky advice to follow. I wrote a tutorial: felixrieseberg.com/how-to-be…
35
212
10
1,248
217,176
Seungwook Kim retweeted
OpenAI 前首席科学家 Ilya Sutskever 曾给传奇程序员 John Carmack 开过一份神秘书单。 Ilya 说:“如果你真正吃透这 30 篇论文,你就掌握了如今 AI 领域 90% 的精髓。” 但纯啃英文 PDF 和数学公式太反人类了。最近有位硬核小哥用 30 天时间,把这 30 篇神作全部手搓复现成了极简 Python 代码,并完整开源! 想从底层彻底看懂现代 AI?这个宝藏项目建议先马后看 🧵👇
62
391
11
2,693
159,080
Seungwook Kim retweeted
A great read. The blog post captures years of progress in mechanistic interpretability and offers an optimistic vision: perhaps we really can open the black box and understand how neural networks work. I share that optimism, but with some reservations. I'm skeptical of explanations that stop at individual neurons, accompanied by compelling visualizations that seem to "explain everything." Humans are remarkably good at convincing themselves. It's easy to cherry-pick evidence that supports a hypothesis, especially when the visualizations look convincing. I'm also not convinced that "Recursive Generalization" is fundamentally different from hierarchical representation learning. As representations become more abstract, they discard details while preserving invariances. Isn't that essentially what hierarchical representations have always been about? IMHO, we need to move beyond a largely descriptive, behavioral approach toward a principled mathematical framework. And we have a natural starting point: gradient descent. Rather than just asking what a network has learned, we should understand why learning dynamics produce these structures in the first place. Of course, this is extraordinarily difficult. Look at OpenAI's 719 mathematical proofs. We see plausible proofs tackling problems at the Fields Medal level, yet no comparable breakthrough that cracks deep learning theory. That contrast is striking. Part of the difficulty is that there are countless plausible theories of how deep networks work. Proving even one can be mathematically formidable. And a proof alone isn't enough. Each theory needs empirical validation, often requiring substantial GPU compute. It's a daunting combination of mathematical and experimental challenges. Still, I believe this is the way forward. Interpretability gives us observations and hypotheses. But ultimately, we need a theory that explains why these phenomena emerge from the learning process itself. That's how we turn compelling stories about neural networks into actual understanding. And hopefully recursive self-improvement can help with that 😀
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works already-clean.goodfire.com/
12
17
2
182
15,523
Seungwook Kim retweeted
This is f*cking gold.. A Chinese lab just dropped "The End of Documentation" and it reads like an obituary for every README on GitHub. The claim in one line: documentation is a rumor about the program, the artifact is the truth. The text was written once, at release. The code moved on the next morning. Every agent built on the docs is built on a memory. The fix is one reading procedure the model runs on the artifact itself. Six steps: inventory, structure, behavior, data flow, rebuild, verify until the diff is empty. Across 26 systems the recovered spec beat the official docs in 23 and surfaced 61 behaviors the docs never mentioned. So you stop being the person who reads the manual. You become the person who points: you choose what gets opened and you check the diff. Docs turn disposable. The artifact turns into the job. That is the exact system I run with REA + Kimi K3 and documented below: one prompt, 14 binaries, 14 rebuildable specs, zero docs opened. The whole paper in one line: the program is the spec, the model is the reader ↓
13
16
120
7,865
Seungwook Kim retweeted
Representation conditioning is better than representation alignment. The two recent papers reported similar observations. Time for some new trials? RECON: arxiv.org/html/2610.20608v1 Dino forcing: arxiv.org/abs/2610.11751v1
6
31
150
10,740
S4VY is an incredible advancement in the field of real-time 4D joint segmentation and depth prediction! We’ll be releasing the code and checkpoints soon, so stay tuned for more updates.
1/9 Introducing S4VY: Segment Anything, now in 4D 🧵 From any set of RGB observations (a video, a few scattered frames, or an unordered photo collection), S4VY segments every object in 4D, each with one consistent identity across space and time.
1
12
106
9,408
Seungwook Kim retweeted
Here is the full talk:
Had fun writing this talk for our MPhil students today: "How to avoid AI Brain Rot" Discussing how to avoid intelligence atrophy from AI use in work and study. And how to stand out on an ever-narrowing bell curve
78
1,213
70
9,579
607,302
Seungwook Kim retweeted
有点震撼。。。我们可能真的快要搞清楚大模型内部到底是怎么运作的了。 OpenAI早期机械可解释性研究的参与者,昨天突然提出了一个很大胆的结论: 我们可能一直误解了神经网络。 昨天, @nickcammarata 发了一篇很有意思的文章,叫《Neural Networks Are Already Clean》。 这哥们早年就在OpenAI做机械可解释性研究,2023 年还牵头启动了@OpenAI 第一个大规模自动化神经元解释项目。 当时他们发现,现有工具只能解释大约 2% 的神经元。 Nick因此得出了一个很悲观的判断:神经网络内部存在大量纠缠,想搞清楚它们到底在干什么,可能非常困难。 结果三年后,他带着更先进的工具和Agent重新研究,发现情况完全不同。 在他们测试的现代视觉模型中,最后一层 99%~100% 的神经元都能找到相当清晰的解释。 甚至在Qwen3-32B这样的语言模型中,初步研究也发现,约90%的神经元都有一个可以识别的主要功能。 最有意思的是,他还重新研究了过去的一些经典案例。 比如曾经有一个神经元,既会对猫脸产生反应,也会对汽车前脸产生反应。 以前大家觉得,这是典型的Superposition,多个不相关的概念纠缠在了同一个神经元里。 结果重新研究后发现,这个神经元可能只是在识别一种共同的V形视觉结构。 猫脸和汽车前脸恰好都有这种特征。 全世界的家人们折腾了这么多年,可能连最基础的观察方式都有问题。 而且我觉得这篇文章最值得关注的,其实是他们使用 Agent的方式。 他们让Agent自动观察神经元、提出功能假设、设计实验、分析电路,再通过新的输入验证假设,甚至自动审查了10 万张神经元归因图。 这其实也是我最近一直关注的一个方向:让 AI 自己去做科学研究,把大量观察、假设、验证的工作自动化。 过去需要研究员花大量时间逐个分析的神经元,现在已经可以让Agent成规模地开展研究了。 当然,目前这些结果还处于早期阶段,距离完整解释大模型的内部机制,还有很多工作要做。 但我觉得Nick提出的这个观点非常有意思: 神经网络内部可能一直都很有规律,只是过去我们的工具太原始,根本看不清楚。 如果这个方向最终被验证,机械可解释性领域过去很多基础假设,恐怕都需要重新审视了。
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works already-clean.goodfire.com/
59
448
37
2,970
252,783
Seungwook Kim retweeted
Very ambitious work by Seungone on trying to figure out if LLMs can actually write strong follow-up papers across the natural sciences. Based on our previous work on automated paper review, we apply rigorous standards to the generated papers and find that even with frontier models, none are hallucination-free and meet the bar of the scientists working in each domain. We believe that this poses a challenge for upcoming LMs, can they actually do an end-to-end research work and generate an artifact that could move science forward?
🧪Can AI agents take a scientific paper and write a meaningful follow-up to it? We introduce Into the Unknown, an AI agent benchmark of 15 open scientific research problems. Each problem comes from a scientist currently or formerly at US DOE National Labs: @LosAlamosNatLab @SandiaLabs @BerkeleyLab @AmesNatLab @PPPLab @Livermore_Lab
3
8
100
6,771
Seungwook Kim retweeted
For anyone interested in the theory of neural net representations, and representational alignment, I highly recommend Sharut’s PhD defense talk, linked below, which is an amazingly clear and deep presentation! youtu.be/CW1g43R9E3s
Seems like a good time for a little life update ✨ I recently defended my PhD at @MIT_CSAIL and joined @GoogleDeepMind as a Research Scientist on the RSI team! My years at MIT were quite a ride! I got to follow my curiosity wherever it took me, work on ideas that sometimes sounded better on a whiteboard than in practice, and learn a lot along the way. Looking back, I wouldn’t have had it any other way. I owe so much to my wonderful advisors @phillip_isola and @StefanieJegelka for believing in me and giving me the freedom to find my own way. I’m also grateful to my committee members Kaiming He and Tommi Jaakkola for their thoughtful feedback. And to all my mentors, collaborators, labmates and friends, thank you for making the years as fun as they were intellectually rewarding ❤️ It's also been a pretty exciting first month at DeepMind, getting to work on the newly released Gemini 4 Argon! Couldn’t have asked for a more fun start ✨ For anyone curious, here’s my PhD defense talk: youtube.com/watch?v=CW1g43R9…
5
21
429
32,829
Seungwook Kim retweeted
The AI Scientist v3 is out: The Agentic Scientist. GitHub: github.com/SakanaAI/AI-Scien… Blog: pub.sakana.ai/agentic-scient… arXiv: arxiv.org/abs/2610.18933 The Agentic Scientist is the 3rd iteration of our automated AI Scientist research system, and the shift this time is from “one agent writes a paper” to “a scaffold of parallel agents orchestrates the full research loop.” The system now orchestrates a pool of parallel workers that share memory and distilled conclusions. Crucially, it features Evidence Generation: isolated agents automatically run controlled baselines and ablations to prove why an idea actually works. It works as a co-scientist. We used a development version as a co-scientist to explore biologically plausible learning rules. The agent handled the empirical heavy lifting, proposing mechanisms and running experiments. We stayed in the loop to steer direction, verify results, and moderate claims. That feels like the right division of labor for where the technology actually is. This resulted in a new biologically plausible learning algorithm, and the resulting paper was accepted at ALIFE 2026 as an oral presentation. We are also releasing AutoIllustrate. It generates editable draw.io vector figures instead of uneditable, hallucinated raster images. Real scientists need vectors, not pixels. I am an SVG junkie ;) Incredible work by Robert Lange, Luca Grillotti, Yutaro Yamada, Rujikorn Charakorn, and the team!
The Agentic Scientist: Orchestrating Coding Agents for Automated Scientific (Co-)Discovery Blog: pub.sakana.ai/agentic-scient… GitHub: github.com/SakanaAI/AI-Scien… arXiv: arxiv.org/abs/2610.18933 Since 2024, we’ve used The AI Scientist as a way to test how far AI alone can take scientific research, from v1 to v2 to the paper published in Nature earlier this year. The Agentic Scientist is the next step in that line of work, built for the era of coding agents. Coding agents such as Codex and Claude Code now carry out long software tasks on their own. Handing them a research problem, however, tends to produce a strong hyperparameter optimizer rather than a scientist: scores go up, but ideas stay narrow, and the reasons for the gain are rarely examined. The Agentic Scientist makes the research loop explicit. It wraps off-the-shelf coding agents in a scaffold that controls what they see, what they write, how they are evaluated, and how results are handed to the next stage. Parallel workers test hypotheses against rubrics defined before any experiment runs, share successes and failures through a common memory, and then ablate the winning program to generate evidence. We used an early version of the system to study Error Diffusion, a biologically motivated alternative to backpropagation. Our researchers set the question and the rubric, chose directions at each stage, and verified the results, references, and claims before submission. The resulting paper was accepted as an oral presentation at ALIFE 2026. In autonomous runs, the system also raised Doc-to-LoRA retrieval accuracy from 52% to 98% on multi-needle tasks and designed a learned evolution strategy that improves over its CMA-ES reference on unseen tasks. Alongside it we release AutoIllustrate, a pipeline for scientific figures that keeps an editable draw.io file as the source of truth, so every diagram can be corrected by hand. Humans define the question and the standards of evidence; agents expand the experiments researchers can run. We call this Co-Discovery, and we are releasing both systems as open source for the community to build on.
90
287
19
1,840
83,509
Today, we introduce Priming Models, a new paradigm for training one-step generative models from scratch. No diffusion teacher. No flow map. No CFG. No JVP. We believe this work will challenge how we think about building one-step models. Joint work with @sitanch, @du_yilun, @parkwbin03, Bohyung Han. 🔥 1.19 FID one-step on ImageNet 256×256 while using substantially less compute, e.g., 12X less than Drifting-B, 6X less than Drifting-L 🎨 Scales to text-to-image generation (GenEval 0.82) 🤖 World modeling on RT-1 / Bridge-V2 Priming Model is built on one core message: [Reconstruction pretraining --> Distributional post-training].
38
128
24
1,451
98,623
Seungwook Kim retweeted
Very cool internship project by @ziqi__ma !
Very excited to share LoGo, my internship project at @theworldlabs on post-training world models! We found that reward design matters a great deal in post-training long-horizon video gen for 3D consistency, echoing what we see in other domains, e.g. LLM reasoning. Check out more 👉 ziqi-ma.github.io/logo-websi…
2
36
3,189
Seungwook Kim retweeted
这篇我很推荐,尤其是现在做科研的人。 作者 Pierre Marion 提了一个很有意思的判断。LLM 如果真的把算公式、写代码、搜文献、做实验这些环节加速 10 倍甚至 100 倍,科研里最稀缺的东西会慢慢转向选什么问题、怎么建模、以及怎么判断结果有没有价值。 更值得警惕的是,AI 省下来的时间如果只是拿去发更多论文,最后很可能只是把 publish-or-perish 加速到失控。作者甚至直接判断,以论文数量和 h-index 为核心的评价体系正在失效。 作者最后给的方向是让 AI 扩大探索空间,把人的时间留给判断、品味、理解和培养下一代研究者。 以后科研能力可能像「你会不会驾驭一支极快的 AI 研究团队」,而不是一个人能亲手完成多少步骤。 文章: pierremarion23.github.io/tho…
37
245
12
1,209
77,176
Seungwook Kim retweeted
New paper from the team at Sakana AI: Recursive Self-Improvement through Multi-Agent Self-Supervision (MASS) How do we keep improving AI when the tasks become too complex for humans to reliably evaluate? For open-ended research, we eventually hit a "supervision bottleneck." If we use the model itself as the solver, optimizer, and evaluator, it gets trapped in a homogeneous loop, reinforcing its own blind spots rather than correcting them. To break this loop, we drew inspiration from Marvin Minsky’s Society of Mind. What if you prompt a single model to organize copies of itself into a structured team, then distill what the team learned back into the base model? MASS alternates between two loops: 1/ Workflow Optimization: the model iteratively evolves its own multi-agent workflows, figuring out how subagents should communicate and route information to each other. 2/ Workflow Internalization: fine-tune the shared model on the successful multi-agent trajectories so the coordination behavior gets baked into the weights. After two cycles with a 27B open-weights model, performance improved 1.2-1.6x per output token on tough research benchmarks (MLR-Bench, ScienceAgentBench, DSBench). Multi-agent traces were also much better training data than single-agent ones. The model learns faster from jointly learning high-level orchestration and bounded subagent execution. The results showing that the model can bootstrap its own coordination skills by organizing copies of itself into teams feels promising as a mechanism towards self-improvement. Blog: pub.sakana.ai/mass/ Paper: arxiv.org/abs/2610.12176 Incredible work by Hyunin Lee, Jinglue Xu, Jeffrey Seely, Yujin Tang, and our collaborators from UC Berkeley (Somayeh Sojoudi, Matei Zaharia, Donghyun Lee)!
Introducing Multi-Agent Self-Supervision (MASS): Recursive self-improvement through collective intelligence Blog: pub.sakana.ai/mass/ As AI systems tackle increasingly open-ended research problems, human experts may struggle to assess results and guide further progress. The model itself may be the best available optimizer and evaluator. The challenge is obtaining useful self-supervision within that homogeneous loop, where a model risks reinforcing its own blind spots rather than correcting them. We ask whether a model can learn from the collective intelligence of its own team. We introduce Multi-Agent Self-Supervision (MASS). One shared model solves tasks through a team of virtual subagents, searches for better multi-agent workflows, then uses its own judgments to select the best ones. Training on the best team’s executions provides self-supervision by distilling the team’s collective experience into the shared model. We ran two cycles using a 27B open-weights model on synthetic open-ended research tasks. After the second cycle, score per output token reached 1.2-1.6x the base model's level across four research benchmarks. Three key results: First, MASS enables role generalization within an RSI loop: training only on task-solving data from the optimizee naturally improves the capabilities of both the optimizer and the evaluator. Second, multi-agent trajectories are more efficient training data than single-agent trajectories for RSI. Third, optimizer capacity, rather than evaluator capacity, can be the bottleneck in the speed of improvement. We believe the community needs to investigate homogeneous RSI carefully to support AI safety. When a model is also its own evaluator, shared blind spots could allow mistakes to be accepted as progress and reinforced through training. For AI to improve beyond human expertise safely, we need to understand these failures and preserve effective ways to detect them and intervene. Paper: arxiv.org/abs/2610.12176 Code: github.com/SakanaAI/mass This work was a collaboration with researchers at UC Berkeley.
96
248
23
1,761
93,820
Seungwook Kim retweeted
JEPA is one of the best ideas in how machines can learn about the world, however, it is still under-explored which parts of the state it learns stay the same and which parts change. For example, a coffee mug looks different in bright morning sunlight and in dim light at night, but my hand still reaches for the same spot, because my brain keeps where the mug is separate from the lighting, which changes but does not matter for reaching. Our new paper, SplitJEPA, gives JEPA this same separation in the state it learns. In fact, the LeJEPA line of work from @ylecun, @randall_balestr, and @klindt_david already proves something strong here. Under stationary Gaussian predictive dynamics, they show that latent prediction with Gaussian regularization recovers the complete latent state up to a global orthogonal transformation, with no reconstruction needed at all. That remaining transformation is where our paper begins. Picture a map where every distance between cities is correct but there is no compass. Technically, nothing is missing, yet you still cannot tell which way is north. For LeJEPA, any orthogonal transformation of the learned state is an equally optimal solution, so each dimension of the learned representation can be a linear combination of invariant factors, which are shared across related observations, and variant factors, which change between them. SplitJEPA adds that compass. It divides the representation into an invariant block and a variant block, and alongside the predictive pairs it also trains on invariant pairs, where the two observations in a pair share the invariant factors and differ in the variant factors, such as the same physical state rendered from two cameras. An agreement loss requires the invariant block to match across each pair, which removes its linear dependence on the variant factors. Because the full transformation is orthogonal, once the invariant block holds exactly the invariant factors, the variant block is left with exactly the variant factors. The key condition is sufficient variation. SplitJEPA requires the second-moment matrix of the variant changes across pairs to be full rank, meaning no variant factor, and no combination of them, remains unchanged across all pairs. A single pair does not need to change every factor though, since the condition is on the aggregate coverage. We prove that under these conditions the invariant and variant subspaces are identified up to independent block-wise isometries, without introducing an observation decoder. The result is also stable. When the agreement holds only approximately, the leakage of variant factors into the invariant block is bounded in terms of the consistency error and the smallest eigenvalue of the variation matrix. We evaluate SplitJEPA on PushCube, a ManiSkill simulation task where a robot arm must push a cube into a goal region on a table. When the test camera is rotated 10 degrees around the scene (a camera pose not seen in training), behavior-cloning policies succeed only 12.8% of the time on raw pixels, and 48.0% on frozen R3M (a pre-trained visual representation for robot manipulation), while SplitJEPA achieves a 71.5% success rate. LeJEPA showed that the latent state can be recovered without reconstruction, and SplitJEPA extends this to show that its invariant-variant organization can, and should, be recovered the same way.
Replying to @Helios_Hua
The key idea of SplitJEPA is to combine predictive learning with matched observations that share invariant factors but differ in variant factors, using their complementary information to resolve latent block ambiguity.
6
26
2
316
12,334