Benchmarking Action-Conditioned World Models against Real-World Executions.
Joined October 2026
- Tweets11
- Following4
- Followers13
- Likes8
We adapt and evaluate Cosmos 3 Nano in WorldOracle.
We initialize it from the official 16B checkpoint and compare its predicted rollouts with paired real-world executions.
#NVIDIACosmos #PhysicalAI
Cosmos evaluation includes physical state correctness and interaction spatial alignment.
WorldOracle's metric case studies show why both matter:
Aligned action, incorrect physical outcome.
Correct physical outcome, misaligned action.
🔗Link:worldoracle-benchmark.github…
Does the predicted future match reality?
Our benchmark WorldOracle evaluates action-conditioned world models against paired real-world executions through three stress tests:
1. Causal obedience
2. Distribution transport
3. Temporal endurance
#PhysicalAI
These comparisons are scored along four complementary dimensions of execution coherence.
In a study with 128 prediction pairs and 119 participants, all four metrics achieve 88.1-93.2% overall agreement with human preferences.
🔗Link:worldoracle-benchmark.github…
To run these stress tests, we replay recorded actions open loop, including successful and failed attempts.
We test across AgiBot G2, YAM, and human-hand embodiments and a G2 policy-rollout split, then track execution coherence over long, multi-segment rollouts.
WorldOracle is a benchmark for evaluating action-conditioned world models (ACWMs) against paired real-world executions.
Given the same initial observation and low-level action trajectory, does the predicted future match reality?
🔗Link：worldoracle-benchmark.github…
Multi-View Action Coherence evaluates predicted robot motion against paired real-world execution across synchronized views.
Robot-arm alignment is scored independently in each view, then the scores are averaged. Video: AgiBot G2 real execution.
🔗Link:worldoracle-benchmark.github…
Same initial observation. Same action trajectory.
In the "pick fruit" example, the real gripper misses the fruit, but the model hallucinates a successful grasp.
WorldOracle evaluates execution coherence: whether the predicted rollout matches its paired real-world execution.
The ACWM replays the recorded action trajectory open loop. WorldOracle evaluates the predicted rollout against its paired real-world execution using:
Physical State Correctness
Interaction Spatial Alignment
Key Interaction Area Fidelity
Multi-View Action Coherence
Hello, X! 👋
We are WorldOracle Bench, dedicated to evaluating frontier AI’s ability to understand the world, reason about reality, and predict the future.
Follow us as we explore the boundaries of AI capabilities.
#AI #Benchmark #WorldOracleBench