Founder at Noumenal Labs. Building Physical AI.
Dallas, TX
Joined May 2008
- Tweets4.8K
- Following1.4K
- Followers1.3K
- Likes7.2K
Excited to share these early results in which we combine our harness with a VLA and a frontier model to resolve tasks without the latency and cost of relying on it completely. We have more in the works on this front, so follow @noumenal_labs and stay tuned!
LLMs can drive robot arms now. So why don’t you just ask Astra?
On the same task, same robot, the LLM alone needs ~120 calls and 22 minutes. π0.5 with our harness needs 3.5 calls and under 2 minutes. π0.5 alone fails.
A thread about what we have done @noumenal_labs
What a great day. Amazing companies building real robotics products for agriculture, sharing their experiences with the industry. We need more of this. @bernsteinres and @reservoirfarms are clearly onto something special here. Looking forward to next year!
6AM of Ruggedize. Quiet for another hour.
Soon the farm fills and ANYTHING in agriculture starts to feel possible. Resilience feels possible. Automation feels possible. Upskilling feels possible. Cutting the chemicals we put in the ground feels possible.
Thank you to our new community, and to the Reservoirians and partners who built this.
cc @westerngrowers @johndeere @reservoirfarms
All of this
imo if you're starting a robotics company and you're not taking a straight shot at either 1) owning the hardware or 2) owning the labor (or ofc owning both), you're pretty cooked from day 0
software-only dev tool type products: 1) solve mostly pretty menial, low-pri problems for companies actually trying to deploy hardware (but they sound cool!) 2) have a smaller addressable market today than the robotics hype train wants you to believe, and in practice you'll end up having to sell to more "old school" big manufacturers to get your business to work (which comes with it's own problems / unlikely to work from an ivory tower in SF)
Jason Fox retweeted
Our friends @noumenal_labs strapped an XTR-0 to an agriculture bot and used it as the brains 😁
Not how I imagine how people would use the desktop form factor but very cool!
This is why we are creating Z1 m.2 sticks, which will be far more compact and 100k× more pbits than X0
Very important updates from @extropic. Their hardware will open up a host of sampling-based control routines for Physical AI and we’re here for it!
Major update from us @extropic
Announcing Torx and Thermalizers, basically our version of PyTorch + kernels, but for Thermo hardware
Also: Z1 is now taped out and in fabrication! Will be available on 0.5M pbit sticks, and 4M pbit PCIe cards
+1 more thing at the end... enjoy!
This was fun little detour from a build out with @PHuembeli. To my knowledge, the first robot in the world with thermodynamic hardware in its control loop. Powered by @extropic.
This quoted post is unavailable.
A very insightful reality check on where real value capture sits in the robotics industry. Hypey demos get you conversations - full stack deployment makes you sticky.
I think @cixliv has it right. Hard not to conclude that the near term use of humanoid robots that makes the most commercial sense is entertainment. Especially, since every demo video from Chinese manufacturers is literally robots performing martial arts moves.
Why humanoid robots, why not? A rant.
The humanoid form is the right form if you want generalized “physical AI”. As the method to train it will be human data, into a human-like robot.
Then economies of scale will make humanoids cheaper than specialized robots due to mass production. Thus making it lucrative to get generalized AI.
But there are four main issues: economics, data, hardware and distribution.
Economics are a problem because unlike ChatGPT that can be a pocket lawyer (thus saving you $1000 an hour) most blue collar labor is much cheaper. Replacing a $15 an hour job with a $50,000 robot (that will perform more poorly) is not economical.
Data estimates are Humanoid robot training data ≈ 0.000001% to 0.00000001% of LLM training data. The only way we bridge this gap is a massive data collection effort or very robust life like simulators burning through GPU farms all day.
Hardware: Humans have around 250 DOF while the top of the line humanoids have around 40 DOF. Although we don’t need all this degrees of freedom that a human has to solve most tasks. Humanoids now only last about 1-2 hours on battery life, most aren’t water proof, and still far from parity with the human body.
Distribution: Say we solve all the above 3 issues, we still need to mass produce the robots. “Physical AI” isn’t something we can simply access with a browser or a phone. You need to ship millions of robots in excess of 100-150 pounds that are more complex than cars all over the world. With huge raw materials and rare earth requirements. This will take time.
So while everyone knows the future is humanoid, it will take longer than people realize to become disruptive. So don’t worry about those blue collar jobs being taken by robots for a while.
But you know what will be lucrative in the meantime? Entertainment.
That is why at @rek we are aware of the limitations of humanoids, and will bridge that AI gap with entertainment. Entertainment that will set the framework for a product category that will forever change our world.
To become the next F1, with humanoids.
We are officially entering into the humanoid robotic “gladiator” phase. A natural evolution given where the technology is(strong locomotion, limited agency) and humanity’s enjoyment of sport and entertainment.
Jason Fox retweeted
During this year’s NeurIPS afterhours, we’re hosting an intimate gathering of researchers, founders, and investors exploring the intersection of computation, thermodynamics, and embodied intelligence.
If you’re working at the edge of alternative architectures, stat-phys-inspired ML, or embodied intelligence, you’ll feel right at home.
What to expect:
• Thought-provoking conversations on alternative compute paradigms
• A curated group of technologists & builders
• Great food, great drinks, great company
Hosted by cyber•Fund, Noumenal & collaborators. Registration & approval required: luma.com/tqgzktjg
Looking forward to connecting with the people shaping the next chapter of intelligent systems.
@mjdramstead @cyberfund
Jason Fox retweeted
New blog post by @noumenal_labs: “WTF is the FEP? A short explainer on the free energy principle”: noumenal.ai/post/wtf-is-the-…
Really happy to share this one! We discuss the free energy principle: What it is, what it is not, what promise it holds, why it can be extremely useful, and why it has yet to live up to the hype.
Jason Fox retweeted
New blog post by @noumenal_labs: “Grounded rewards in the era of experience: A commentary on ‘Welcome to the era of experience’”: noumenal.ai/post/grounded-re…
Here’s the tl;dr:
• This post is a commentary on a new paper by Silver and Sutton, entitled “Welcome to the Era of Experience” (2025).
• Silver and Sutton (2025) provide a thought-provoking discussion of the last decade of research and development in the field of artificial intelligence (AI), and where the field is heading. The core idea is that we have reached a performance ceiling for AI agents trained via supervised learning from human data — and that we have entered a new epoch in the development of AI, which the authors call the “era of experience.”
• The era of experience, as the authors describe it, is a forthcoming phase in the development of AI that will be characterized by “grounding” in the real world, online action-perception loops, physical embodiment, environment-sourced reward signals, and online real-time experiential learning.
• In particular, Silver and Sutton argue that the era of experience heralds a shift from hand-crafted, user-specified reward functions and the heavy use of human expert feedback and supervision, towards “grounded rewards,” which are measured and evaluated by AI agents themselves by continually assessing the sensory consequences of their actions in real time.
• Here, we review and evaluate their argument. We enthusiastically embrace several aspects of their discussion and offer some constructive feedback pertaining to the learning of grounded reward functions.
Physical AI is a new frontier that presents challenges beyond the scope of current approaches to AI. Deep Learning works great in use cases where data is abundant, but in the physical world data is sparse and ever-changing. This necessitates a new set of architectures. Let's go!
I’m thrilled to share a blog post by @noumenal_labs: “From Natural Intelligence to Physical AI”:
noumenal.ai/post/from-natura…
Here’s the tl;dr:
• Physical AI is the next big wave of research and development in the field of artificial intelligence. Its proponents claim that Physical AI holds the promise of revolutionizing industry.
• But state of the art AI will not deliver on these promises, because it is not capable of understanding the structure, variability, and complexity of the physical world that we inhabit.
• Noumenal Labs is a newly formed deep tech company that is laser focused on building digital brains for Physical AI — so it can be deployed profitably, efficiently, safely, and at scale.
• We are using our unique, proprietary macroscopic physics discovery technology to build object centered world models that will power the brains of autonomous systems — unlocking machines that can act in intelligent and situationally appropriate ways in the real world, and that can adapt to a changing world in real time.
• Driven by key insights from statistical physics and cognitive science, in particular by Karl Friston’s active inference framework, the approach pioneered at Noumenal Labs unlocks the capability of machines to represent the physical world in the same way we do, enabling them to act safely and in alignment with human values — and thereby, to deliver on the promise of Physical AI.
Jason Fox retweeted
Neuroscientist Dr. Jeff Beck from @noumenal_labs discusses the fundamental nature of representation, understanding, and modelling, comparing biological intelligence with current artificial intelligence.
Jeff argues that *how* information is represented dictates predictive ability and that LLMs, while impressive at symbol manipulation and pattern matching (like next-word prediction), lack the *grounded*, causal understanding of the world inherent in biological systems.
Timestamps:
00:00 - Cat visual cortex experiments & discovering orientation sensitivity (slide projector analogy)
01:49 - Representation choice and neural coding (orientation vs. feature intensity)
02:30 - Choice of representation impacts predictions; generative models
03:15 - Importance of choosing the right generative model for predictions
03:35 - The problem: We don't know the brain's true generative model
03:55 - Theory of Mind (ToM) in LLMs
04:05 - Jeff Beck's ToM tests on early ChatGPT (stapler example)
05:40 - ChatGPT recognizing the ToM test vs. passing it
06:32 - Analogy: LLMs recognizing known problems vs. generalizing (sum/product riddle)
07:25 - Do LLMs implicitly build world models? Vicarious experience analogy
07:59 - The difference: Grounding symbols in reality outside language
08:35 - AI Alignment: Difficulty in capturing human reward functions & belief formation
09:21 - Nightmare scenario: Humans as "complacent value function selectors"
09:44 - Hope: AI enhancing human understanding, not replacing thought
10:08 - Philosophy of science: Science realism vs. modeling pockets of regularity
10:39 - Noise in models as ignorance or deliberate exclusion (design choice)
11:00 - Design choices in science, controlled experiments, and induced bias
11:29 - Are there true, discoverable mathematical laws of the universe?
11:41 - Is there a "true" ground truth distribution (P)? Beck's answer: No (with nuance)
12:55 - Ontological vs. Epistemological divide: Perfect models vs. models of regularities
13:21 - Are scientific models "false by definition"? The Bayesian perspective
14:07 - "All knowledge is conditional"; Are foundational theories (e.g., FEP) true or just perspectives?
14:51 - FEP as a mathematical framework, not a theory; models are just models
15:54 - Legibility vs. Utility: Useful but illegible AI models
16:01 - Prediction vs. Explanation: Trusting black boxes can be unsatisfying
16:30 - Why understanding AI matters: Ensuring alignment with human decisions/values
17:08 - Line-of-sight legibility as an alignment approach
17:14 - Benefits of explainable AI: Human understanding and value alignment verification
18:21 - RL components: Prediction engine, reward function, policy; the alignment challenge
19:25 - Trusting AI = Trusting its policy aligns with our reward + its superior beliefs
19:57 - Language: Intrinsic representation vs. pointers between shared minds
20:21 - Why language works: Shared internal models and common grounding
21:00 - Basis of shared understanding: Not linguistic, but shared experience/intuitive physics
22:44 - Consciousness and language as lossy, simplified summaries of complex brain processes
23:22 - Evidence for simplification: Brain regions, perception vs. representation; limits of language models
24:32 - Counterpoint: Language captures complex/ambiguous human concepts
24:54 - Language as massive compression: The information bottleneck (Meister's paper)
26:22 - Implication: Language/actions are poor representations of internal understanding
27:03 - Can language models understand? The mimicry argument (Piantadosi)
27:33 - Beck's skepticism: LLMs excel at prediction/mimicry, not true understanding
28:09 - LLM explanations replicate structure but lack grounding
28:54 - Beck's test for LLM understanding: Genuine novelty beyond training data
29:19 - Summary: Symbol manipulation is not understanding; grounding is key
30:06 - Abstraction and Idealization in scientific modeling ("The Brain Abstracted")
30:45 - Revisiting Newton: Intuitive physics is correct for our world; idealizations are simplifications
32:01 - Sophistication & boundaries: Nested systems vs. one complex system?
32:32 - The boundary problem in FEP/Markov Blankets: Where to partition?
33:41 - Beck's research: Finding principled partitions based on interaction dynamics
35:44 - Beyond direct experience: Imagination, language, and learning
36:16 - Human creativity: Creating new *things* by combining modeled objects (Systems Engineering)
37:43 - Goal for AI: Automating systems engineering for creative combination
38:17 - Sutton's "Reward is Enough" paper
38:25 - The challenge of "Reward is Enough": Defining and obtaining the *right* reward function
39:02 - Difficulty of eliciting individual reward functions
39:52 - The core alignment problem: Accessing and representing individual reward functions
40:13 - Impossibility: Disentangling beliefs and rewards from observed actions
41:51 - Argument analogy: Disagreements stem from different beliefs or values
43:00 - Prerequisite for value inference: Understanding belief formation
43:13 - Building aligned systems: Sparsity of data, meta-models vs. base system modification
43:46 - Proposed solution: AI layer that models the human's belief formation system
44:40 - Alignment process: Align beliefs first, then address value differences
45:00 - Conclusion
CC @mjdramstead
Beautiful work
In the physical world, almost all information is transmitted through traveling waves -- why should it be any different in your neural network?
Super excited to share recent work with the brilliant @mozesjacobs: "Traveling Waves Integrate Spatial Information Through Time"
1/14
Jason Fox retweeted
New blog post by @noumenal_labs: “AI alignment and theory of mind”
noumenal.ai/post/ai-alignmen…
Here’s the tl;dr
• In this blog post, we discuss the approach to AI alignment pursued by Noumenal Labs
• The goal of AI alignment is to develop methods and protocols that ensure that artificial agents make decisions and execute actions that are consistent with the goals and values of humans. Technically speaking, this means designing machine intelligences that share a policy or reward function with their human users.
• This is an important goal. But state of the art approaches in artificial intelligence are not designed in a way that can deliver on these promissory notes.
• There is no normative solution to the problem of reward function selection. Expert trajectory replication, reinforcement learning with human feedback (RLHF), and the automated generation of reward functions from the stated preferences of users are not viable solutions to the AI alignment problem.
• The missing ingredient for AI alignment is the capacity to take the perspective of another and evaluate their beliefs — what is known as theory of mind.
Jason Fox retweeted
New blog post by @noumenal_labs: “Filling the gaps in active inference”
noumenal.ai/post/filling-the…
Here’s the tl;dr
• In this blog post, we discuss active inference. Specifically, we discuss key gaps in state-of-the-art applications of active inference in artificial intelligence — and how Noumenal Labs is working to fill them
• Active inference is a form of Bayesian machine learning that replaces the objective function used in traditional machine learning — the cost or reward function — with an information theoretic surprise minimization objective, which is used for both inference and learning.
• While theoretically well grounded, active inference has yet to yield state of the art performance in practice.
• This gap in performance is largely due to the relative simplicity of the generative models, underlying state space, and relationships typically used in the active inference literature. Generically, these models lack the requisite structure to represent the world as we understand it, i.e., as composed of distinct objects and object types, as well as object-type-specific interactions, unless hand-specified by the user.
• Noumenal Labs is developing a new approach to macroscopic physics discovery which enables us to build machine intelligences that discover the different objects/concepts and object/concept types that generate time series data, as well as the type-specific rules that govern their behavior — doing so directly from data, in an unsupervised manner.
• Noumenal Labs’s approach is to build models that are explicitly structured — in a manner that is consistent both with the means by which we intuitively understand and conceptualize the world, and the means by which our best scientific models describe it.
Jason Fox retweeted
I’m happy to share the new @noumenal_labs preprint, “Dynamic Markov blanket detection for macroscopic physics discovery,” by Jeff Beck and me
arxiv.org/abs/2502.21217:
Here’s a thread!
This paper resolves a key outstanding issue in the literature on the free energy principle (FEP), namely, to develop a principled approach to the detection of dynamic Markov blankets. The FEP has been proposed as a generalized modeling method capable of mathematically describing arbitrary objects that persist in random dynamical systems; that is, a mathematical theory of “every” “thing.”' The FEP starts with a mathematical definition of a “thing” or “object”: any object that we can sensibly label as such must be separated from its environment by a boundary. Under the FEP, this boundary is formalized as a Markov blanket that establishes conditional independence between that object and its environment.
Nearly all work on the free energy principle has been devoted to explicating the dynamics of information flow in the presence of a Markov blanket, and so the existence of a Markov blanket is usually assumed. Garnering significantly less interest is the question of how to discover Markov blankets in the first place in a data-driven manner.
Accordingly, in this preprint, we leverage the free energy principle (FEP), and the associated constructs of Markov blankets and ontological potentials, to develop a Bayesian approach to the identification of objects, object types, and the macroscopic, object-type-specific rules that govern their behavior. This is accomplished by reframing the problem of object identification and classification and the problem of macroscopic physics discovery as Markov blanket discovery. More specifically, we develop a class of macroscopic generative models that use two types of latent variables: (1) macroscopic latent variables that coarse-grain microscopic dynamics in a manner consistent with the imposition of Markov blanket structure, and (2) latent assignment variables that label microscopic elements or observations in terms of their role in a macroscopic object, its boundary, or the environment.
Crucially, these latent assignment variables are also allowed to evolve over time, in a manner consistent with Markov blanket structure. As such, this algorithm allows us to identify not only the static Markov blankets that have concerned the literature to date, but also — and most importantly — to detect and classify the dynamic, time dependent, wandering blankets that have caused controversy in the literature since the turn of the 2020s. This allows us to model the exchange of matter and energy between an organism and its environment, but at the cost of massively expanding the set of Markov blankets that we need to consider. We handle this additional complexity by applying the automatic Occam’s razor effect of Bayesian inference to select the best partition.
In developing this algorithm, we provide an empirical, definitive conclusion to the debates about whether objects like flames and living creatures, which undergo material turnover, have Markov blankets. We demonstrate that they do, providing simulations that identify the Markov blankets of Newton’s cradle, a burning fuse, a Lorenz attractor, and an artificial lifeform generated by Particle Lenia.
We also make philosophical and theoretical contributions to the free energy principle. We offer a new, deflationary take on the free energy principle, focusing on the role of the modeler. In the extant literature on the free energy principle, the role of surprise minimization is treated tautologically, not empirically: the idea is that “things minimize surprise”. Our treatment focuses more on the pragmatics of empirical modelling, where the role of surprise is better summarized as “labeling something as an object of a specific type minimizes my surprise.”
