hacker-in-residence/ principal devrel engineer. data curation for physical and visual ai systems. in my physical ai era

I ship daily
Joined April 2020
VOXEL51 IS REBORN TODAY!!!! Starting today, anyone can try Voxel51's paid features for FREE with just an email. For anyone who hasn’t tried our product yet, here are three things I’d try first:  → Voxel51 Search: Hybrid search across embeddings and structured metadata. All data uploaded to Voxel51 is automatically indexed, so you can search by semantic similarity alongside exact conditions, metadata, and multimodal events out of the box. → Agentic Labeling: Powered by VLMs, generate labels from natural-language instructions and visual examples, then review and refine the prompts before applying them more broadly. → Voxel51 Agent: Everything you can do with Voxel51 can now be initiated with a prompt. Ask and answer questions about your data and models, and turn those actions into repeatable workflows. Try it on your own data and tell me in the comments what works and what doesn’t: lnkd.in/gx4_kV_C
4
1
9
361
Tactile is making its first steps in @LeRobotHF datasets, and @DataScienceHarp has you covered on how to visualize it 🫆
we gave robots eyes and called it manipulation but try tying your shoes with numb fingers robots need to feel their data, not just see it i pulled some of the most interesting tactile lerobot datasets into fiftyone, all ready to use on the hub, plus what to check before you train on touch huggingface.co/blog/harpreet…
1
3
120
harpreet retweeted
Jimmy at Voxel51
@jimmy_voxel51Oct 13
Exact-duplicate search misses mirrored, cropped, or filtered copies of your images — different bytes, same picture - voxel51.com/blog/image-copy-… That’s train/test leakage, wasted labeling spend, and licensing headaches waiting to happen. Voxel51’s fix: combine two signals that fail in different places. 🔹 pHash — reads pixel layout, breaks on mirrors/crops 🔹 CLIP — reads meaning, fooled by look-alikes 🔹 A rule you control to combine them (“any” vs “all”) 🔹 Every link shown as a graph, with evidence 🔹 Scored against ground truth with native evaluation Check out @DataScienceHarp's post! #ComputerVision #MachineLearning #OpenSource #AI #DataCuration
3
7
394
harpreet retweeted
Jimmy at Voxel51
@jimmy_voxel51Oct 12
Failure mining in robotics: find one failure, surface every similar one - voxel51.com/blog/failure-min… Most robotics teams find failures by scrubbing hours of recordings by hand. This Voxel51 post shows a faster workflow in FiftyOne: * Start with one confirmed failure (a robot drops a cloth mid-fold) and mark it with a temporal tag * Use segment-level embedding similarity search to surface every visually similar moment across the dataset — not whole episodes, but 5-second windows * Embeddings find similar situations, not failures — a dropped cloth and a deliberately placed one look nearly identical. Scoping the search to episodes already flagged as failed cut candidates from 16 down to 2 * Review and confirm the real failures, then tag + save them as a reusable set for retraining or evaluation The result: a repeatable loop — find, search, narrow, confirm, tag — that turns a messy pile of recordings into a growing, labeled failure dataset. #Robotics #PhysicalAI #MachineLearning #ComputerVision #DataCuration #AI #FiftyOne #Voxel51 #RoboticsAI #DataFlywheel
1
2
104
harpreet retweeted
The robotics data ecosystem keeps getting stronger 🦾 @Voxel51 now reads LeRobot datasets natively, with episode visualization, embedding exploration and semantic search across embodiments 🤗
you balanced your robot dataset by embodiment. the embeddings say you balanced it by recording session i pulled 10 episodes from each of the 50 robot types in lerobot's community dataset and embedded every clip with qwen3-vl 1. 495 of 497 clips have their nearest neighbor in the same session 2. 292 score below 0.1 uniqueness 3. the two "most unique" clips are single-frame recordings, 0.02 seconds long filter the session out and the cross-embodiment signal shows up: a cloth fold on one rig lands next to so100 arms folding cloth i indexed all 497 episodes in fiftyone so you can search by text, cut the near-duplicates, and export a curated lerobot v3 subset one prompt to an agent with fiftyone skills built the whole thing start here, read the full blog: huggingface.co/blog/harpreet… read the dataset card: huggingface.co/datasets/Voxe… @LeRobotHF @Alibaba_Qwen
3
6
70
5,044
harpreet retweeted
Jimmy at Voxel51
@jimmy_voxel51Sep 29
Tokyo's most accident-prone intersection, turned into a dataset you can actually explore. Check out this lightning talk from @DataScienceHarp on the Hard Intersections Multimodal Dataset from Dynamic Map Platform Co., Ltd. - huggingface.co/datasets/dyna… * 6 synchronized camera views (full 360°) * 29 classes of road infrastructure — labeled entirely by models, zero manual annotation * Trajectory data at 200Hz * HD maps in 5 industry formats * ~35M-point aggregated LiDAR All in one interactive viewer with FiftyOne. No exporting. No jumping between tools. #ComputerVision #AutonomousVehicles #FiftyOne #OpenSource
3
6
323
harpreet retweeted
Jimmy at Voxel51
@jimmy_voxel51Sep 24
VLA models don't learn from folders of images. They learn from episodes—vision, state, and action locked to one clock, frame by frame - voxel51.com/blog/lerobot-v3-… We loaded 11 LeRobot v3.0 datasets into FiftyOne so you can inspect them before you download. * Single Franka arms to full humanoids * Real teleoperation + pure simulation * Fingertip tactile sensing (20 streams) * From 10 to 27,500 episodes What we found digging in: * InternData-A1 is 100% synthetic—and looks photorealistic * droid_3d's depth streams silently drop (importer can't decode the dtype) * Licenses vary within a single export Look at your data before you train on it. Check out @DataScienceHarp 's post. #PhysicalAI #Robotics #VLA #LeRobot
3
5
166
Link to the full tutorial: youtu.be/Zqno_vux6d8?si=KBjt…
GOOGLE HA SOLTADO ALGO QUE NADIE ESPERABA. Un tutorial de 8 minutos sobre cómo construir tus primeros agentes de IA.
2
5
33
4,436
harpreet retweeted
Jimmy at Voxel51
@jimmy_voxel51Sep 23
When a racing drone's onboard navigation drifts, does it recover? Usually not - voxel51.com/blog/visual-iner… Explore the tii-ratm-drone-racing dataset in FiftyOne. 6 indoor quadrotor flights with onboard VIO recorded next to motion-capture ground truth. We checked whether the drift ever corrects itself: * 4 of 6 flights: worst error was the very last pose (no recovery) * RMSE alone gets it backwards, the flight that recovered had higher RMSE than ones that didn't * The fisheye tracker dropped ~45% of every frame across all flights * Worst flight drifted 1.66m RMSE The lesson: don't rank a SLAM stack on RMSE. Compute final-over-max, does it end on its worst moment, or correct itself? Why it matters → warehouse robots, inspection drones, any GPS-denied vision-only navigation. Check out @DataScienceHarp 's post! #PhysicalAI #Robotics #ComputerVision #SLAM #DroneRacing #VIO #FiftyOne #Voxel51
1
4
5
350
🚗🌧️ Multi-Sensor Autonomous Driving + @Voxel51 The CMHT Autonomous Dataset brings together: 🔹 LiDAR 🔹 mmWave Radar 🔹 RGB 🔹 Thermal/IR 🔹 GPS + IMU 🔹 9,000+ labeled frames You can explore it with Voxel51 FiftyOne: import fiftyone as fo from fiftyone.utils.huggingface import load_from_hub dataset = load_from_hub( "Voxel51/cmht-autonomous-driving" ) session = fo.launch_app(dataset) 🔥 A great dataset for exploring sensor fusion, 3D perception, object detection, tracking & adverse-weather vision. 🤝 Open to ML / Computer Vision collaborations! 👉 Follow me for more AI, CV, YOLO & CVOps content. @jimmy_voxel51 @DataScienceHarp #Voxel51 #FiftyOne #ComputerVision #AI #MachineLearning #AutonomousDriving #LiDAR #SensorFusion
3
7
157
harpreet retweeted
Jimmy at Voxel51
@jimmy_voxel51Sep 22
Join Voxel51 and Nebius at hubs.ly/Q04y0cx50 on Sept 26 for an afternoon of Physical AI talks and a hands-on robotics workshop - hubs.ly/Q04y0czK0 * A hands-on workshop taking real robot logs all the way to a trained detector * We’ll curate, embed, auto-label, and fine-tune YOLO11n in one sitting using Nebius Serverless AI Jobs * Explore multimodal robot data — camera, proprioception, and language on one synced timeline in FiftyOne. No ROS install required. Bring a laptop. Space is limited. Pre-registration is mandatory. #PhysicalAI #Robotics #ComputerVision #MachineLearning #FiftyOne #berlin
1
3
2
168
harpreet retweeted
Jimmy at Voxel51
@jimmy_voxel51Sep 21
"Humanoid dataset" is doing a LOT of heavy lifting - voxel51.com/blog/humanoid-le… We pulled 5 humanoid + bimanual datasets into LeRobot v3.0 + FiftyOne. Same broad task — grab a thing, move it. But how does each robot describe that task? * State vectors range from 23 to 216 dimensions — a 9x spread * Same task category, totally incompatible representations * GR1T2 vs Reachy2: identical benchmark, identical cameras — state still differs by 8 dims, purely from kinematics * No coarse alignment applied — raw, as each team built them That gap is embodiment heterogeneity, and it's why a policy trained on one robot can't natively touch another's data. Each dataset ships with a live FiftyOne Space! Check out @DataScienceHarp 's post. #Robotics #PhysicalAI #VLA #MachineLearning #LeRobot #EmbodiedAI
2
7
358
harpreet retweeted
Jimmy at Voxel51
@jimmy_voxel51Sep 18
The largest robot dataset in our new field guide: 637,498 trajectories. All synthetic - voxel51.com/blog/lerobot-dat… The two real, human-teleoperated ones? 4,258 and 1,804 episodes. That gap explains everything about VLA training right now. Real robot data doesn't scale like web data. We pulled 5 LeRobot v3.0 datasets into FiftyOne and compared how teams close it: * 2 real teleoperation sets (Franka + Flexiv arms) * MimicGen — re-stitches 1,250 human demos into new scenes * RoboTwin — LLM writes tasks, VLM catches failures, zero human demos * InternData-A1 — fully scripted, no demos at all The catch: only InternData-A1 reports actual zero-shot sim-to-real transfer — and it hasn't been independently reproduced. Synthetic data is scaling fast. Proof it transfers to real robots is still catching up. Check out @DataScienceHarp 's post! #Robotics #PhysicalAI #VLA #SyntheticData #MachineLearning #LeRobot #EmbodiedAI
3
1
7
546
harpreet retweeted
Jimmy at Voxel51
@jimmy_voxel51Sep 18
Importing LeRobot datasets into FiftyOne with an agent! One sample = one episode. No parser required - voxel51.com/blog/lerobot-imp… The new fiftyone-lerobot-import agent skill: * Detects whether a dataset is still on the old v2.x layout * Auto-converts v2.x → v3.0 before import * Stops to ask before installing packages or creating a dataset * Runs as a thin layer over the open-source FiftyOne SDK — same import from a chat prompt or a checked-in script Chck out @DataScienceHarp's post! #PhysicalAI #Robotics #FiftyOne
2
2
5
345
harpreet retweeted
Jimmy at Voxel51
@jimmy_voxel51Sep 14
Meta's Aria Digital Twin ships one episode as 7 files, 12.4 GB — and none of it opens in a viewer - voxel51.com/blog/aria-digita… Here’s hot to turn every ground-truth stream into one synchronized MCAP file for FiftyOne's multimodal viewer. ~300 lines of Python: * 12 channels — RGB, depth, segmentation, 3D boxes, skeletons, eye gaze, point cloud, trajectory, IMU — 181,444 messages, one file * Y-up vs Z-up: fix the axes or nothing lines up * 304 of 354 objects never move — log them once, don't redraw every frame * A clean run ≠ a correct run: add a physical-plausibility check message counts will never catch Check out @DataScienceHarp 's post! #PhysicalAI #Robotics #MCAP #FiftyOne
2
2
348
harpreet retweeted
Jimmy at Voxel51
@jimmy_voxel51Sep 14
Your MCAP file writes clean. Validator passes. Message count matches. You open it in a viewer and nothing plays - voxel51.com/blog/mcap-author… That's hand-authored MCAP: the worst bugs stay silent until playback. * foxglove.CompressedImage written as JSON → every topic "Encoding unsupported" * Full epoch value in nsec → OverflowError * Calibration casing (D/K/R/P vs d/k/r/p) → 3 failed attempts on one message * Hand-rolled PLY parsers silently dropping faces → just use trimesh * Flipped sign or wrong calibration matrix → decodes fine, plays fine, still wrong * Two sensors, one schema, two clocks → half your channels land in 1970 The most expensive ones never throw an error at all. Check out @DataScienceHarp 's guide! #PhysicalAI #Robotics #MCAP #FiftyOne
3
5
284
harpreet retweeted
Jimmy at Voxel51
@jimmy_voxel51Sep 9
Join Voxel51 and Nebius at hubs.ly/Q04w_0ZB0 on Sept 26 for an afternoon of Physical AI talks and a hands-on robotics workshop - hubs.ly/Q04w_0_P0 * A hands-on workshop taking real robot logs all the way to a trained detector * We’ll curate, embed, auto-label, and fine-tune YOLO11n in one sitting using Nebius Serverless AI Jobs * Explore multimodal robot data — camera, proprioception, and language on one synced timeline in FiftyOne. No ROS install required. Bring a laptop. Space is limited. Pre-registration is mandatory. #PhysicalAI #Robotics #ComputerVision #MachineLearning #FiftyOne #berlin
1
4
5
145
harpreet retweeted
Jimmy at Voxel51
@jimmy_voxel51Sep 9
Robot data doesn't fit in a folder of images - voxel51.com/blog/mcap-datase… A driving rig fires its camera, LiDAR, IMU, and GPS on four different clocks. Force that into "one image = one label" and you lose the timing or explode your sample count. So we repackaged 10 robotics datasets as MCAP episodes you can explore natively in FiftyOne: * 89 episodes, 6 domains — driving, SLAM, aquatic, off-road forest, social navigation, and one synthetic set. * One sample = one full recording — cameras, LiDAR, radar, sonar, IMU, GPS decoded live and scrubbable in sync. * Honest about labels: only 3 of 10 ship real object detections. We flag exactly which — plus every license quirk. Check out @DataScienceHarp 's post! #PhysicalAI #Robotics #MachineLearning
2
7
16
541
harpreet retweeted
Jimmy at Voxel51
@jimmy_voxel51Sep 9
The smoothest robot episode in the batch also failed the task - voxel51.com/blog/motion-smoo… We scored all 127 episodes of a cube-in-bowl task from RoboLab-EgoX on motion smoothness. The result: * One episode ranked 8th-smoothest of 127 — and still failed. Zero flags, no anomalies, every check clean. * Metrics like SPARC and LDLJ measure how the arm moved, not whether the cube landed in the bowl. A fluid path to the wrong place scores as flawless. * Lesson: score for triage, not gates. Smoothness tells you which episodes to watch — never which ones passed. Check out the post by @DataScienceHarp #PhysicalAI #Robotics #MachineLearning
1
3
6
191