Tesla's FSD v12 is a REALLY big deal. Last night @elonmusk livestreamed a ~45-minute video of a Model S driving itself around using Tesla's latest self-driving software, FSD v12. Self-driving is something the company has been trying to solve since Tesla Autopilot was released in 2015. Since then, Tesla has made steady progress improving the code to handle all kinds of road situations, has added cameras around the car, and removed sensors, all the while rewriting the code multiple times to solve for things the car couldn’t handle. And even though Tesla cars can drive themselves in many situations, the biggest challenge with self-driving cars is the thousands (or millions) of situations drivers face on a daily basis that are completely unexpected or difficult to solve for, like other drivers acting irrationally, inclement weather, debris on the road, weird (or lack of) lane markings, etc. Up to this point, Tesla and other companies have to spend a large amount of time running through simulated scenarios to generate code that would teach the system how to handle these situations. This code is oftentimes written by a human and needs to account for every variation of something happening on the road. And even then, there are thousands (or millions) of situations that a simulation won’t come with, since real life is so damn complicated and complex. The approach many have taken to try and solve this problem is by overfitting their self-driving cars with a ton of different sensors like LIDAR, radar, ultrasonics, cameras, and other sensors in addition to generating High-Definition maps of the areas where the cars are meant to be driven in. They’ve done this in hopes that there’s a combination of sensors and map data that would allow them to solve for the most unexpected situations. Here’s a picture of a GM Cruise vehicle that uses this approach. However, yesterday’s video has demonstrated a breakthrough in how self-driving cars can operate. Instead of using many sensors and hardware on their cars to process the world, Tesla is using 8 cameras and a computer that’s specifically built to process video data. That’s 9 total parts vs everyone else’s 30, 40, 50+ parts to process the world. Using only vision to process the world and not needing things like LIDAR, radar, and other sensors to interpret the physical objects around the car is impressive enough, but HOW the car learns to do this is what the real breakthrough is. With FSD v12, Tesla takes the video data that is collected by its fleet of ~4 million cars and runs it through an AI that is built using Neural-Nets (like ChatGPT but for the real world), and then the AI figures out what the car should do based on what it sees. What’s important to highlight here is that Tesla has done 0 work in telling the car how it should interpret the world. That means that the AI doesn’t explicitly know what a lane is, what a traffic light is, what a stop sign is, what a cone is, what a pothole is, what rain is, etc. What Tesla does is it shows the AI a metric-ton of video of a car driving around using the 8 cameras that are outfitted around the car, and the AI learns how to do the same. The more video Tesla feeds it, the better the system gets. The more unique situations that are collected from the fleet of Teslas, the better the system gets at accounting for those situations. This “AI code” will then be beamed to every Tesla in the world, and the on-board computer will be able to process its surroundings without the need to connect to Tesla’s AI server. It’s no different than you or I getting into a car and driving it. Instead of our 2 eyes collecting video around us and using our brain to process the info, the 8 camera system will collect the video and Tesla’s on-board computer will process it using the “AI code”. In other words, Tesla has moved away from humans figuring out how to write code that tells the car what to do, and instead feeds video to an AI, and the AI figures out the best way to account for every situation. The only thing Tesla needs moving forward is more video and more compute power (chips) to process videos. That’s it. This is profound. Everything that is captured on video with the 8 camera system is something the AI will be able to figure out how to navigate through. Snow. Potholes. Deer. Cyclists. Aliens. You name it. And this “code” that is generated by the AI will get better and better as Tesla’s compute capabilities grow with their purchases of NVIDIA’s H100 chips and the build-out of their in-house DOJO compute system. Not to mention the growing fleet of Teslas that will capture more and more video data around the world. Every Model S, 3, X, and Y that is sold today is a video-capturing robot that feeds the AI. And every Cybertruck and Compact Car will be the same. And believe it or not, it gets even crazier. Now that Tesla has come up with a real-world data collection and processing system with its cameras and on-board computer, this same system can be used on other physical products that can learn how to move around its surroundings. This is where Tesla’s Optimus Bot comes into play. Tesla will be able to use its 8 camera system on the Tesla Bot to collect video of its surroundings, beam it back to the AI, the AI figures out how to best do that thing that it’s collecting video for, and then beam back the “AI code” for the Bot to process with its on-board compute. We are not far away from a world where a humanoid robot watches you do the dishes, sends back the data to the mothership to process it, and then the next morning the Bot has learned how to do the dishes - not only using the video it gathered from you doing it, but from every other person in the world washing dishes. Do this process with literally anything. ChatGPT showed the masses what the potential of AI can be. NVIDIA showed the masses just how much demand there is for hardware that is used by AI systems. And now, Tesla just showed the masses what AI means for real-world physical applications. The world has changed - again.