Will Tesla actually be able to do full self-driving?

Based on my original article at Quora.

I’ve driven every incarnation of Tesla’s Autopilot system, from the Autopilot Convenience Feature using the Mobileye chip, to Enhanced Autopilot using TeslaVision and the NVidia computer, to the FSD computer and Tesla’s limited distribution Full Self-Driving Beta 10.10.2. I have some basis for an informed opinion.

Autonomous driving has two parts: building a computer model of the roadway surrounding the car, and deciding how to drive on it.

Both Tesla and Waymo (Google’s autonomous ride-sharing project) use artificial neural networks to map sensor data into a model of the car, the roadway and objects on the road. They both use expert-written computer code to tell the car what to do, and they are both trying to increase the role of machine learning and reduce the amount of computer control code. Here is a diagram from a Waymo presentation showing the overlap (ML is Machine Learning):

Neither Tesla nor Waymo to my knowledge have disclosed how big each piece is, but Musk (in the interview below) indicated that a significant shift will occur towards the ML side in Version 11 of their FSD that’s in alpha test now, with a possible release date of February 2022 missed.

But Musk says that the driving rules are the easy part; vision is the hard problem. Waymo simplifies the problem with specially-prepared high-definition maps. The car already knows every curb and every driveway before it starts. (Preparing those maps will delay deployment by Waymo to new locations.)

Vision is the hard part, but look at where Tesla was as of version 10 of its FSD Beta:

It sees cars, trash cans, motorcycles, pedestrians, stop signs, speed limit signs, lane lines, mile markers, turn lane markings, traffic lights and even dogs.

It’s easy for me to understand why Waymo uses high-definition maps when I drive and notice all the weird stuff in the real world — roads that are really parking lots, and nonstandard intersections. I run across things that I can’t visually figure out at night.

As a user of the cutting-edge pre-release Tesla system, it’s easy to be both optimistic and pessimistic. It does wonderful things, and it has colossal failures.

What I believe, however, is that things are headed in the right direction and that we’ll experience more rapid improvement soon, coming from:

• Merging the city and highway software stacks (one stack to rule them all)
• The transition from human-coded rules to machine learning
• The transition from processed images to photon counts as inputs to the neural network (this will greatly improve night driving)
• Faster neural net training when the Dojo computer comes online in mid 2022
• Reduced latency (the time it takes to respond to an input) due to software optimization and a new C++ compiler.

We should see these in FSD Beta 11.

The most pessimistic sign is when Elon Musk says “Level 4 autonomous driving in 2022.” This is the same guy who said a fleet of robotaxis by the end of 2019.

So yes, I believe Tesla will reach Level 4 autonomy. I expect it to be pretty much done in 2022. I’ve been wrong before, but I know more now than I did before.

Update

It’s been over a year since this article was written. Today is April 28, 2023. Version 11 was actually released this year to all the purchasers of FSD. I’m currently on 11.3.6.

We still don’t have Level 4 autonomy, and Tesla’s newest cars have a different camera suite than mine, plus a faster computer.

Version 11 has some nice improvements, and it certainly drives more smoothly on curvy roads. It respects pedestrians and does more aggressive lane changes. The city and highway stacks have been merged. But it still does stupid stuff, especially getting into turn-only lanes when it’s not turning. Autopark is mostly not working these days since they turned off the ultrasonic sensors in favor of cameras.

So maybe in 2024.