This Changes Everything (Part II--How the magic happens)

Oh, please stick with me here. I have no idea whatsoever how to do what I’m about to do, but I’m hoping we can muddle through it together. Today I am writing the second of a three-part series on how unexpectedly my life has been changed by the purchase of a new “self-driving” Tesla. In the first part, I recounted the extraordinary difficulty I had in deciding to make a purchase that would enrich its loathsome manufacturer. Writing that was first-person storytelling—familiar territory for me.

But now I’m jumping into the deep end of the pool. This crazy car has just driven Patti and me 852 miles from our home in Princeton to our farm in Maine, and back again—and I never touched the steering wheel or pedals even once. An utterly amazing experience. Impossible, actually. But it happened. So I just have to understand how that was possible. And that means renouncing my comfort zone.

Here’s the deal: I am the farthest thing from a scientist. I earned three degrees after high school—one each in social sciences, educational psychology, and theology. But today I have the temerity to attempt an explanation of how/why the “Full Self-Driving (Supervised)” technology of my new Tesla actually does what it does to keep me and others safe while driving. The experience of being driven around by it seems miraculous to me (and our friends), so I’m determined to take a shot at understanding the science behind it. (Next week, I’ll get back to casual storytelling with a description of how I feel letting myself be driven around by this vehicle, and how that is actually changing me.)

As I edge into this scientific terra incognita, I’m going to use a few analogies to initially situate us in something familiar, then grope along those handrails into the mysteries, in hopes of not losing my way and not losing you along the way.

For instance: you may not be familiar with the technology of electronic pixels, but you’ve probably heard of the “Impressionist” period of art. It was a time of daring innovation when some artists experimented with painting not by brush strokes but by applying individual dots or points of paint. That technique was called “pointillism” (get the “point”?) Here’s “The River Seine at la Grande Jatte” (oil on canvas, 1888), a pointillist masterpiece by Georges Seurat, who used thousands of those dots to compose this painting.

You are actually more familiar with this technique of composing images using dots or points than you may think, since the “picture” on your TV is nothing more than a huge mass of dots called “pixels”. As the teensiest electronic bits of light called photons come into a camera, they cluster into individual pixels which are vastly more numerous than the thousands of paint dots Seurat used. Your home TV may display about five million pixels. And of course, they blaze through camera lenses and constantly shift around before your eyes literally at the speed of light.

There are eight camera lenses mounted around the body of a Tesla. They continually capture and rearrange photons and transform them into pixels that create images. Together they create a comprehensive 360-degree moving picture of everything that’s going on all around the car, and what the car itself is doing.

Some of the cameras have long-range telephoto lenses looking a quarter of a mile ahead; others are more wide angle to see vehicles in the lanes right next to you or behind you. At any moment, these eight cameras are noticing about forty million pixels and compiling them in visual images. Forty million pixels. Like, eight of your TV screens’ worth of images. The car is surrounded by these pictures of the driving situation in every direction, every instant. And a bank of ultra-high speed computers in the car are constantly viewing these pictures and making instantaneous decisions about what to do with the car based on the images being composed—begin moving ahead, or just keep going as is, or turn, stop, slow down, accelerate, swerve, slam on the brakes, park, or whatever.

But wait. Everything changes constantly. Second by second. That car in the oncoming lane suddenly turns left right in front of you, without signal or warning. Or that person waiting on the sidewalk with the baby carriage rolls it into the pedestrian-crossing. Or the traffic light flips from yellow to red just as your car approaches the intersection. Or the road-construction worker with his handheld Stop/Slow sign to control temporary one-way traffic flips it from “Slow” to “Stop” as you approach.

So the computer’s instructions to the car better change just as fast as the situation changes, right? And, how fast would that be?

Here is where I have to ask you to hang in with me. Because the numbers, and the reality, get really nutty now.

So, okay. We have these eight cameras which—at this particular instant—have compiled those forty million pixels of visual information into eight current images that the computer studying them is using to instruct the car how to behave this very instant (proceed, turn, stop, slow down, accelerate, swerve, slam on the brakes, park, or whatever). But something new has just happened (the left turn without a signal, the baby carriage in the pedestrian crossing). Better think fast, right? How fast?

Thirty times per second.

What? Say again?

Thirty times every second, the AI-powered super-computer chips notice everything that has changed in the last thirtieth of a second. It learns this by processing 1.44 BILLION PIXELS PER SECOND. That’s 1.44 billion pixel changes within each second. Now, not even the most blazingly fast chip can do that alone, so the Tesla system stacks three chips to cooperate in handling that workload—and (hold onto your hats here) to check on each other.

Huh? Oh, surely you know all about “Voting-Based Redundancy” in the world of computer chips, don’t you? Okay, okay, I’ll explain it to you. So as the car is driving along, one of the cameras reports “something” up ahead. Is it perhaps a shadow across the road, or a tree fallen across the road, a whiff of fog, or nothing? What is the computer to make of that report? Perhaps one of the chips interprets this “whatever” seemingly across the road as an actual obstacle and thinks the car should slow down or stop. Well, the other two chips take a look and decide it’s NOT an obstacle—so they outvote the chip that’s mistaken and the system keeps the car proceeding without interruption. All this happens in a fraction of time so infinitesimal that there’s not even a term to describe it.

And on it goes. Each second this trio of chips updates those portraits of reality thirty times, by continually recomposing 1.44 billion pixels and employing whatever else they have been trained to consider, based on similar situations captured from the Tesla fleet. They keep imagining what is happening this instant—or MIGHT happen in the next.

What do you mean “MIGHT happen”? Well, for a decade now, the millions of Teslas on the road have been noticing, capturing, and collecting gazillions of images. They are bundled together in a ginormous juke box in the sky where they represent an incomparable treasury of visual information about driving situations. The incomprehensible number of calculations these new Nvidia superchips can produce in nanoseconds makes these situations instantly comprehensible to the computers controlling the car’s operations.

This treasury of images has taught the computer that a parent with the baby carriage who is still on the sidewalk MIGHT move it out onto crosswalk. Parents are known to do that, and the massive Tesla fleet records these events and contributes them to Tesla’s massive body of data where they are distilled into finer understandings. And so when Tesla updates every car’s self-driving technology every couple of weeks, it makes each car’s computer aware of the newest understandings. That’s why the car has learned to slightly slow down in anticipation of its possibly happening. And a couple of hundredths of a second later, as the sensors catch the slight flinch of the carriage’s direction toward the crosswalk, the computer recognizes that what might happen is actually happening, and the computer orders the car to stop right there right now, short of the crosswalk. And it does.

Once the parent and baby in the carriage safely reach the far sidewalk, the car resumes its progress.

Now here is where Tesla’s computational system differs from others’. So bear with me for another homespun analogy like pointillist painting. This one comes right out of our kitchen, this afternoon. We have two beloved adult daughters who are both superb chefs. Ali and Shannon both set magnificent meals before us time and again. Their differing approaches to cooking provide a perfect illustration of Tesla’s system vs. the others’.

Just an hour ago, I watched Shannon return from the market and carefully array on the kitchen counter in precise order of eventual involvement various ingredients just purchased. I saw her lay open a cookbook, smooth out her recipe, and proceed to follow it assiduously step by step down to the least quarter-teaspoon measure of ingredients—that would be a leveled-off precise quarter-teaspoon, not a heaping quarter-teaspoon—in exactly the proper order and method. Some while later, she will present the splendid result to the eager diners at our table, according to appropriate protocol for the meal.

Meanwhile, up in Maine our daughter Ali is a different chef entirely. She opens a refrigerator, rummages around in whatever edible contents it happens to contain, pulls some stuff out, plunks the stuff onto the kitchen counter, and then plunges herself into doing something intuitive that transforms this random batch of ingredients into a stunningly delicious something-or-other. (Improbably, she follows the same intuitive approach during her periodic stints as a visiting chef delighting the guests at a local inn, or preparing food in the galley of a yacht in the Caribbean to nourish a sailboat full of ravenous scuba divers.)

I’m thinking that perhaps Tesla was inspired to overhaul their previous self-driving system after watching Ali prepare a meal. See, until a couple of years ago, Tesla’s driver-assist technology had a recipe book for instructing their cars how to behave while driving. Their recipe was hand-written by human beings trained to write computer code. Their recipe contained ~300,000 lines of code written in a highly sophisticated computer language called C++. Each line was an instruction, sort of like: “When you see a roadside sign that is a red octagon bearing the word ‘Stop’ on it, apply the brakes and bring the car to a halt behind or beside the sign.”

Three hundred thousand lines of code—a third of a million lines of code—compiled and written in the hope of addressing nearly every imaginable image or cue to which an automobile should respond. It’s what all the car companies did, and most still do. And the new superchips that companies like Nvidia create can blast through those zillions of rules to read them and apply them and tailor responses that create and implement instructions for the car. They do all that really, really fast, but still, it means making sequential calculations through about four or five steps—and each takes a little time, each offers the possibility of a mistake.

Tesla realized that no matter how many smart people they had writing however many lines of code trying to anticipate every conceivable situation a vehicle encounters while driving, it could never be totally up to date and totally comprehensive. The real-life conditions of driving change too constantly and rapidly and unpredictably—like temporary detours, or sudden torrential rainstorms, or a sticker pasted across “Stop” on a stop sign, or a kid whose bike hit a pothole and tumbles onto the road right in front of the car.

Couldn’t there be some way to leapfrog over the laborious, incessant, and necessarily incomplete recipe for instructing a car’s behavior in every second of its operation?

Enter “end-to-end neural networking”.

It’s intuitive. It’s the instantaneous, non-cognitive process of grasping whatever possibilities are contained in a mass of stuff. It’s whatever Ali senses…not calculates…when she opens that refrigerator and starts pulling out an armful of edibles from which some possibilities will form in her head without analysis or premeditation.

Let me mix my metaphor here—I’m going to add my tennis playing to Ali’s cooking. Between the time I learned to play tennis at age 12 until I had to hang up my racket at age 86 or so, I hit millions of shots. And sometimes, during those millions of times the ball was flying toward me on the court, I chose NOT to swing at it. That choice was purely intuitive. I’m not smart enough to have calculated the decision; I just KNEW not to hit it.

Like, say, when I’m playing up at the net. My opponent on the far baseline cracks a hard shot that I might volley back at him or her—but something tells me that if I leave this particular ball alone, it’s likely to land outside the line and give me the point.

“Something tells me…”

Now here’s the deal: that ball was traveling toward me at the net going about 75 miles an hour, coming from about forty feet away. So I have about one third of a second (0.36 second, to be exact) to decide to return it with a volley and then actually execute my volley stroke, or to pull my racket away and let the ball go past me in hopes that it will be “out” by an inch or more.

It is impossible for me to consciously calculate the speed, trajectory, spin, and drop rate of that ball and then instruct my body either to make a volley, or to pull the racket away, in just 0.36 second. Something else has to control that event. That “something” is the sheer mass of experience that has accumulated in my mind and body over half a century. Millions of shots. When that ball is whizzing toward me, I can’t possibly remember and think about any single one of those millions, and then intellectually compare it with this particular instant, and then make a judgmental decision, and then produce a physical action to volley or let it pass.

No, my reaction has to be entirely intuitive, triggered about a tenth of a second after my opponent hits the ball and it begins to head toward me. I am informed by all of those previous millions of shots mushed together in some kind of “fuzzy logic” that is producing my understanding of what was going to happen—and it’s almost always right on the money.

That’s what end-to-end neural networking does. The cameras provide an awareness of everything that the forty million pixels are displaying including my own car’s kinetics (its speed, direction, forces). At the speed of light, those images are transmitted to the car’s computer whose system has been continually updated every couple of weeks with those fresh understandings drawn from literally billions of hours’ worth of previous images captured by the Tesla fleet’s cameras over the past twelve years. It has learned to read and comprehend and obey the word “Stop” on a red octagon by the roadside.

But even if the word itself has been blotted out by a sticker someone slapped across it, the computer knows that red octagons beside the road at intersections always mean “Stop” even if the word is blotted out. Its intuition is informed by that critical mass of prior experience poured into it during periodic updates. And so in real time the computer is doing what I do at the tennis net—perceiving and responding without having to process its decision through ~300,000 lines of C++ code. I just KNOW to volley or duck in an instant; it just KNOWS to proceed, brake, or swerve in an instant.

Have I ever mistakenly let the ball go past me, only to watch it drop into the court rather than go long, resulting in the self-inflicted loss of that point? Of course. My intuition isn’t infallible. It became much better by age 86 than it was at age 12, but it could never be absolutely perfect.

So, too, with the Tesla self-driving system and the decisions it makes. That is why they not only upgrade it constantly, but also insist on labeling it “Full Self-Driving (Supervised)”—always including that “Supervised”. Because it can make mistakes.

Stick with me here.

There is a lot of mass media chatter about totally self-driving vehicles. Unsupervised vehicles. Waymo taxis are the most widely noted, but others are emerging. You may have had a ride in one yourself. In them, nobody is at the wheel. Maybe there isn’t even a wheel at all. Nobody is supervising. Nobody has taken responsibility for correcting any mistakes it might make. Boarding a Waymo is like stepping into an elevator and pushing the button for your particular floor. A couple of generations ago, an uniformed operator was stationed in every elevator to get you there. But both Waymos and elevators get along pretty well without human operators these days.

Right now, totally self-driving vehicles are controlled by a huge and prohibitively expensive array of technologies that restrict them to carefully curated zones and pathways that have been mapped with excruciatingly detailed images, all designed to ensure that the vehicle will not and cannot travel beyond its designated territory. Fine. It’s kind of like those self-driving cars are on rails. You’re okay stepping into an elevator without an operator, because it’s on rails, too, and you’re pretty sure it’s not going to take an unexpected left turn on the seventh floor and propel you out the window.

By dramatic contrast, the Tesla FSD (Supervised) has been designed to let a Tesla freely roam and operate anywhere wheeled vehicles are likely to go. Rather than its system interacting with a limited set of rules about predetermined routes with previously pre-identified situational factors, it guides the vehicle through whatever its physical situation actually is. That situation being determined not only by the 40,000,000 pixel inputs that update thirty times per second but also by what’s been learned by the gazillion pixels of prior travel by Tesla vehicles over those or similar roads and parking lots and driveways—the ones that have been continually recorded and analyzed at Tesla’s home base to enrich the critical mass of “experience” used for periodically upgrading the cars’ computers that intuitively control the vehicles.

And so that is how our new Tesla with FSD (Supervised) was recently able to drive Patti and me 852 miles on a round trip from our home in Princeton to our farmstead on a rural road in mid-coast Maine, and back again, without my ever touching the steering wheel or the pedals. (Most impressive moment? At one point near Somerville, N.J., it merged onto a five-lane highway from the left and deftly worked its way across all five lanes of heavy traffic to an exit ramp just a mile down the road.)

That unique ability to sense how to proceed under any circumstances is also why it cautiously edges through the cramped streets of Princeton where cyclists and pedestrians and runners unexpectedly pop up on the roadways, and parked-car doors suddenly swing open inches away. Or why it was able to deliver us from our driveway in Princeton into New York City, at highway speeds along the N.J. Turnpike, crawling through tunnels under the Hudson River, weaving through mindlessly aggressive traffic on the streets of Manhattan where the use of “lanes” is just a naïve hope, eventually depositing us inside a parking garage on 39^th Street.

The word we and our friends use to describe these experiences is “Amazing!!!”

But Tesla’s FSD (Supervised) DOES MAKE MISTAKES. That is why nobody should ever buy a Tesla with FSD (Supervised) unless they are totally committed to supervising it. I activate the self-driving system almost every time I use the car—(at the moment, that’s 86% of the 2,366 miles I have driven it since new). I absolutely delight in what it does to make me a safer driver, but I never assume it to be infallible. To drive home this point, I am going to describe several mistakes our Tesla FSD (Supervised) has already made in our first month of ownership:

Mistake #1: On a country road outside Princeton, road construction required the closure of the righthand lane in which we were driving. Ahead of us, workers had closed our lane and narrowed the road to one-way-only traffic. A worker stood at the near end of the construction zone with his hand-held sign (“Stop” on one side; “Slow” on the other.) We trusted the FSD (Supervised) to comprehend and obey the “Stop”, which it did. Then, when the worker flipped it to “Slow” to invite us to proceed down the one-way lane on the left, the car detected the change and acted accordingly. UNTIL it make a big mistake! As we were driving along that left lane, with the right lane blocked off for a quarter of a mile or so by periodic orange cones, FSD (Supervised) decided that the right lane—which was not visibly under repair right beside us—looked good enough to re-occupy, and it turned between two cones and resumed driving in that lane. The workers doing repairs well ahead of us looked up in alarm as we headed straight toward them.

I snatched the wheel and pulled us back through the cones onto the left lane—not more than one or two seconds after the car had swung into the closed lane behind the cones. (In every instance I’ll describe, my corrective action has been nearly instantaneous—something that surprises and reassures me.) In retrospect, I have no doubt that the car would have done the same thing well before smashing into the workers standing on the road ahead. But I’m guessing the workers themselves were somewhat less confident, as their unison flipping their middle fingers at me might suggest.

Mistake #2: The traffic light at the intersection of Witherspoon Street and Wiggins Street in downtown Princeton recently installed a “No Turn On Red” sign next to it. I use that intersection many times a week. Almost every time when the light is red for me, my Tesla waits for it to turn green. “Almost” isn’t every time. Several times, when traffic permitted, it has begun to make the turn while the light was still red. When that’s about to happen, I can feel the car twitch a bit as it considers whether to proceed or not. So I step on the brakes (which cancels self-driving) and that stops the impending turn before the car moves more than a foot or so. I then touch the button on the screen to reactivate FSD (Supervised), and the car waits until the light turns green. (Over time, my particular car’s computer will learn from my having corrected it and no longer attempt to turn on red at that corner. But even more to the point, the Tesla FSD (Supervised) upgrading system that constantly ingests information from every car in the fleet will incorporate my car’s corrected behavior into the Tesla “library”, so that as the entire fleet’s computers get continually updated, they will be equally enlightened about not turning on red at Witherspoon and Wiggins when in Princeton, New Jersey.)

Mistake #3: On the aforementioned trip to Manhattan, our car proceeded up 6^th Avenue to the intersection of 39^th Street , where it would turn left and enter our parking garage at 102 39^th Street. But there’s a bike lane running along on the lefthand side of 6^th Avenue, and it was empty, so the car chose to commence the left turn early by moving over into it about a hundred feet short of 39^th Street. I grabbed the wheel the instant it began to turn us into the bike lane, kept us in the car lane the last few feet to 39^th Street, then resumed self-driving to let it find and enter the parking garage. Again, no damage done—but further reinforcement for my being a supervisor.

Mistakes #4…to infinity: I am persistent in expressing my caution to those who are amazed by riding in my Tesla with FSD (Supervised) and are understandably drawn to getting one for themselves. I tell them that, on average, you may need to intercede about ONCE PER DAY. That is why it’s called FSD “Supervised” (and why I have chosen to use that full term every single time in this piece). The supervision is not onerous, but it is casually constant. You don’t need to nervously hover over it and “monitor” it. It is astonishingly trustworthy—99.9% of the time. But there is still that .1% of the time. So you do need to notice when it does something wrong. You’ll be amazed at how intuitive both the noticing and the responding turn out to be. For me, paying enough attention is surprisingly easy to do and doesn’t in the least diminish my monumental glee in its ability to make me a safer driver.

And after a while, I get a little sense of when it may go off kilter. Or can even predict it. Like, one thing it does wrong much of the time is at the very end of any trip—parking. When you commence FSD (Supervised), you tell the system where you want it to go, and also what to do when it gets there—park on the street, park in a driveway, park in a parking lot, or just “pull over”. But occasionally it’ll choose the driveway next door. Or pause just anywhere in a large parking lot, seemingly baffled by too many choices. Or park about a foot away from the curb, intentionally protecting the wheels from scuffing the curbstones, but leaving the car slightly farther into the street than I’d prefer. It’s not big deal: just take the wheel and finish the job yourself, which is what I do almost every time. And Tesla is constantly working on improvements. My FSD (Supervised) software gets automatically updated overnight every couple of days, and I know for a fact that “parking” is exceptionally high on their list of targets for improvement.

All that said, I am hugely predisposed to trust the system’s instantaneous decisions and corrective actions over my own. In my rookie status, I may have overreacted when interceding; more-experienced Tesla owners tell me they can’t recall the last time they ever interceded when “self-driving”. And after a few thousand miles of watching how it behaves, it seems unimaginable to me that it would ever occasion or tolerate a collision of its own making. In the specific mistakes I recounted above, embarrassing or illegal though they may have been, I am confident that the car would have avoided doing any damage if I had not intervened. After all, the car is checking out everything around it and adjusting its behavior every thirtieth of a second. I’m not.

That’s enough. Probably too much. And it’s not even a fraction of everything I learned about what is behind FSD (Supervised), which is only Level Two in the hierarchy of self-driving (with six levels eventually). There is a frantic effort among major companies to develop those four higher levels of self-driving, many of which may employ a withering constellation of other technologies I haven’t even touched on here—LiDAR, radar, ultrasonic, microphones, moisture and temperature sensors, GPS and satellite inputs—all in addition to the visual sensors used by Tesla for creating external images and for monitoring the driver’s attention to matters at hand. It does seem that the other manufacturers will be hard-pressed to ever find a way to surpass the value of Tesla’s incomparable treasury of images to inform computers’ intuitive conclusions and decisions.

But we shall see. It is clear to me that they will all do whatever they possibly can in relentless pursuit of their one common goal: make human drivers as inconsequential as uniformed elevator operators.

Next time: I will conclude this little series by sharing how relinquishing control of driving the car feels to me, and how that experience has actually changed me in surprising ways.