Self-Driving Cars And The Kobayashi Maru

In 1966, Gene Roddenberry’s Star Trek would boldly go where no man had gone before, telling the tale of Captain Kirk and his crew as they explored the galaxy while taking on myriad sci-fi adventures.

In the opening scene of the franchise’s 1982 motion picture, Star Trek II: The Wrath of Khan, the U.S.S. Enterprise responds to a distress call from another ship, the Kobayashi Maru. Stranded in an area of space that the Enterprise can’t enter without risking interstellar war, the limping ship has almost 400 souls on board and is quickly losing life support. These people are going to die without help; the captain has an impossible choice to make.

The scene is later shown to be an unwinnable simulation, created as part of a training scenario. Deciding to not aid the Kobayashi Maru results in the death of its crew and passengers. However, acting to help the stranded ship will trigger conflict and result in the death and destruction of the Enterprise. The theme of a no-win scenario is prevalent throughout the rest of the film, and many Star Trek fans have colloquially come to call “damned if you do, damned if you don’t” situations by the name of the ship: Kobayashi Maru.

The idea of the no-win situation has gotten more attention over the last couple of years, as Google has been making strides with the driverless vehicle and Apple is rumored to be getting into the same market. But how does the Kobayashi Maru relate to self-driving automobiles?

Imagine you are driving down the road and you suddenly find yourself boxed in. In front of you is a large semi-truck with heavy crates on the back, to your right is a person on a motorcycle and to your left is a big SUV. All of a sudden, one of the crates falls off the back of the semi, directly in your path. What do you do?

If you swerve to the right, you’ll live, but the move would probably end up costing the person on the motorcycle their life. If you swerve left, you’ll collide with the SUV and possibly kill both yourself and its inhabitants — but there’s still a chance you’ll all survive the incident (albeit sustaining injury) because of the SUV’s high safety ratings. If you don’t swerve either way, you won’t injure anybody, but you’re definitely going to wreck and possibly die. So what should a driver do in this situation? What is the right answer?

This scenario comes from TEDEd, and is meant simply to illustrate that there is no right answer, especially in a scenario where there is little time to think. Each choice has a negative consequence, and the driver simply has to determine which option is, in their mind, the lesser of the evils.

Unfortunately, a person’s reactions in situations like these are more instinctual than they are based on decision or logic, simply because humans can’t process information that fast. Computers, on the other hand, can.

The driverless car as an invention has the potential to prevent approximately 1.3 million deaths annually, as well as between 20 and 50 million injuries, according to ASIRT. They are able to network with other smart cars and stop lights so that 151 million Americans can get to work faster and more safely. Because machines don’t blink. They don’t sleep or get drowsy. Machines don’t get drunk and drive.

In the only accident to date involving a self-driving car, it was determined humans were at fault, not machines — and yet, therein lies the problem. Accidents will happen, and a computer must be programmed to react in those situations, sometimes when death is inevitable. In those instances, it’s succinct to say that we’ll have to program computers to kill.

Let’s take a look at another scenario. There is a thought experiment called the trolley problem that asks you to imagine a runaway trolley headed for a group of five people tied up in its path. You’re standing near a lever, however, that will send the trolley to a different set of tracks if you flip it — the only problem is that there is a person tied up on those tracks, as well. You have two options: Do nothing, letting the trolley kill all five people on the main track, or flip the switch and send the trolley to the side track where it will kill one person.

In the most recent iteration of this problem, facilitated by researchers at Michigan State University, 147 subjects were given 3D headsets so they could actually experience this dilemma in an environment as close to reality as possible. Ninety percent of the participants flipped the switch, saving five people to kill one. This isn’t that surprising, as most people would say that five lives saved over one is ethically the right choice — but what happens when we switch the problem up a little bit?

Let’s say there is no side track the trolley will divert to if you flip the switch; instead, you’re standing next to a person large enough to stop the vehicle. The only caveat is that you must push him onto the track. The second variation of the problem produces different results, because there is a perceived difference between killing somebody and letting them die. The trend you come across is that not as many people would choose to kill the large man, even if it meant saving more lives overall, because they don’t want to be held personally responsible for his death.

Here’s a third scenario: What if you were the large person that could stop the trolley via self-sacrifice? Even better, what if your self-driving car turns a corner only to see a crowd of five people standing in the road? Your car either can hit them, sparing your own life, or the onboard AI can run your car off of the road, killing you and saving five lives.

If you answered that flipping the switch in the first iteration of the trolley problem was the right choice, because one death is better than five, then logically you would agree that your self-sacrifice is necessary to save the lives of the five people in the road ahead of you, right? Interestingly, if you defy the framework of logic and would rather choose self-preservation in this scenario, you’re actually in the majority.

Jean-Francois Bonnefon and the Toulouse School of Economics in France concluded from their own studies that these types of logical fallacies run rampant. As such, they believe it will be interesting to watch public opinion inevitably play a role in deciding how the ethics of AI works. Says Bonnefon and company: “[Participants of our study] actually wished others to cruise in utilitarian autonomous vehicles, more than they wanted to buy utilitarian autonomous vehicles themselves.”

Essentially, the problem is that people actually want driverless cars to sacrifice the occupant in favor of saving a higher number of lives — but only if they don’t have to drive one themselves. Unfortunately, the biggest catch-22 is that people won’t buy autonomous vehicles if they’re designed to kill their passengers, meaning that the status quo allowing split-second human decisions will continue to define accidents and reactions around the world. If we never legalize self-driving cars, our own human driving will continue to contribute to more than a million deaths globally.

Employing a fourth scenario, Robohub.org ran a reader poll that showed similar results trend toward self-preservation: You’re driving through a tunnel and a child appears at the opening and trips, blocking your exit. You can’t stop, so you’re left with the choice of swerving into a wall to save the child, or running over the child to save yourself. Of 110 people polled, 64 percent said they would continue straight and kill the child.

When asked which entities should determine how an autonomous car responds to the tunnel problem, 44 percent of respondents thought it should be the passenger of the vehicle, while 33 percent thought it should be lawmakers. Twelve percent thought the manufacturers or designers should be burdened with that choice; 11 percent responded “other.”

Determining who will control these “ethical settings” that guide no-win responses is a huge problem that self-driving cars are going to have to face in terms of liability. Because if a car will have to be programmed to choose between two lives, that means whoever decides how the algorithm is going to function is also possibly condemning to death either bystanders or passengers.

This type of “predetermined” action, an algorithm that chooses to spare children over adults, for example, would almost vicariously put the programmer in the driver’s seat, lending truth to the Department of Public Safety’s comment that a self-driven car will always have a “determinable human operator.”

Insurance companies are going to have to wrestle with that one, because in any instance, somebody will be liable if an autonomous vehicle gets into a wreck. If you get to decide on your car’s ethics settings and decide to continue straight and kill the child in the tunnel situation, does that make you liable for that child’s death? If it’s left up to the auto company, will they be liable?

The repercussions of these decisions extend much further into the future than anybody is able to foresee. As artificial intelligence advances, it may very well use the programmable ethics settings found in self-driving cars as a platform to build upon. Isaac Asimov once suggested there should be three laws of robotics that govern AI:

• A robot may not injure a human being or, through inaction, allow a human being to come to harm.
• A robot must obey orders given it by human beings except where such orders would conflict with the first law.
• A robot must protect its own existence as long as such protection does not conflict with the first or second laws.

Obviously, the first law doesn’t work in this context, and is in danger of being trampled by militaries the world over searching for autonomous soldiers and vehicles (such as assault drones). Weapons aside, there are obvious reasons explored above that an AI would inevitably have to break the First Law of Robotics when faced with a Kobayashi Maru, and however we decide they should respond may constitute some kind of basis for how AI develops and writes its own ethical programming in the future.

If we determined today that favoring “quantity of lives” is the sole rule to follow for self-driving cars, for example, a much more developed, Skynet-esque AI of the future might calculate that citizens of industrialized countries are making the world uninhabitable for a majority of people and their many generations of offspring. Ethically, that AI could justify eradicating a large swath of the population so that an even larger percent can live.

Of course, there are much more immediate concerns that we’ll have to deal with in response to the self-driving car. What is going to happen to everybody in the trucking industry? Or to cab and Uber drivers? What happens if somebody remotely hijacks your car via the Internet of Things and crashes it with you in it?

The autonomous vehicle is still in the very early stages of development, but the way we decide to build its AI will set precedents. Unfortunately, we’re a species that still fights wars over land and money, that murders over passion and justifies the actions of the wicked.

On the other hand, we do have the capacity to love and sacrifice self for causes greater than our own. Sometimes it seems like we’re these creatures trying to program ethics into machines, when, in reality, we barely seem to understand or practice ethical behavior ourselves.

Nevertheless, we have an opportunity here to discuss these ethics and decide what type of character we want to define humankind, collectively, when faced with a Kobayashi Maru. Only once we’ve done our soul-searching and overcome that obstacle will we be able to follow in the footsteps of Captain Kirk, and bravely go where no man has gone before.