Transportation

Self-Driving Cars And The Kobayashi Maru

Comment

Image Credits: Rihardzz (opens in a new window) / Shutterstock (opens in a new window)

Andrew Heikkila

Contributor

Andrew Heikkila is a tech enthusiast and writer from Boise, Idaho.

More posts from Andrew Heikkila

In 1966, Gene Roddenberry’s Star Trek would boldly go where no man had gone before, telling the tale of Captain Kirk and his crew as they explored the galaxy while taking on myriad sci-fi adventures.

In the opening scene of the franchise’s 1982 motion picture, Star Trek II: The Wrath of Khan, the U.S.S. Enterprise responds to a distress call from another ship, the Kobayashi Maru. Stranded in an area of space that the Enterprise can’t enter without risking interstellar war, the limping ship has almost 400 souls on board and is quickly losing life support. These people are going to die without help; the captain has an impossible choice to make.

The scene is later shown to be an unwinnable simulation, created as part of a training scenario. Deciding to not aid the Kobayashi Maru results in the death of its crew and passengers. However, acting to help the stranded ship will trigger conflict and result in the death and destruction of the Enterprise. The theme of a no-win scenario is prevalent throughout the rest of the film, and many Star Trek fans have colloquially come to call “damned if you do, damned if you don’t” situations by the name of the ship: Kobayashi Maru.

The idea of the no-win situation has gotten more attention over the last couple of years, as Google has been making strides with the driverless vehicle and Apple is rumored to be getting into the same market. But how does the Kobayashi Maru relate to self-driving automobiles?

Imagine you are driving down the road and you suddenly find yourself boxed in. In front of you is a large semi-truck with heavy crates on the back, to your right is a person on a motorcycle and to your left is a big SUV. All of a sudden, one of the crates falls off the back of the semi, directly in your path. What do you do?

If you swerve to the right, you’ll live, but the move would probably end up costing the person on the motorcycle their life. If you swerve left, you’ll collide with the SUV and possibly kill both yourself and its inhabitants — but there’s still a chance you’ll all survive the incident (albeit sustaining injury) because of the SUV’s high safety ratings. If you don’t swerve either way, you won’t injure anybody, but you’re definitely going to wreck and possibly die. So what should a driver do in this situation? What is the right answer?

This scenario comes from TEDEd, and is meant simply to illustrate that there is no right answer, especially in a scenario where there is little time to think. Each choice has a negative consequence, and the driver simply has to determine which option is, in their mind, the lesser of the evils.

Unfortunately, a person’s reactions in situations like these are more instinctual than they are based on decision or logic, simply because humans can’t process information that fast. Computers, on the other hand, can.

The driverless car as an invention has the potential to prevent approximately 1.3 million deaths annually, as well as between 20 and 50 million injuries, according to ASIRT. They are able to network with other smart cars and stop lights so that 151 million Americans can get to work faster and more safely. Because machines don’t blink. They don’t sleep or get drowsy. Machines don’t get drunk and drive.

In the only accident to date involving a self-driving car, it was determined humans were at fault, not machines — and yet, therein lies the problem. Accidents will happen, and a computer must be programmed to react in those situations, sometimes when death is inevitable. In those instances, it’s succinct to say that we’ll have to program computers to kill.

Let’s take a look at another scenario. There is a thought experiment called the trolley problem that asks you to imagine a runaway trolley headed for a group of five people tied up in its path. You’re standing near a lever, however, that will send the trolley to a different set of tracks if you flip it — the only problem is that there is a person tied up on those tracks, as well. You have two options: Do nothing, letting the trolley kill all five people on the main track, or flip the switch and send the trolley to the side track where it will kill one person.

In the most recent iteration of this problem, facilitated by researchers at Michigan State University, 147 subjects were given 3D headsets so they could actually experience this dilemma in an environment as close to reality as possible. Ninety percent of the participants flipped the switch, saving five people to kill one. This isn’t that surprising, as most people would say that five lives saved over one is ethically the right choice — but what happens when we switch the problem up a little bit?

Let’s say there is no side track the trolley will divert to if you flip the switch; instead, you’re standing next to a person large enough to stop the vehicle. The only caveat is that you must push him onto the track. The second variation of the problem produces different results, because there is a perceived difference between killing somebody and letting them die. The trend you come across is that not as many people would choose to kill the large man, even if it meant saving more lives overall, because they don’t want to be held personally responsible for his death.

Here’s a third scenario: What if you were the large person that could stop the trolley via self-sacrifice? Even better, what if your self-driving car turns a corner only to see a crowd of five people standing in the road? Your car either can hit them, sparing your own life, or the onboard AI can run your car off of the road, killing you and saving five lives.

If you answered that flipping the switch in the first iteration of the trolley problem was the right choice, because one death is better than five, then logically you would agree that your self-sacrifice is necessary to save the lives of the five people in the road ahead of you, right? Interestingly, if you defy the framework of logic and would rather choose self-preservation in this scenario, you’re actually in the majority.

Jean-Francois Bonnefon and the Toulouse School of Economics in France concluded from their own studies that these types of logical fallacies run rampant. As such, they believe it will be interesting to watch public opinion inevitably play a role in deciding how the ethics of AI works. Says Bonnefon and company: “[Participants of our study] actually wished others to cruise in utilitarian autonomous vehicles, more than they wanted to buy utilitarian autonomous vehicles themselves.”

Essentially, the problem is that people actually want driverless cars to sacrifice the occupant in favor of saving a higher number of lives — but only if they don’t have to drive one themselves. Unfortunately, the biggest catch-22 is that people won’t buy autonomous vehicles if they’re designed to kill their passengers, meaning that the status quo allowing split-second human decisions will continue to define accidents and reactions around the world. If we never legalize self-driving cars, our own human driving will continue to contribute to more than a million deaths globally.

Employing a fourth scenario, Robohub.org ran a reader poll that showed similar results trend toward self-preservation: You’re driving through a tunnel and a child appears at the opening and trips, blocking your exit. You can’t stop, so you’re left with the choice of swerving into a wall to save the child, or running over the child to save yourself. Of 110 people polled, 64 percent said they would continue straight and kill the child.

When asked which entities should determine how an autonomous car responds to the tunnel problem, 44 percent of respondents thought it should be the passenger of the vehicle, while 33 percent thought it should be lawmakers. Twelve percent thought the manufacturers or designers should be burdened with that choice; 11 percent responded “other.”

Determining who will control these “ethical settings” that guide no-win responses is a huge problem that self-driving cars are going to have to face in terms of liability. Because if a car will have to be programmed to choose between two lives, that means whoever decides how the algorithm is going to function is also possibly condemning to death either bystanders or passengers.

This type of “predetermined” action, an algorithm that chooses to spare children over adults, for example, would almost vicariously put the programmer in the driver’s seat, lending truth to the Department of Public Safety’s comment that a self-driven car will always have a “determinable human operator.”

Insurance companies are going to have to wrestle with that one, because in any instance, somebody will be liable if an autonomous vehicle gets into a wreck. If you get to decide on your car’s ethics settings and decide to continue straight and kill the child in the tunnel situation, does that make you liable for that child’s death? If it’s left up to the auto company, will they be liable?

The repercussions of these decisions extend much further into the future than anybody is able to foresee. As artificial intelligence advances, it may very well use the programmable ethics settings found in self-driving cars as a platform to build upon. Isaac Asimov once suggested there should be three laws of robotics that govern AI:

  • A robot may not injure a human being or, through inaction, allow a human being to come to harm.
  • A robot must obey orders given it by human beings except where such orders would conflict with the first law.
  • A robot must protect its own existence as long as such protection does not conflict with the first or second laws.

Obviously, the first law doesn’t work in this context, and is in danger of being trampled by militaries the world over searching for autonomous soldiers and vehicles (such as assault drones). Weapons aside, there are obvious reasons explored above that an AI would inevitably have to break the First Law of Robotics when faced with a Kobayashi Maru, and however we decide they should respond may constitute some kind of basis for how AI develops and writes its own ethical programming in the future.

If we determined today that favoring “quantity of lives” is the sole rule to follow for self-driving cars, for example, a much more developed, Skynet-esque AI of the future might calculate that citizens of industrialized countries are making the world uninhabitable for a majority of people and their many generations of offspring. Ethically, that AI could justify eradicating a large swath of the population so that an even larger percent can live.

Of course, there are much more immediate concerns that we’ll have to deal with in response to the self-driving car. What is going to happen to everybody in the trucking industry? Or to cab and Uber drivers? What happens if somebody remotely hijacks your car via the Internet of Things and crashes it with you in it?

The autonomous vehicle is still in the very early stages of development, but the way we decide to build its AI will set precedents. Unfortunately, we’re a species that still fights wars over land and money, that murders over passion and justifies the actions of the wicked.

On the other hand, we do have the capacity to love and sacrifice self for causes greater than our own. Sometimes it seems like we’re these creatures trying to program ethics into machines, when, in reality, we barely seem to understand or practice ethical behavior ourselves.

Nevertheless, we have an opportunity here to discuss these ethics and decide what type of character we want to define humankind, collectively, when faced with a Kobayashi Maru. Only once we’ve done our soul-searching and overcome that obstacle will we be able to follow in the footsteps of Captain Kirk, and bravely go where no man has gone before.

More TechCrunch

Avendus, the top investment bank for venture deals in India, confirmed on Wednesday it is looking to raise up to $350 million for its new private equity fund.  The new…

Avendus, India’s top venture advisor, confirms it’s looking to raise a $350 million fund

China has closed a third state-backed investment fund to bolster its semiconductor industry and reduce reliance on other nations, both for using and for manufacturing wafers — prioritizing what is…

China’s $47B semiconductor fund puts chip sovereignty front and center

Apple’s annual list of what it considers the best and most innovative software available on its platform is turning its attention to the little guy.

Apple’s Design Awards nominees highlight indies and startups, largely ignore AI (except for Arc)

The spyware maker’s founder, Bryan Fleming, said pcTattletale is “out of business and completely done,” following a data breach.

Spyware maker pcTattletale says it’s ‘out of business’ and shuts down after data breach

AI models are always surprising us, not just in what they can do, but what they can’t, and why. An interesting new behavior is both superficial and revealing about these…

AI models have favorite numbers, because they think they’re people

On Friday, Pal Kovacs was listening to the long-awaited new album from rock and metal giants Bring Me The Horizon when he noticed a strange sound at the end of…

Rock band’s hidden hacking-themed website gets hacked

Jan Leike, a leading AI researcher who earlier this month resigned from OpenAI before publicly criticizing the company’s approach to AI safety, has joined OpenAI rival Anthropic to lead a…

Anthropic hires former OpenAI safety lead to head up new team

Welcome to TechCrunch Fintech! This week, we’re looking at the long-term implications of Synapse’s bankruptcy on the fintech sector, Majority’s impressive ARR milestone, and more!  To get a roundup of…

The demise of BaaS fintech Synapse could derail the funding prospects for other startups in the space

YouTube’s free Playables don’t directly challenge the app store model or break Apple’s rules. However, they do compete with the App Store’s free games.

YouTube’s free games catalog ‘Playables’ rolls out to all users

Featured Article

A comprehensive list of 2024 tech layoffs

The tech layoff wave is still going strong in 2024. Following significant workforce reductions in 2022 and 2023, this year has already seen 60,000 job cuts across 254 companies, according to independent layoffs tracker Layoffs.fyi. Companies like Tesla, Amazon, Google, TikTok, Snap and Microsoft have conducted sizable layoffs in the first months of 2024. Smaller-sized…

11 hours ago
A comprehensive list of 2024 tech layoffs

OpenAI has formed a new committee to oversee “critical” safety and security decisions related to the company’s projects and operations. But, in a move that’s sure to raise the ire…

OpenAI’s new safety committee is made up of all insiders

Time is running out for tech enthusiasts and entrepreneurs to secure their early-bird tickets for TechCrunch Disrupt 2024! With only four days left until the May 31 deadline, now is…

Early bird gets the savings — 4 days left for Disrupt sale

AI may not be up to the task of replacing Google Search just yet, but it can be useful in more specific contexts — including handling the drudgery that comes…

Skej’s AI meeting scheduling assistant works like adding an EA to your email

Faircado has built a browser extension that suggests pre-owned alternatives for ecommerce listings.

Faircado raises $3M to nudge people to buy pre-owned goods

Tumblr, the blogging site acquired twice, is launching its “Communities” feature in open beta, the Tumblr Labs division has announced. The feature offers a dedicated space for users to connect…

Tumblr launches its semi-private Communities in open beta

Remittances from workers in the U.S. to their families and friends in Latin America amounted to $155 billion in 2023. With such a huge opportunity, banks, money transfer companies, retailers,…

Félix Pago raises $15.5 million to help Latino workers send money home via WhatsApp

Google said today it’s adding new AI-powered features such as a writing assistant and a wallpaper creator and providing easy access to Gemini chatbot to its Chromebook Plus line of…

Google adds AI-powered features to Chromebook

The dynamic duo behind the Grammy Award–winning music group the Chainsmokers, Alex Pall and Drew Taggart, are set to bring their entrepreneurial expertise to TechCrunch Disrupt 2024. Known for their…

The Chainsmokers light up Disrupt 2024

The deal will give LumApps a big nest egg to make acquisitions and scale its business.

LumApps, the French ‘intranet super app,’ sells majority stake to Bridgepoint in a $650M deal

Featured Article

More neobanks are becoming mobile networks — and Nubank wants a piece of the action

Nubank is taking its first tentative steps into the mobile network realm, as the NYSE-traded Brazilian neobank rolls out an eSIM (embedded SIM) service for travelers. The service will give customers access to 10GB of free roaming internet in more than 40 countries without having to switch out their own existing physical SIM card or…

18 hours ago
More neobanks are becoming mobile networks — and Nubank wants a piece of the action

Infra.Market, an Indian startup that helps construction and real estate firms procure materials, has raised $50M from MARS Unicorn Fund.

MARS doubles down on India’s Infra.Market with new $50M investment

Small operations can lose customers by not offering financing, something the Berlin-based startup wants to change.

Cloover wants to speed solar adoption by helping installers finance new sales

India’s Adani Group is in discussions to venture into digital payments and e-commerce, according to a report.

Adani looks to battle Reliance, Walmart in India’s e-commerce, payments race, report says

Ledger, a French startup mostly known for its secure crypto hardware wallets, has started shipping new wallets nearly 18 months after announcing the latest Ledger Stax devices. The updated wallet…

Ledger starts shipping its high-end hardware crypto wallet

A data protection taskforce that’s spent over a year considering how the European Union’s data protection rulebook applies to OpenAI’s viral chatbot, ChatGPT, reported preliminary conclusions Friday. The top-line takeaway…

EU’s ChatGPT taskforce offers first look at detangling the AI chatbot’s privacy compliance

Here’s a shoutout to LatAm early-stage startup founders! We want YOU to apply for the Startup Battlefield 200 at TechCrunch Disrupt 2024. But you’d better hurry — time is running…

LatAm startups: Apply to Startup Battlefield 200

The countdown to early-bird savings for TechCrunch Disrupt, taking place October 28–30 in San Francisco, continues. You have just five days left to save up to $800 on the price…

5 days left to get your early-bird Disrupt passes

Venture investment into Spanish startups also held up quite well, with €2.2 billion raised across some 850 funding rounds.

Spanish startups reached €100 billion in aggregate value last year

Featured Article

Onyx Motorbikes was in trouble — and then its 37-year-old owner died

James Khatiblou, the owner and CEO of Onyx Motorbikes, was watching his e-bike startup fall apart.  Onyx was being evicted from its warehouse in El Segundo, near Los Angeles. The company’s unpaid bills were stacking up. Its chief operating officer had abruptly resigned. A shipment of around 100 CTY2 dirt bikes from Chinese supplier Suzhou…

2 days ago
Onyx Motorbikes was in trouble — and then its 37-year-old owner died

Featured Article

Iyo thinks its GenAI earbuds can succeed where Humane and Rabbit stumbled

Iyo represents a third form factor in the push to deliver standalone generative AI devices: Bluetooth earbuds.

2 days ago
Iyo thinks its GenAI earbuds can succeed where Humane and Rabbit stumbled