A chain saw, sporting all the safety interlocks, might still kill you if you use it carelessly. You’re self-confident and you suffer the usual optimism bias. Do you buy the chainsaw?
A driverless car (autonomous vehicle, or AV) is programmed to kill you under certain conditions. It’s clear, up front, that those conditions will be beyond your control; your self-confidence is irrelevant. Will you pay for this machine? Is the manufacturer insane to believe you would buy your own possible execution?
The second scenario refers to the Trolley Problem – the (usually) hypothetical dilemma in which you must choose to sacrifice your own life in a traffic situation, or kill one or more pedestrians. Recent articles have put forward the view that the Trolley Problem inevitably will be faced by AVs, and must be considered in their programming.
I believe the two scenarios describe two different psychological situations and lead to two different customer decisions. As the question seems urgent – all signs point to AVs appearing in the marketplace soon and inevitably – I have asked a guest blogger to add his views to mine.
Scenario 1 is a common situation. We often buy products that endanger us at moments when our skill and alertness – or the skill and alertness of those around us – are not near optimum. Ordinary automobiles are a perfect example. We pay our money, we do our best, and we take our chances. This is how humans have acted for thousands of years.
Cigarettes present a situation more similar, but not identical, to AVs. Smoking-related cancer is a crap-shoot; it may happen sooner, later, or not at all. The difference is that only tobacco addicts take the gamble. AVs will protect or endanger everyone who drives, rides or walks on or near a roadway.
Regular readers know I am a technological optimist. Today though, I am declaring that AVs in their currently envisioned form (i.e., on streets rather than on tracks) are unsuited for the market. Readers will also notice that this management blog usually deals with uncertainty reduction, a goal beloved by managers. I’m now highlighting a case where we embrace uncertainty. Why would humans embrace uncertainty? The reason that’s pertinent here is, “It’s the only way we can indulge our optimism bias.”
A Public Broadcasting System web site lets you self-test on the Trolley Problem and other moral choices. When someone – even PBS! – asks you the Trolley question, they commit a fallacy by assuming only two possible courses of action. If you, as subject, go along with the assumption, all you’re doing is acting as the questioner’s enabler. (Who’s testing whom, really?) The best possible reply to the trolley problem is, "I don't do hypotheticals." There are always unique and extenuating conditions in every real instance of risk, including traffic risk. There is always a third choice.
This makes one ask whether a machine can react appropriately to the unique conditions, and take that third or fourth option. I believe they cannot. First, static programming can prepare the machine to deal only with foreseen circumstances.
The previous sentence was just a scene-setting straw man; there will be no static programming. The central program governing the whole population of deployed AVs will “learn” and modify itself as each vehicle encounters new conditions. One problem though: The program’s creators won’t know what has been learned, except by watching the cars’ movements. The cars’ adaptive behavior will become unpredictable, and perhaps detrimental to human safety.
Okay, human drivers can be unpredictable and dangerous too. But there are additional problems with the AVs.
They are, for most of their operating time, autonomous from human control. However, they are not autonomous from each other. AVs communicate with each other and with a central computer program. This means the vendor’s challenge is not just to control an individual vehicle, but to control the entire network of vehicles.
The testing of small AV networks has gone fairly well, with only one collision I recall hearing about.* The tests won’t scale to the mass market, however, because the complexity of the system, and thus the chance of system breakdown, increases super-linearly with the number of networked vehicles.
The AV programmers may be trying to emulate flocking behavior. Each bird in a flock, following a few simple rules (for example, “Remain about two wing-lengths away from the bird on your left”), creates highly coherent mass behavior. The problems with translating this to AV networks are (i) the rules for avoiding other AVs, pedestrians, buildings, etc., have to be more complex than for schools of fish or flocks of birds; (ii) AVs will sometimes revert to the human driver’s control; and (iii) tests have shown the most accident-prone moments are the instants during that changeover from machine to human control. Some manufacturers are talking about AVs that will not allow a human driver to take control. I don’t know whether that is more scary or less, but I don’t think it would be practical for our transportation needs.
Consumers can’t be sure the computer program’s prime directive is to protect human life. Programmers may have given the AV network a directive to smooth traffic flow, or to maximize rider convenience. (Look at the pharmaceutical industry for examples of how things can go wrong in terms of prime directive.) Indeed, though I have not had time to fact-check it before writing this column, it’s my impression that articles on AVs tout life-saving as a by-product of autonomous vehicle deployment, not as the primary purpose.
Bonnefon et al (2015) surveyed public attitudes about AVs programmed to solve the Trolley Problem using the “Utilitarian” principle.
“Although [respondents] were generally unwilling to see self-sacrifices enforced by law, they were more prepared for such legal enforcement if it applied to AVs, than if it applied to humans. Several reasons may underlie this effect: unlike humans, computers can be expected to dispassionately make utilitarian calculations in an instant; computers, unlike humans, can be expected to unerringly comply with the law, rendering moot the thorny issue of punishing non-compliers; and finally, a law requiring people to kill themselves would raise considerable ethical challenges.
“Even in the absence of legal enforcement, most respondents agreed that AVs should be programmed for utilitarian self-sacrifice, and to pursue the greater good rather than protect their own passenger. However, they were not as confident that AVs would be programmed that way in reality—and for a good reason: They actually wished others to cruise in utilitarian AVs, more than they wanted to buy utilitarian AVs themselves.”
News items published after I began drafting this column suggest AVs will be rented by endusers, not purchased. That is, the manufacturers are targeting the taxi and Über markets. OK, but what’s really a horse of a different color is that they are pushing for urban districts in which human-driven vehicles will be prohibited. Seen from one angle, the logic is clear: In a collision between an AV and an ordinary vehicle, it’s easier to blame a machine than a human driver, so the liability issues get big. Seen from another angle, it raises a fundamental question about our future: Are our cities for machines, or are they for people?
In certain (admittedly, probably rare) AV crisis situations, you’re dead, and you know this in advance. When you are driving, there is always hope. Where there’s life, there’s hope. Where there’s machines, who knows? Manufacturers are rushing AVs to market without due regard for the human psychology that makes AVs unworkable.**
Now, over to my guest, who adds still another important dimension to the argument. Joe Rabinovitsj makes the dark implication that the easy market for AVs will be people who don’t mind – probably because they are accustomed to it – feeling helpless.
Agency and Autonomous Vehicles: a Hopeless Deal; a Response to Fred Phillips
There is a consideration that our discussion of psychological hang-ups presupposes — that of agency. Specifically, I argue that whether one will have an optimism bias toward the dangers of AVs is really a matter of whether one feels in control of one’s actions when facing those dangers.
Why is it important that the our possible inability to form optimism bias toward the dangers of AVs depends on agency? AVs are distinct from other dangerous technology products in a fundamental way. Namely, AVs prevent their users from feeling like they have control over those dangers, as opposed to many other dangerous consumer products that allow consumers to at least feel as though they can control the possibility of being harmed.
Let me consider Dr. Phillips’ two examples in greater detail. First we have the chainsaw. As we have mentioned, when we walk into the hardware store and buy a chainsaw we probably know how dangerous it is (and if not, we should probably watch Texas Chainsaw Massacre for more information). But, as Dr. Phillips suggests there is uncertainty about whether you will get hurt by that chainsaw. This uncertainty likely leads to the optimistic belief that you won’t get hurt — what we have called the optimism bias — and allows you to buy the chainsaw despite its dangers.
But it looks like we’ve missed a step in our analysis: why exactly are we uncertain that a chainsaw will hurt us? This question leaves open why exactly we develop an optimism bias in this case.
We see an analogous unanswered question in our example of purchasing an AV. Indeed it will be pre-determined by an AV’s programming whether or not it will hurt us under certain conditions. But when deciding to purchase an AV, it is still uncertain that we will ever encounter those circumstances and therefore it is also in some sense uncertain whether or not we will get hurt. So we have uncertainty of harm in both the chainsaw and AV cases but optimism bias in the former and not in the latter. So, if we are committed to the idea that AVs resist the formation of optimism bias towards their dangers, it is unlikely that uncertainty of harm is the only factor determining our lack of optimism bias in these cases.
I would like to propose that considering issues of agency can help explain why we seem inclined to form an optimism bias when it comes to the dangers associated with AVs rather than chainsaws. I am neither trying to nor am equipped to provide a fully fleshed-out, general theory of agency. Rather, I only intend to make claims about the extent to which people feel in control of their actions in certain situations.
Why does the uncertainty that a chainsaw will hurt us inspire an optimism bias? Part of what may lead us to form the optimistic belief that we won't get hurt by the chainsaw, despite its potential dangers, is that we feel confident in our ability to handle a chainsaw. This idea that degrees of confidence in one’s ability to mitigate uncertain danger seems to explain why certain people purchase dangerous equipment and others don’t. Many people do not feel comfortable in their ability to handle a chainsaw, which may help explain why a lot of people don’t form optimism biases about the dangers of chainsaws and therefore don’t purchase them. On the other hand, professional loggers probably don’t think twice about purchasing chainsaws because they are so confident in their ability to use them, despite the fact that even loggers are injured by chainsaws.
But what about AVs? Talking about degrees of confidence in one’s ability to mitigate the dangers of AVs at first glance seems silly. There is nothing we can do to prevent an AV from killing us in cases in which it is programmed to do so: we cannot mitigate the dangers associated with AVs. But that precisely is the interesting point. This feeling of helplessness before we even set foot in an AV could explain why we would be unlikely to develop an optimism bias towards the dangers of AVs.
What is this feeling of helplessness really a feeling of? This feeling seems rooted in a question of agency — of how much control we have over our actions. This also could explain the development of an optimism bias when it comes to the dangers of chainsaws: the degree of confidence we have in our ability to mitigate the dangers of a chainsaw is an expression of how much control we feel we have over our actions.
Where, though, does this feeling of helplessness — the feeling of a lack of agency — come from? Of course, considerations of human psychology have something to do with it. And if we are to take seriously the lack of optimism bias formation as a problem for AVs as an emerging technology, study of these considerations of human psychology will be helpful. But this I leave to the cognitive scientists, psychologists, neuroscientists, and decision theorists. The point that I want to offer to the broader discussion about autonomous vehicles and add on to Dr. Phillips’ suggestion is that there is something about AV technology itself that may cause individuals to feel this helplessness and be unable to develop optimism biases against the dangers of AVs in contrast to the way they can develop optimism bias towards the dangers of other technologies. And this ‘something’ seems to be that, unlike other dangerous technologies, AV technology takes the metaphorical and literal wheel out of our hands when it comes to dangerous situations. It is an AV’s source code rather than our shaky — but nonetheless our — hands that may steer us to our deaths. It is this aspect of AV technology, I believe, that makes us feel helpless at its prospect.
- Joe Rabinovitsj 4/24/16
* By some accounts, that collision involved a human driver crashing into an AV. The manufacturer claimed this incident revealed nothing wrong with the AV system. I heard this kind of upside-down logic once before, when karate master Mas Oyama knocked out a bull by punching it in the head. Oyama’s detractors said, “Nah, the bull was drugged.” As if just anybody could punch out even a drugged bull.
**My lecture slides on other aspects of AV technology assessment are downloadable from Slideshare, http://www.slideshare.net/fredphillips/the-selfdriving-car?qid=0c2ac803-13a0-4ae1-9dcb-5cb56e6d9a0b&v=&b=&from_search=1
Reference
J.-F. Bonnefon, A. Shariff and I. Rahwan (2015) Autonomous Vehicles Need Experimental Ethics: Are We Ready for Utilitarian Cars? http://arxiv.org/pdf/1510.03346v1.pdf
Comments