Two Penn professors explore whether
algorithms should be
made more ethical.
When people marvel at the things computers can do, what they are really talking about much of the time is the power of algorithms—precise sets of instructions telling the computer how to complete a well-defined task.
Algorithms are already common in such important arenas as criminal justice, matchmaking, lending, investing, and insuring against risk. Some are so advanced they can even learn on their own. And they raise difficult questions. If gender can help predict the likelihood of a driver having an accident, should algorithms incorporate that factor? How about race? How much accuracy are we willing to forego for the sake of our principles when lives are at stake?
In their recent book, The Ethical Algorithm: The Science of Socially Aware Algorithm Design, Michael Kearns, the National Center Professor of Management and Technology in the Department of Computer and Information Science, and Aaron Roth, a professor in the department, examine the ways algorithms, by doing precisely what we have asked them to do, can deliver results that contravene our ideals. To combat this, the authors argue, algorithm design should explicitly take into account social values such as privacy and fairness.
Gazette contributor Daniel Akst C’78 spoke with Kearns over Zoom about what might make an algorithm ethical and how we can manage the tradeoffs this might entail.
Your book is about the need for ethical algorithms. What gives an algorithm moral properties? How can a set of computer instructions run into ethical or moral issues?
It’s about outcomes. These days an algorithm could be making lending decisions or criminal sentencing decisions, and could produce socially undesirable outcomes. For example, a lending algorithm could produce a certain number of unwarranted loan rejections, and the rate of these could be twice as high for Black applicants as for white ones. If we saw that behavior institutionally in a human lending organization, we would say something in this organization is resulting in apparent racial discrimination. Algorithms can do the same thing.
Do algorithms do such things on purpose?
Computers are precise. They do exactly what they are told to do, and they do nothing more and nothing less. And that precision is what has gotten us into the problems that we talk about in the book.
When you tell an algorithm to find a predictive model from historical data that is as accurate as possible, it’s going to do that, even if that model exhibits other behaviors that we don’t like. So it could be that the most accurate model for consumer lending overall produces disparate false rejection rates across different racial groups, and we would consider that racial discrimination. What we are advocating in the book is that the entire research community thinks about these problems. The fix for this is to be precise about the other goals that we have. When we said maximize the predictive accuracy, we didn’t realize that might result in a racially discriminatory model. So we have to go back and mention our other objective precisely in the algorithm. We have to say something akin to: “OK, maximize the predictive accuracy, but don’t do that if it results in too much racial disparity.”
The workings of algorithms can be quite complex, even opaque. How would we know if an important algorithm was doing things we considered bad?
You can audit an algorithm for misbehavior. So if we take a particular definition of fairness—because remember, you have to be very precise—we can experiment with the algorithm and measure the extent to which it violates this fairness notion. Now, large companies often won’t allow this. The excuse companies will use is, “If I give you the ability to experiment with the input/output behavior of my algorithm, you can reverse engineer what the algorithm is doing. And that’s my intellectual property.” And they have technical means to prevent you from sending hundreds of millions of search requests to, say, Google, to test an algorithm. But there’s no reason that a regulator shouldn’t be able to do that, because a regulator would be legally forbidden from disclosing proprietary information. Also, the companies are starting to do this auditing internally themselves.
Is there a role for universities in this kind of inquiry?
There certainly is. And in fact, there have been such efforts, including ones that I’ve been involved in. I think it’s still early days, but there are similar efforts underway at other universities and also nonprofit groups. There’s a well-known organization called the AI Now Institute [at New York University] that works on these topics.
Your book talks about privacy as another prominent realm in which algorithms can fall badly short of society’s values.
Right. Suppose we use Hospital of the University of Pennsylvania medical records to build a predictive model for some disease based on symptoms and health history. When using medical records to train that model, you should not expect that that model will preserve patient privacy all by itself. You have to be very careful to build that in. Otherwise the output could include information that, matched against other databases, would allow identification of people with certain diagnoses, violating patient privacy.
The power of precision is a theme that runs throughout your book. Can creating (and testing) algorithms force us to make clearer moral choices?
Absolutely. The effort to be exceedingly precise by what we mean when we use words such as “privacy” and “fairness” has great merit in its own right—both because it is necessary in the algorithmic era, and because doing so often reveals hidden subtleties and flaws in our own intuitions about these concepts.
Like Penn criminologist Richard Berk, who works on criminal justice [“Black Box Justice,” Sep|Oct 2017], you’ve said that there can be unavoidable tradeoffs in algorithms between accuracy and fairness, or even between competing conceptions of fairness.
Yes, and these tradeoffs aren’t going to be easy. But pretending they don’t exist, or refusing to look at them, doesn’t make them go away. You’re going to be making some tradeoff between accuracy and fairness, for example, and if you willfully ignore it, you might be doing worse on both dimensions than you could have by acknowledging this tradeoff and making a hard decision about the relative importance of fairness and accuracy in a given domain.
Who should be making these decisions?
Stakeholders—political leaders, philosophers, others in society—have to make judgments about these things, and they have to be precise about it. Once they’re precise about it, we can change our algorithms to give the behaviors or enforce the social norms that they want. That dialogue is in its infancy. One of the reasons the path is not easy is that enforcing a social norm is going to be a constraint, and constraints come with costs. If the most accurate model exhibited racial or gender discrimination, then it’s a tautology to say that by eradicating that discrimination, I will have a less accurate model. The result could be anything from seeing Google ads that are less relevant to getting medical diagnoses that are less reliable, all in service of, let’s say, racial or gender equality. We shouldn’t expect to get these social norms for free.
