Scenario: Profiling

Constanze Kurz, Debora Weber-Wulff

Herbert, a recently retired data center employee, is waiting impatiently for the letter carrier. Four days ago, he ordered a new laptop from a major mail-order company. Normally, it would only take two or three days for the product to be delivered; Herbert has used the company many times before. After several days, the package still hasn’t arrived, so he sits down at his computer to track the order using the company’s tracking system and find out where his laptop is.

Herbert can’t believe his eyes when he sees that the order was canceled two days earlier—but not by him, he’s a hundred percent sure. He was happy to have taken advantage of a special offer and gotten a rebate on his machine. He immediately calls the hotline to ask what’s going on.

After what feels like an eternity, he finally gets a human being on the line. “What’s up with my order?” he asks, impatiently. A woman answers, “Your order was canceled.” Herbert knows that already: “I can see that myself, but I never canceled it. I want to know why it was canceled. I also want to re-order the computer using the same special offer that was available four days ago.”

The best the woman on the hotline can muster is “I’m sorry, but all I can see is that your order has been canceled. I can’t tell you why. And the rebate offer has expired. Is there anything else I can help you with?” Herbert angrily hangs up without even saying goodbye.

He decides to figure out whether the problem is merely an exception impacting only this one order for a laptop or whether it’s more systemic in nature. He selects a small item to order, clicks the “add to cart”-button, “on credit”, same as usual. Two days later—what do you know—the order was canceled.

That weekend, his daughter Inga drops by for coffee; she lives in a small house in another section of town. When he complains about what the company is doing, Inga tells him that she and her boyfriend also ran into problems with the same company. Her boyfriend had ordered quite a few items but wasn’t able to pay for them immediately because his company was having a short-term cash flow issue. Now, not only was he prevented from making any online purchases, she couldn’t make any either. And it wasn’t just with this one company—they could no longer order from any major retailers.

Herbert knows, of course, that online retailers keep blacklists of customers to warn each other about potential fraudsters. He has some degree of understanding for these kinds of lists because he, too, has been forced to contend with delinquent accounts more than once in his professional career. But he has always paid his bills on time and doesn’t understand why he suddenly can’t place an order.

Then he remembers a younger colleague, Sven, who changed jobs and went to work for this distribution company: maybe he could help! Herbert tracks down Sven’s email address and asks him if he might be able to help sort things out and let him know what is going on.

Sven calls him back the next day and asks Herbert to describe the problem in exact detail. After Herbert has finished telling his tale of troubles, Sven says: “A few weeks ago, we had new, more rigorous profiling software installed. Maybe that’s the issue.” Herbert asks how this profiling software works. Sven elaborates: “Our system is based on a combination of newly developed data mining algorithms to predict as accurately as possible precisely which customers are likely to pay their bills and which are not. You know, our rate of defaults is around seven percent of all billings, and that is of course way too high. We always end up having to hire collection agencies and it takes forever to get the money we’re owed for goods that have long since been delivered. Let me see if it’s the new software that’s keeping you from shopping on credit.”

Sven calls back just a few hours later. “Indeed, Herbert, you’ve been flagged as a potential problem. If I’m reading the results of the algorithm right, you’ve been classified as someone who is closely related or in domicile with a major debtor. Your surname is not a common one—according to our database there are only eight people with the same last name in Germany, and two of them are in your city. Statistically, it’s been shown that relatives of debtors often don’t pay their bills either. So that’s what’s going on here.”

Sven provides a few more details—for example, the fact that the software also factors in age, and that’s why retirees are downgraded from the get-go. But Herbert is hardly even listening. Sven offers to manually remove the flag from the database; after all, he knows Herbert personally and has also gone ahead and checked to see if there were any outstanding bills in his name: there weren’t. “No,” Herbert said, “that won’t be necessary. I won’t be shopping at any retailer that imposes guilt by association anymore. But thanks for your help. At least now I know what happened here.”

Questions:

1. What is the ethical verdict on profiling software in and of itself?
2. Should companies be required to notify clients of profiling practices before an order is even placed? Should companies who prevent clients from purchasing on credit based on internal systems be forced to offer a cash payment option?
3. Shouldn’t the company at least notify the customer about a canceled order?
4. Is using profiling software a legitimate means of minimizing defaults on accounts? Does it make any sense to use this kind of software for this purpose?
5. Was it OK for Sven to offer to simply delete the problem flag?
6. Is Sven allowed to silently look up whether or not Herbert has any outstanding balances without Herbert’s prior consent?
7. Was Sven allowed to reveal the inner workings of the system? After all, the algorithms may in fact be confidential. Is Sven morally obligated to provide any information to Herbert, even if it involves the potential revelation of trade secrets? What is Sven’s obligation to his employer? Is Sven required to act once he has discovered that the profiling software is not doing what it’s supposed to do? If so, what should he do?
8. Is it even legitimate to predict the payment activity based on the behaviors of relatives or housemates and not on any criteria related to the solvency of the actual customer?
9. Should individuals who have defaulted on any retail transaction be excluded altogether from conducting retail business online?
10. An increasing number of products is available on the market to make online transactions faster, cheaper, or to make some things available for the first time ever. Does an ethical dilemma arise when these profiling systems render some of these products entirely inaccessible to certain individuals? Do people have a fundamental right to access any and all products the market has to offer?
11. Is it an ethical issue for a mail order company to operate profiling software behind its customers’ backs?
12. Shouldn’t a company be required to inform the public—or, at the very least, actual customers—about the process behind certain results?
13. How can you protect yourself against profiling software?

Published in Informatik Spektrum 37(3), 2014, S. 259–261.

—Translated from German by Lillian M. Banks