Scenario: The Nightmare

Christina B. Class & Debora Weber-Wulff

Andrea, Jens, and Richard are sitting at their favorite pizza joint in Hamburg, celebrating the successful completion of software tests for their “SafeCar”-project with beer and pizza. They can take the project to the DMV in Hamburg tomorrow for inspection, and with that—they’re done. They won’t have to pull an all-nighter.

They work for SmartSW GmbH in Hamburg, where they are tasked with developing a software control system for the safety component on the new car model KLU21 produced by German automaker ABC.

ABC has come under increased economic pressure of late because of its inordinate focus on internal combustion engines over electric motors. Emissions scandals surrounding diesel engines in recent years have made things increasingly difficult for the company.

KLU21 is the company’s latest energy-efficient hybrid model that will be equipped with a new intelligent driving control system and an enhanced safety system. This new intelligent driver-assistance feature will provide drivers with real-time data. Using the latest communications technologies, ABC is hoping to improve its tarnished reputation as it re-brands itself as a state-of-the-art company and a pioneer in the field. It’s hoping this will help win back some of the market share it lost and hold on to some jobs.

As part of the SafeCar project, a safety system for KLU21 with artificial intelligence was developed. The software prevents the car from drifting out of its lane using information from external cameras linked to the vehicle’s location, current traffic data on road conditions, and satellite data loaded in real-time from a secure cloud. The vehicle is also aware of other cars on the road and is programmed to maintain a safe driving distance.

At the same time, a camera aimed at the driver detects when the driver’s eyes are closed, which could mean that they have nodded off or lost consciousness. An alarm will sound to wake the driver. If the system does not register an immediate reaction, the autopilot takes over and brings the vehicle to a stop at the side of the road, taking into account road conditions and traffic. To minimize the risk to all parties involved in any traffic event, the maximum driver reaction time before the software takes control of the vehicle depends on site-specific road conditions and traffic.

Andrea, Jens, and Richard suggested to the project director that the software should also detect when a driver is fiddling with their cell phone and send and display a warning on the dash and an audible warning signal. Jens and Richard took it a step further to suggest that the software detect when a driver is applying lipstick and the car should whistle, though not every time, only every sixth time or so. They were just kidding, and this feature was not included.

It was easy to send the images to the cloud cluster over the 5G network, where they could be broken down to see if they could recognize anything. However, detecting drivers nodding off and losing consciousness as accurately as possible requires that the program be trained using mountains of data specifically designed for that purpose. Recognizing someone using a cell phone or lipstick involved simple finger exercises, and ample “training data“ (images and videos) was readily available at no cost on the Internet.

In the event of a mechanical breakdown, or if the driver loses consciousness or is involved in an accident, an emergency call is automatically placed to the police with the relevant data about the car, the type of breakdown or accident, whether an ambulance is needed, and the vehicle’s exact location.

After all the overtime they’ve put in, Andrea, Jens, and Richard are having a good time celebrating the successful completion of the tests. After many successful in-house tests, they’ve spent the last two days driving through city streets and the greater metropolitan area with a system installed on the passenger side, logging all the information and commands from the system. Results show that the system accurately recognized every situation it encountered. Their boss has already congratulated them on a job well done. They’re hyped and sit around chatting until well past midnight.

After cabbing it home, Andrea is suddenly thirsty, so she plops down on the couch with a bottle of mineral water and turns on the TV. There’s a rerun of a talk show airing a segment about the current status of the 5G network expansion in Germany—a topic that’s been in the news for months. Once again, the discussion turns to how many dead spots in rural areas have yet to receive coverage. To top things off, rules to regulate national roaming fees remain inadequate.

One corporate executive claims that once the 5G rollout is complete, there will only be a few dead spots remaining. But a viewer vehemently insists that even with the new network, she only has reception in her garden. Her new neighbor has a different network provider but has to go to the cemetery at the edge of town because there is no roaming mandate. A heated discussion ensues….

Andrea is exhausted. She leans back. Her eyes fall shut … She’s driving a car along a winding road in foggy weather. Visibility is next to zero. She’s glad she’s driving a KLU21; it helps her to stay in the right-hand lane. But suddenly she notices that the car’s steering assistance has stopped working. She is unprepared for this, and before she can respond, her car leaves the road and crashes sideways into a tree. She is thrown forward and hits the airbag at an awkward angle. She passes out.

She can see herself sitting unconscious in her car. Help is on the way; luckily, KLU21 has placed an emergency call. She looks around the car: the emergency light isn’t blinking; why hasn’t an emergency call been made? She’s bleeding! Why isn’t anyone coming to help? Suddenly, she knows! She’s in a not spot … The steering assistance can’t function as it should without real-time data, nor can the emergency call center be notified …

“Hello? Can anybody hear me? Why aren’t there any cars on the road here? Why isn’t anyone coming to help? Help, I’m bleeding! I’m bleeding to death!”

Andrea wakes up in a cold sweat! It takes her a moment to calm down and realize that it was just a dream, that she is at home in her apartment, and that she is okay.

Then she remembered today’s software tests in and around the city and the talk show with those complaints about inadequate network coverage in some rural areas. Roads run through those areas, too. Why didn’t it occur to them while they were developing the software for KLU21? Why didn’t they ever bring it up with their client? Her aunt lives in Sankt Peter-Neustadt and has often complained about the poor connectivity. She needs to talk to her boss …

Questions:

1. Designing tests is very difficult. In this example, the developers were supposedly heavily involved in the tests. Is this justified? What guarantee do we have that the tests won’t be manipulated to produce inaccurate results? How strictly should test parameters be required to be directly dependent on the safety relevance of the software? How heavily do production deadlines and economic pressures influence the implementation of software tests?

2. No amount of testing can guarantee that any software product is free from error. It’s even harder to prove that the software won’t do something it hasn’t been asked to do. In this scenario, a so-called “Easter egg” was to be included—the lipstick recognition feature—but not activated. What dangers might be involved in this kind of “Easter egg”? Does it matter if the developers have programmed the Easter egg during their “free time”?

3. This scenario involves a software application that is critical to the safety system and was trained using training data. How can the quality of training data be guaranteed? What guidelines should/could be set in this regard? When and under what conditions should a system be forced to be re-tested using new data? What form should such new testing take once the system has been fed new training data? Do all the tests that have previously run need to be repeated? Should the same people responsible for defining the training data also be involved in developing tests? If not, why not?

4. The internet is filled with images and videos that are “free”? Can these materials be used to generate training data? If not, why not? If so, should any restrictions be placed on the types of things they can be used for?

5. Open data (readily accessible, application-specific data sets) exist for specific applications, such as data mining and AI contests. Is it permissible to use these data sets for other purposes/research questions? What requirements must be placed on the documentation of these datasets, specifically concerning data source, data selection, and conclusions drawn? How great is the likelihood that false conclusions will be drawn when using these data types?

6. In current discussions about 5G, there is often talk about the need for nationwide coverage and national roaming—how important is this? Is it reasonable to create applications that are critical to safety before there is any guarantee that infrastructure is available nationwide? Would it be reasonable to write applications strictly for specific locations—for use in the city, for example? What kind of compensation is due if a user relocates and can no longer use the system?

7. Many areas today are already adversely impacted by “not spots” (areas with no reception). For example, in places where rescue teams or police can’t be reached by cell phone or emergency services cannot contact the call center, to what extent is political intervention needed to force regulation? How great is the digital divide in Germany? As an industrialized nation, can we afford to let this stand or even worsen, especially now with the expansion of the 5G network?

Published in Informatik Spektrum, 42(3), 2019, S. 215-217, DOI: 10.1007/s00287-019-01171-4

— Translated from German by Lillian M. Banks