Gewissensbits

Christina B. Class, Otto Obert & Rainer Rehak

Are you looking for a little help from AI? These days, many software developers are doing just that. But how much can you trust generative tools? The following scenario illustrates how important it is to pose this question early on.

Three weeks ago, André was hired as a developer by Smart4All, a small firm specializing in custom software solutions for small to mid-sized companies. Recently, there has been an increased demand for AI-based services, both internally and externally, and André has been assigned to work in this area. He’s a newly minted BA who did reasonably well as a business and information technology major. At the moment, there is no shortage of IT job offerings. And yet, here he is—a guy whose strong suit wasn’t exactly statistics, programming, and AI—working in an IT department. Suffice it to say that while in school, he profited greatly from collaborations with his fellow students, especially in these areas.

He got lucky with his bachelor’s thesis: he completed the work at a mid-sized company where his job was to evaluate the potential for data mining and AI to minimize costs and optimize the preparation of proposals. After researching various blogs and on the code-sharing platform CoDev, André could find most of the code he needed. Then came the first version of Easy-AI-Code-Pilot—a tool for automatically generating code that was first introduced on CoDev. While he was skeptical at first, before long, he no longer had any qualms about using it all the time. It was no small task to get enough of a grip on the individual fragments to combine them to do what he wanted. At the time, the company was happy with his work and wrote him a glowing letter of recommendation.

Now, André is sitting at his new desk, staring out the window, when the team leader, Verena, comes in, smiles, and says she has an important assignment for him. BioRetail, a major nationwide distributor of organic products, has contracted Smart4All to develop new software solutions to integrate existing in-house programs for customer management, accounting, ordering, and warehousing. The client wants a solution to forecast incoming orders in a B2B format. There was data to prepare, processes to test, and everything to be documented in Python Notebooks…the usual. Verena flatters him—“That’s right up your alley!” she says. He subtly hints that these aren’t exactly his core competencies and that, while completing his bachelor’s degree, he’d relied primarily on such resources as blog entries and mainly used CoDev and Easy-AI-Code-Pilot to generate code. Verena grins at him and says, “That’s what everyone’s doing nowadays.”

André’s concerns thus fade, and he gets to work. Understanding and cleaning the data is no small feat for him, but he finds a snippet of code from a hackathon that involved scrubbing very similar types of data. He uses various code snippets culled from the internet that apply different cleaning methods. He tests the code snippets and evaluates them on the basis of the usual quality-control criteria. Easy-AI-Code-Pilot offers good suggestions for small subtasks, but André struggles to integrate all these different pieces of code. Even though there are times that he’s not quite sure of himself, in the end, everything looks plausible and consistent enough. However, he cannot guarantee that he may not have overlooked or incorrectly assigned one thing or another in all the data and code fragments. Nor did he adhere to any strict separation between training, validation and testing data sets. Time is ticking, and he brushes aside his creeping doubt because the results look convincing. He had, after all, run tests on various models using multiple hyperparameters for each one, and he correctly documented everything in Python Notebook files. He may not have come up with the perfect solution, but André is confident that the result is not too shabby.

Three weeks later, Verona and André are called in by Frederic, the IT product director and account executive for BioRetail. When they enter the room, they see Geraldine—the representative from BioRetail. The atmosphere is cold, and Frederic asks everyone to have a seat. Then Geraldine begins: at first, she was enthusiastic about the prognostic notebooks, but when she tried analyzing more recent data, it was spitting out results that made no sense whatsoever. Since the new data looked slightly different, she wanted to adjust the notebooks herself. That’s when she noticed that different program components were all based on different features. Then she saw even more inconsistencies, so she sat down with her IT people and reviewed everything. She couldn’t believe her eyes. The code was awful; it had zero homogeneity, and the data models were way too different. The documentation was atrocious. Everything was a hodgepodge slapped together from various methods that couldn’t function reliably. It was totally unacceptable, and any company capable of delivering such a slipshod product was certainly in no position to integrate multiple programs. Verena and Frederic exchanged glances before they looked at André and said: “So, you’ve got to have some explanation for this, right?”

Questions:

• What is the value of using such tools as Easy-AI-Code-Pilot to generate code automatically?

• What basic principles should be followed when using these assistance systems?

• Should André have been more diligent about telling his team leader he wasn’t qualified to take on the assignment?

• What was Verena’s role in this? As team leader, what should she have done better? Who is most at fault for the whole fiasco?

• Does the platform provider CoDev have any responsibility for making a product like Easy-AI-Code-Pilot accessible free of charge? Is it enough for CoDev to display a text warning about potential product misuse?

• What steps should providers of these kinds of tools take to live up to their obligations?

• What do you think about Verena humiliating André in front of the IT product director and the BioRetail representative? What ethical principles should apply to management personnel in this situation?

• Is it even possible to implement ethical principles in AI? What are the implications for the way we deal with AI? What might a code of ethics look like?

Published in .inf 05. Das Informatik-Magazin, Frühjahr 2024, https://inf.gi.de/05/gewissensbits-softwareentwicklung-mit-kollege-ki.

— Translated from German by Lillian M. Banks

Christina B. Class & Debora Weber-Wulff

For years, they’ve been preparing for this. But now, 1950s-era dreams of a self-driving vehicle are finally coming true. They christened their creation “Galene”—the self-driving car. It performed like a champ on the test track. Even in test drives on American roads—for which Galene had to be shipped to the US—everything was swell. There were fewer regulations in the US, where endless highway stretches and good visibility combined allowed ample room for experimentation.

Everything was more complicated in Germany, and obtaining the necessary permits for testing on public thoroughfares took longer. The press has been invited to tomorrow’s widely publicized “maiden voyage.” Jürgen, one of Galene’s proud “parents,” has gotten approval from his team leader to take his baby out for a spin on the planned course before the press gaggle gets underway, just to be sure everything runs smoothly. He’s a good engineer, so his planning has been meticulous. It’s a Sunday afternoon when these roads won’t have much traffic. And he’ll be seated at the wheel himself to intervene should anything go wrong. He’s confident he won’t call attention to himself or annoy other drivers or passersby.

He tells the voice computer where he wants to go, and Galene confirms the destination. Then, he calculates the route, taking into account current traffic reports, known construction sites, and the weather forecast. Everything is good to go: there is no construction along the route, no rain, no fog, and only a slight breeze. It’s a sunny autumn day—perfect for the first test drive!

Jürgen is enjoying his ride along this route, which he knows well. It is a great feeling to let someone else do the driving, even though it still seems strange not to put on the gas, hit the brakes, or take the steering wheel. Galene enters the expressway flawlessly, passes a classic car, takes the next exit, slows to a crawl, and stops at the light. She always keeps her distance from the vehicle ahead of her. The steering is so precise that it could be set to approach within less than an inch of the car she is trailing. But that would put other drivers needlessly on edge, so Galene’s been programmed to maintain a distance of about 15 inches.

Jürgen would love to use his cell phone to film how he catches a “green wave” before he hangs a left at the third light—he wasn’t quite sure whether Galene would accurately calculate all the signals involved to make that happen, but she did: perfect! If he would pull out his phone and start filming, he could hardly sustain the illusion that he was the one driving the car. As they enter a newer residential district, Galene reduces her speed to the 30km/h limit. There’s a school on the left, with bus stops for school buses on both sides of the street. They invested a lot of time preparing Galene to deal with this type of traffic scenario.

Luckily, fall recess happens to be in session. To their right, they pass a park with sprawling grassy areas. He hears kids screaming and looks to his right. Jürgen sees dogs romping, brightly colored balls bouncing in the grass, and kites flying even brighter than that in the air. When the wind blows them in his direction, Jürgen instinctively grabs the steering wheel, knowing as he does that children at play ignore traffic.

Especially when he first started “driving” the self-driving car, this happened a lot: he would get nervous, reach for the steering wheel, and switch to manual control so he could take over. But he never needed to, so he gradually learned to relax and leave the driving to Galene. Now, though, suddenly, it happened: some kid with a kite in hand darted out onto the street from between two parked cars and was hit by Galene. The kid falls to the ground unconscious.

Galene immediately hits the brakes because her sensors have detected the impact. At the same time, Jürgen pulls the emergency stop button. Galene comes to a halt, and the hazard lights are activated. Jürgen gets out and runs toward the child, whose mother soon appears and starts going off on Jürgen. Another young woman exits the car, driving behind Jürgen, and begins administering first aid. She says she’s a nurse.

A dog owner visiting the park has already placed an emergency call,, and the ambulance arrives promptly to transport the child and his mother to the nearest hospital with blue lights flashing. The police are also on the scene to file an accident report. Jürgen appears to be in a state of shock. The young woman who administered first aid immediately approached the police, even before they had the chance to question Jürgen. She tells them her name is Sabine, and she was driving behind the vehicle involved in the crash. She thinks he was driving too fast. She was driving well below the 30 km/h speed limit—with all the sounds of kids playing in the park, the dogs chasing after balls, and the kites flying, you had to expect something like this to happen!

The police ask Jürgen for his license and registration. He gives them his ID, driver’s license, and the test drive permit. The cops are taken aback and start asking questions about the car—they’re intrigued. Since this involves a road test, but the vehicle is technically not licensed for operation on public roads, they insist on having Galene towed, mainly because the data had to be analyzed more thoroughly. Jürgen is sure that Galene followed the rules of the road, but the accusation made by the witness, Sabine, still weighs heavily on him. Tomorrow afternoon’s “maiden voyage” and press conference are in jeopardy. It’s a PR disaster—especially now after this accident.

Questions:

• The car had an official operating license for road tests. Was it okay to take it out for a test drive before the road test was completed?
• Airplane pilots are repeatedly required to undergo training to guarantee they can respond quickly in an emergency and take over controls from the autopilot. Will this type of training also be needed for self-driving cars? Should Jürgen have been permitted to sit back and relax during the road test?
• As soon as Jürgen saw children playing in the park, he instinctively grabbed the steering wheel. As a driver, should he be required to take control of the vehicle in a situation like this, where he could expect children to run into the street?
• Galene was following the 30 km/h speed limit, but the witness complained that this was too fast when there were so many kids playing in the park. When calculating speed, to what extent can and should algorithms account for activities along the roadway?
• Unforeseen events will always cause accidents, whether a child running out into the street, a wildlife crossing, or a tree branch down on the road. Disaster is often averted by a driver’s speedy response or instinctive hesitation. Should algorithms be programmed to emulate some instinct? To what extent can self-training systems be of use in this regard?
• Sometimes, rear-end collisions result from a driver following the rules of the road “too closely”: for example, stopping at a yellow light on a busy highway or sticking to the posted speed limit in the blind bend on an expressway exit ramp. Self-driving vehicles are programmed to adhere strictly to the rules. Should they be programmed with a built-in “bending of rules” based on the behavior of cars driving in front and behind them?
• It would be impossible to test for every imaginable scenario, so the software on a self-driving car may respond inappropriately. In that case, who is liable? The developer? The manufacturer? The driver who is seated at the wheel “just in case”? Or would we take these cases in stride in exchange for the greater security these cars provide in other circumstances? Where do we draw the line?
• How and when should software updates be installed for self-driving cars? Only at the dealership or wherever the vehicle happens to be located, as long as it is stationary. Who oversees and determines whether an update has been installed and when? What happens if an accident could have been prevented if a software update had been installed? And who is liable?

Published in Informatik Spektrum 38(6), 2015, S. 575–577.

— Translated from German by Lillian M. Banks