Scenario: Developing Software with your AI Assistant

Christina B. Class, Otto Obert & Rainer Rehak

Are you looking for a little help from AI? These days, many software developers are doing just that. But how much can you trust generative tools? The following scenario illustrates how important it is to pose this question early on.

Three weeks ago, André was hired as a developer by Smart4All, a small firm specializing in custom software solutions for small to mid-sized companies. Recently, there has been an increased demand for AI-based services, both internally and externally, and André has been assigned to work in this area. He’s a newly minted BA who did reasonably well as a business and information technology major. At the moment, there is no shortage of IT job offerings. And yet, here he is—a guy whose strong suit wasn’t exactly statistics, programming, and AI—working in an IT department. Suffice it to say that while in school, he profited greatly from collaborations with his fellow students, especially in these areas.

He got lucky with his bachelor’s thesis: he completed the work at a mid-sized company where his job was to evaluate the potential for data mining and AI to minimize costs and optimize the preparation of proposals. After researching various blogs and on the code-sharing platform CoDev, André could find most of the code he needed. Then came the first version of Easy-AI-Code-Pilot—a tool for automatically generating code that was first introduced on CoDev. While he was skeptical at first, before long, he no longer had any qualms about using it all the time. It was no small task to get enough of a grip on the individual fragments to combine them to do what he wanted. At the time, the company was happy with his work and wrote him a glowing letter of recommendation.

Now, André is sitting at his new desk, staring out the window, when the team leader, Verena, comes in, smiles, and says she has an important assignment for him. BioRetail, a major nationwide distributor of organic products, has contracted Smart4All to develop new software solutions to integrate existing in-house programs for customer management, accounting, ordering, and warehousing. The client wants a solution to forecast incoming orders in a B2B format. There was data to prepare, processes to test, and everything to be documented in Python Notebooks…the usual. Verena flatters him—“That’s right up your alley!” she says. He subtly hints that these aren’t exactly his core competencies and that, while completing his bachelor’s degree, he’d relied primarily on such resources as blog entries and mainly used CoDev and Easy-AI-Code-Pilot to generate code. Verena grins at him and says, “That’s what everyone’s doing nowadays.”

André’s concerns thus fade, and he gets to work. Understanding and cleaning the data is no small feat for him, but he finds a snippet of code from a hackathon that involved scrubbing very similar types of data. He uses various code snippets culled from the internet that apply different cleaning methods. He tests the code snippets and evaluates them on the basis of the usual quality-control criteria. Easy-AI-Code-Pilot offers good suggestions for small subtasks, but André struggles to integrate all these different pieces of code. Even though there are times that he’s not quite sure of himself, in the end, everything looks plausible and consistent enough. However, he cannot guarantee that he may not have overlooked or incorrectly assigned one thing or another in all the data and code fragments. Nor did he adhere to any strict separation between training, validation and testing data sets. Time is ticking, and he brushes aside his creeping doubt because the results look convincing. He had, after all, run tests on various models using multiple hyperparameters for each one, and he correctly documented everything in Python Notebook files. He may not have come up with the perfect solution, but André is confident that the result is not too shabby.

Three weeks later, Verona and André are called in by Frederic, the IT product director and account executive for BioRetail. When they enter the room, they see Geraldine—the representative from BioRetail. The atmosphere is cold, and Frederic asks everyone to have a seat. Then Geraldine begins: at first, she was enthusiastic about the prognostic notebooks, but when she tried analyzing more recent data, it was spitting out results that made no sense whatsoever. Since the new data looked slightly different, she wanted to adjust the notebooks herself. That’s when she noticed that different program components were all based on different features. Then she saw even more inconsistencies, so she sat down with her IT people and reviewed everything. She couldn’t believe her eyes. The code was awful; it had zero homogeneity, and the data models were way too different. The documentation was atrocious. Everything was a hodgepodge slapped together from various methods that couldn’t function reliably. It was totally unacceptable, and any company capable of delivering such a slipshod product was certainly in no position to integrate multiple programs. Verena and Frederic exchanged glances before they looked at André and said: “So, you’ve got to have some explanation for this, right?”

Questions:

• What is the value of using such tools as Easy-AI-Code-Pilot to generate code automatically?

• What basic principles should be followed when using these assistance systems?

• Should André have been more diligent about telling his team leader he wasn’t qualified to take on the assignment?

• What was Verena’s role in this? As team leader, what should she have done better? Who is most at fault for the whole fiasco?

• Does the platform provider CoDev have any responsibility for making a product like Easy-AI-Code-Pilot accessible free of charge? Is it enough for CoDev to display a text warning about potential product misuse?

• What steps should providers of these kinds of tools take to live up to their obligations?

• What do you think about Verena humiliating André in front of the IT product director and the BioRetail representative? What ethical principles should apply to management personnel in this situation?

• Is it even possible to implement ethical principles in AI? What are the implications for the way we deal with AI? What might a code of ethics look like?

Published in .inf 05. Das Informatik-Magazin, Frühjahr 2024, https://inf.gi.de/05/gewissensbits-softwareentwicklung-mit-kollege-ki.

— Translated from German by Lillian M. Banks