Scenario: Behavior Recognition

Benjamin Kees, Rainer Rehak

While Hendrik was completing his computer science degree, a new game console concept came on the market in which the player’s own body replaced the game controller based on a camera’s recorded analysis of the player’s movements. Hendrik was thrilled to get his hands on one of these game consoles. He was so fascinated by the technology behind it that he wrote his final thesis on the unique process for modeling and recognizing human movements in video images.

Not long after he defended his thesis, Hendrik included information about his research in a profile he created for an online business portal. He promptly snagged a lucrative job offer from “v-Watch,” a fledgling security company that sold modern video surveillance systems. Their innovative approach rendered the constant monitoring of camera images superfluous because their systems were designed to detect conspicuous behavior automatically.

While he’d have preferred working with motion-based computer games, he was genuinely interested and accepted the offer. Before long, he began collaborating with Franziska, another recent hire who was working on fully automated computer learning methods. Manually modeling all possible behavioral abnormalities was too time-consuming, so they devised a concept that linked Franziska’s learning process with Hendrik’s motion recognition system.

Based on sampled video material, their system was designed to learn to differentiate between normal and conspicuous behavior. Once it was in place, any behaviors that deviated substantially from pre-programmed “normality” would trigger an automatic alert to security personnel.

After several failed attempts at recognizing complex behavioral sequences, Hendrik and Franziska concentrated on evaluating body language. In their weekly team meetings, there was consensus over the notion that conspicuous behaviors of this nature were a promising index of potentially imminent violent or otherwise criminal activities—that is, relevant security concerns.

However, since the company didn’t have any suitable film stock to train the system and had no budget to hire actors, they decided on the fly to use employees for filming. Hendrik wasn’t altogether happy with the idea and doubted how valuable the resulting material would be.

Overall, though, the recording session made for a fun day that helped improve the work atmosphere for the v-Watch team. In the morning, they taped material for teaching normal behaviors, and that afternoon, they shot footage for conspicuous behaviors, which would later be used to test the system. Weeks after viewing the footage, Hendrik’s colleagues still jokingly referred to him as “the Gorilla guy.” Franziska, the only female member of the small team, was the only one who wanted nothing to do with the whole shebang.

Franziska soon began adapting the learning process to yield the results they anticipated based on their recordings. After some successful testing, Hendrik’s initial concerns were dispelled by his colleagues’ enthusiastic reception and positive reinforcement, who were as happy with the quality of collaboration as with the successful functioning of the prototypes.

After several intense months and tons of overtime, the finished product was installed for the first time in a shopping mall. Hendrik’s and Franziska’s module, in particular, sent out a lot of alerts. At a performance review, a department store security guard boasted about always keeping a close eye on people the system had flagged as suspicious. Even though the store saw a significant decrease in overall criminal incidents, some cases proved that the system’s alerts were justified, most often in the case of young adult males: The people it flagged for closer surveillance had been frequently involved in thefts and other brushes with the law. When he heard that, Hendrik began questioning whether “his” system was all that great a solution to the problem at hand because this wasn’t a computer game, and here, there were real-world consequences for making the wrong moves.

Questions:

Usability:

• How do you define “conspicuous” and “suspicious” behavior, and to what extent are the two related?
• What preconceived associations between conspicuous behavior and criminal activity form the basis for the system depicted here?

Relatability:

• Can the in-house detective relate to the behaviors flagged as “conspicuous,” and is he right to consider these people “suspect”?
• Can Hendrik and his team relate to the behaviors flagged as conspicuous?
• Should relatability on the part of security personnel be a prerequisite for this kind of work?

Test Subjects and Real-World Individuals:

• Would the practical application have yielded different results if the test subjects in the video were not (exclusively) young adult men living in the real world?
• Would it make a difference if paid student actors had been hired or if actual video footage from the department store had been used?
• What are the implications of the above-described system for young adult males? What are the implications for older women? Is there any way to avoid this kind of discrimination?

Ramifications for the Public at Large:

• What impact does it have on individuals to know that computers are constantly analyzing every move they make in public spaces?

Erschienen in Informatik Spektrum  37 (5), 2014, S. 503–504.

— Translated from German by Lillian M. Banks