Two women sitting at a round table
Image from lab AG Renner © E. Böker, CASCB

Human-in-the-loop analysis of collective eating behaviour

Figure 1: Study setup and conceptualized analysis goals. © AG Keim & AG Renner

How can data analysis facilitate the extraction of meaningful patterns in collective eating behaviour? Computer scientists Matthias Kraus, Hanna Hauptmann and Daniel Keim are working in the field of human-in-the-loop, which is a branch of artificial intelligence that leverages both human and machine intelligence to create machine learning models. The three researchers are developing a state-of-the-art visual analytics framework for interactive extraction and analysis of behaviour patterns (see Figure 1).

The initial prototype focuses on the auditory channel and can be used to analyze and streamline audio data from study trials (see Figure 2).

Figure 2: Initial web prototype for the interactive analysis of speaker diarization results, i.e. the results of a semi-automatic segmentation pipeline that splits one input audio stream by speaker. © Image from protoype tool AG Keim & AG Renner

One important step, which is supported by the framework, is speaker diarization, which is the segmentation of a single audio file into individual audio streams per speaker for further analysis. To obtain data on individual and collective human behaviour when eating, the group collaborates with cluster researchers Britta Renner and Jana Straßheim, who are conducting the project Individual and collective appetite – how is eating shaped by social influence?. “Gaining new insights from a study sample in seconds by visualizing an abstraction of its features – without going through the original video recordings – is incredible!” says Daniel Keim. They started with a modular base framework that facilitates a complete data analysis pipeline, loading raw study data to the visualization of extracted features. Their next step will be the development of a metric that quantifies sentiment or emotion from sequences of a conversation. “Ultimately, we aim for a deployment of our framework in the evaluation process of future user studies in the domain of psychology,” adds Matthias Kraus.