Noura Howell

Exploring the promise and peril of emotion AI, designing for emotional meaning-making with data, and imagining an affirmative biopolitics with data

Emotion AI, which predicts psychological characteristics from data, offers potentially transformative benefits for societal well-being, productivity, and security. Drawing on increasingly available biodata--data about people’s bodies and behaviors, such as video, audio, or heart rate--emotion AI predicts emotions, stress, focus, and other characteristics. Emotion AI increasingly informs sensitive decisions in many varied contexts, from social media to online education, online job interviews, or security surveillance systems and criminal investigations.

A key challenge to emotion AI is that algorithmic ways of modeling emotion differ fundamentally from human ways of understanding emotion, making emotion AI predictions difficult to meaningfully interpret and apply in real-world contexts. In addition, even people aware of widespread video surveillance may be unaware that an additional layer of algorithmic surveillance using emotion AI is making sensitive predictions about their inner psychology from video of their facial expressions, leading to privacy and civil liberties risks.

My design research explores both the promise and peril of emotion AI, and contributes design tactics to more effectively support social, embodied, and emotional meaning-making with data. Combining building custom biosensing technologies and realtime data displays with concepts from the arts and humanities, my work explores, how might we imagine a more affirmative biopolitics with data?

I was invited to present this work at the Georgia Tech GVU Center in September 2021. event info slides

Charting conceptual shifts for emotional biosensing

A composite of three self-tracking products: At left, white fingers clip a Spire Stone product into the upper seam of their pants. At middle, a white woman's crossed forearms show her left arm is wearing a red Feel wristband. At right, a black man wears a black Oura ring on his left hand.

From left: Spire Stone monitors breathing to detect states of calm, tension, and focus. The Feel wristband claims to be the world's first emotion sensor and mental health advisor. Oura ring helps track sleep and activity for wellness and productivity.

Emotional biosensing is on the rise in daily life: Technologies use data and algorithms to claim to know how people feel and suggest what they should be doing about it. Prevalent approaches to emotional biosensing are too limited, focusing on the individual, optimization, and normative categorization.

My research helps explore critical alternatives for emotional biosensing design. As a design researcher, I survey and map the landscape of emotional biosensing technologies, and synthesize critical alternative design directions for this space. I work through conceptual moves and apply them to emotional biosensing, drawing from design theories on materiality, performativity, biopolitics, and affect. I call for humility in knowledge claims with emotional biosensing and prioritizing care and affirmation over self-improvement. Biosensing designs need to critically question and generatively re-imagine the role of data in configuring sensing, feeling, ‘the good life’, and everyday experience.

Biosensing technologies track many aspects of daily life, including emotions, in sometimes unexpected ways. For example, video surveillance can be used to ‘detect’ percentages of joy, contempt, or anger, ‘hostile intent’ threats at airports, or heartrate. Heartrate can be linked to stress or future involvement in violent crime. Mobile phone and Instagram data can be linked to depression diagnoses. Abilify, a medication for depression, bipolar, and schizophrenia, tracks whether patients ingest it to ensure compliance.

It is difficult to know or manage how we will be categorized from the seemingly innocuous data we give off by appearing on surveillance cameras or using a mobile phone, or how these correlative categorizations might masquerade as authoritative predictive insights.

For those of us not (yet) categorized as ‘crazy’ or ‘criminal’, or those deemed ‘healthy’, consumer products urge self-improvement via data-driven behavioral modifications. Microsoft Band promised, “This device can know me better than I know myself, and can help me be a better human”, with “actionable insights” to help “live healthier and achieve more ”. Wristband sensor Feel claimed to be the “world's first emotion sensor and well-being advisor”. Wearable breath monitor Spire offered to help “gift wrap peace of mind this Christmas”, reducing stress and improving mindfulness. Some of these products have shifted focus or been discontinued, but they represent a prevalent narrative of data-assisted self-improvement. Certainly we all need help emotionally sometimes, and these technologies may help some people. Yet, these devices illustrate a prevalent approach to emotional biosensing that is far too limited.

Emotional biosensing technologies promote a particular normative vision of ‘the good life’ limited by a focus on individual wellness, self-improvement, and workplace productivity. While there is positive potential here, these limitations point to opportunities for broadening the design space with emotional biosensing.

Materiality of data: I sketch how a broad shift from abstract representationalism to sociomaterial performativity opens critical alternatives for emotional biosensing. Rather than treating data as something immaterial to be represented, attending to the materiality of data sensing and display opens new possibilities for interpretation and experience. (This section draws from Vallgårda, Dourish, Mazmanian, Giaccardi, Karana...)

From representation to performativity: Treating sensing/display not as passive representations but as active performances or responses offers opportunities for social meaning making and experiential, expressive displays. Emotional biosensing designs should treat affect, feeling, and emotion as embodied, dynamic, and fluid rather than abstract, static, discrete categories. (Höök, Wilde, Schiphorst, Klooster, Verbeek...)

Affect-as-interaction: Continuing this performative shift, reframing affect from a kind of abstract information to affect as socially enacted in interaction adds sociocultural nuance to emotional biosensing. We call for humility in the emotional knowledge claims made by design, leaving more open to human interpretation, adaptation, and contestation. (Boehner et al.)

From inter-action to intra-action: Shifting further to posthumanist performativity, we outline how agential realism can help emotional biosensing designs attend to how materials shape meaning-making and prompt reflexivity about which materials, people, or categories are assumed to be separate. (Barad)

Biopolitics: Attending to biopolitics calls for embracing a diversity of voices and ways of knowing with emotional biosensing. We suggest designs with intentionally reduced authority are worth exploring, and that emotional biosensing designs can leverage negative affect as a resource of building community, support, and collective action. (Lorde, Noble, Imhotep, Hirsch et al., Deleuze, Foucault...)

Affect/desire: Reframing affect and desire as pre-individual, we call for emotional biosensing designers to critically reflect on the desires our designs legitimize, and on the societal structures that make those desires seem appropriate. We argue that emotional biosensing designs should prioritize care, affirmation, respect, and recognition over self-improvement. Designs can encourage peopel to trust themselves to be ‘emotionally good/safe enough’ with room to explore and change rather than only seek self-improvement according to predefined norms. More broadly, designs should explore a wide range of desires with emotional biosensing. (Ahmed, Lacan, Deleuze & Guattari, Massumi...)

For full citations please refer to the paper itself.

Biodata as Material

In collaboration with Vasiliki Tsaknaki, Tom Jenkins, Laurens Boer, Sarah Homewood, and Pedro Sanches, we hosted a workshop on challenges and opportunities for designing with biodata as material. This workshop explored four thematic areas of biodata designs:

Making sense of biodata by considering a holistic perspective including body, mind, and emotions

Feminist perspectives of biodata that consider bodily transitions, a plurality of bodies and bodily experiences, and differing bodily temporalities

Non-anthropocentric perspectives with biodata, to more broadly consider other organisms and non-human material agencies

Health and affective contexts of working with data produced by our bodies

Papers

Emotional Biosensing: Exploring Critical Alternatives. 2018. Noura Howell, John Chuang, Abigail De Kosnik, Greg Niemeyer, Kimiko Ryokai. Proceedings of the ACM on Human-Computer Interaction. pdf