The body as data: Immersive technology and biometric artistic performance
The performer Sarah Nicolls sits poised at a piano at a darkened concert hall in London. This is no ordinary recital—strapped around her arms are Myo Armbands that track her arm movements, allowing her to generate curious and unnerving sounds through gesture as she plays Atau Tanaka’s (2019) biometric piano piece Suspensions.
In Madrid, Marco Donnarumma’s (2013) hunched body heaves concrete blocks painfully across a stage in his performance Hypo Chrysos. A bespoke biophysical instrument called XTH Sense is attached to his arm, prompting loudspeakers to play out the distorted sounds of his straining muscles and crackling bones for the audience.
Across the world, University of Colorado Boulder student Anna Pillot flies across the stage in a breathtaking aerial dance titled IMbalance | INbalance (Dever, 2022). The biometric sensors in the fitness watches on her wrists and a 3D motion capture system translate her heart rate into sounds, lights, and real-time data shifting in the background behind her.
The rise of biometric artistic performance
These artists are all part of a growing exploration of biometric art, and more specifically the integration of physiological data into artistic performance. In these performances, artists share their biometric data—from heart rate to brain waves—through sensors and systems that translate this information into visual and auditory outputs.
In some cases, the audience’s biometric data are even integrated into the performance. For example, the participatory artwork GROUPTHINK recorded audience members’ heart rate data, allowing the performers to adjust the tempo and mood of their musical performance in response (Hossaini et al., 2022).
The number of publications and explorations of these kinds of biometric artistic performances has been growing steadily over the past few decades, and there are many reasons behind this.
In our recent study Heart and Soul: The Ethics of Biometric Capture in Immersive Artistic Performance (Sparrow et al., 2024), we conducted a systematic literature review that catalogued these kinds of performances across the past 30+ years.
We found that authors and artists felt that integrating biometric capture into performance could establish unique connections between performers and audience members, help evaluate and train performers, create mutual artworks with audiences, and even build a scientific understanding of the performative experience.
However, while these groundbreaking forms of performance are innovative and fascinating, there are also ethical risks hidden under the surface—especially when immersive technology enters the picture.
Immersive technology and the performing body
As immersive technologies such as virtual reality (VR) headsets have become increasingly commercially available, some artists have also begun using these technologies to collect and display biometric data during performances.
While most of the performances in our dataset were what we call ‘artistically immersive’—that is, they immersed or absorbed the audience in a performance through audiovisual effects—some also used VR or mixed reality (MR) to achieve a different kind of immersive biometric performance.
For example, one study collected the blood volume pulse and electrodermal activity of audience members at a classical music concert, then transformed this data into 3D visualisations that were incorporated into a VR representation of the concert (Meng, He, & Kunze, 2023).
As part of their basic functionality, immersive technologies can sense certain kinds of biometric data from their users, including eye gaze, body movement, gestures and head position. In a recent study, scholars were even able to create a system using a Hololens MR headset that could detect blood flow and other vital signs in passersby (McDuff, Hurter, & Gonzalez-Franco, 2017). Because of these capabilities, alongside their ability to depict immersive visual environments, immersive technologies can be appealing as part of a data collection and display system in biometric artistic performance.
Privacy, identity, and biometric psychography
But there are ethical risks associated with the use of such technologies, both within and beyond performance. Unlike bespoke biometric data collection devices made by artists themselves, it can be unclear how commercially available devices store and use biometric data, leaving this data open to misuse and abuse.
Brittan Heller (2020) argues that biometric data is a particularly sensitive type of data. She coins the term “biometric psychography” to refer to the array of biometric data that can indicate private information about an individual, including their health status and even sexual orientation.
When this data is linked to a person’s identity, it can potentially be used in intrusive advertising practices. As Heemsbergen (2024) argues, this ‘consumer data’ is perhaps closer to medical data, as even “unconscious movements can be exploited for emotional insights or to predict neurogenerative disease”.
Some studies have also shown that biometric data that is not collected for the purposes of identification can still be used to identify people through facial recognition (Shi et al., 2021) or even movement detection (Miller et al., 2020), raising further privacy concerns.
Agency over bodily data
In the literature we examined, ethical risks like these were rarely discussed. However, some sources did touch on potential unintended harms that can arise when integrating biometrics into performance.
One recurring concern related to how measuring the biometric data of a performer could impact the agency of an artist, such as by reducing a performance or performer to “a collection of angles and data points” (Smith-Nunes & Shaw, 2021).
“One recurring concern related to how measuring biometric data of a performer could impact the agency of an artist”
This was especially a concern when this data was used to train performers or evaluate a performance, as in attempts to measure audience biometric data (such as heart rate and galvanic skin response) to determine how engaging a performance was (Thomas, 2018). One study found that performance experts had misgivings around this practice since biometric data may obscure the “ups and downs” that make performances meaningful and unique (Latulipe, Carroll, & Lottridge, 2011), ultimately damaging artistic integrity.
Certainly, biometric data can be an unreliable indicator of engagement. Such data can be incorrectly recorded or measured, and interpretations of even robust data may not align with an individual’s self-reported emotions or preferences (Thomas, 2018). Furthermore, evidence suggests that social emotions can be expressed in the body differently depending on a person’s culture (von Suchodoletz & Hepach, 2021).
A few other ethical concerns arose in the literature, including the issue of how performers or audience members could adequately give informed consent for their biometric data to be used as part of a performance or evaluation. In Fairground: Thrill Laboratory (Schnädelbach et al., 2008), rollercoaster riders’ biometric data were displayed to spectators through a live video feed. While amusing for spectators, this prompted important questions: How can riders opt out from live data collection? And what responsibility do rider operators have to stop the ride if a rider’s data reaches dangerous levels?
But art is art
Artistic performances have long questioned and critiqued the role of technologies in our lives, sometimes even by using that technology in provocative ways as part of the performance itself. The Classification Cube installation (Meshi & Forbes, 2020), for example, explored the practices of machine learning (ML) classification systems by using ML algorithms to estimate the age, gender, emotion and actions of visitors based on their facial and bodily characteristics.
The ethical risks associated with immersive technologies in relation to biometric data do not mean that such technologies should be avoided entirely in artistic pursuits. Indeed, in many ways, performances are ideal ‘liminal spaces’ for interrogating them.
Yet these ethical risks do need to be accounted for, particularly when they unintentionally violate artists’ and audiences’ privacy or take away their agency over their bodies.
Towards an ethical framework for immersive biometric performance
We believe that it is the right time to examine the ethical issues associated with biometric capture in artistic performance more closely, especially when immersive technologies are involved. To assist artists and researchers in navigating the complexities of this space, our paper poses a set of ethical questions and potential actions to consider in these performative contexts.
For example, we raise the following prompt: What kind of biometric data is being collected in the performance (heart rate, brain electrical activity, movement, and so on)? What might this data inadvertently—and even incorrectly—reveal about a person, such as in relation to their health? Is it the sort of thing they would prefer to keep private? These questions point to the value of thorough research and planning when collecting biometric data.
Another question we ask is: How is biometric data to be managed in performative contexts? Does the data need to be stored, and if so, how, where and for how long? What kinds of security measures can be put into place? Here, careful consideration of which devices will be used for data collection, as well as their platform policies, is particularly important.
The relationship between artistic performance and technology is a special one, and the ability to share biometric data between audience and artist opens up a world of novel and exciting aesthetic relationships and interactions. We hope that our research not only supports the growth of these art forms, but also furthers discussion of an important question in our digital age: In the process of sharing our hearts, minds and bodies with one another via technology, how much is too much?
References
Dever, A. (2022). Soaring heart rates: Revealing a performer’s inner world. CU Boulder Today. University of Colorado Boulder. https://www.colorado.edu/today/2022/02/25/soaring-heart-rates-revealing-performers-inner-world
Donnarumma, M. (2013). Hypo chrysos: Mapping in interactive action art using bioacoustic sensing. In Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction – TEI ’13, 2013, Barcelona, Spain. ACM Press. 383–385. https://doi.org/10.1145/2460625.2460705
Heemsbergen, L. (2024). Apple’s vision of ourselves. CAVRN. https://cavrn.org/apples-vision-of-ourselves/
Heller, B. (2020). Reimagining Reality: Human Rights and Immersive Technology. Carr Center for Human Rights Policy Discussion Paper Series. https://carrcenter.hks.harvard.edu/files/cchr/files/ccdp_2020-008_brittanheller.pdf
Hossaini, A., et al. (2022). GROUPTHINK: Telepresence and Agency During Live Performance. Proceedings of the ACM on Computer Graphics and Interactive Techniques 5(4), 1–10. https://doi.org/10.1145/3533610
Latulipe, C., Carroll, E. A., and Lottridge, D. (2011). Love, hate, arousal and engagement: Exploring audience responses to performing arts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Vancouver BC Canada. 1845–1854. https://doi.org/10.1145/1978942.1979210
McDuff, D., Hurter, C., and Gonzalez-Franco, M. (2017). Pulse and Vital Sign Measurement in Mixed Reality Using a HoloLens. In Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology (VRST ’17). New York, NY, USA. ACM. https://doi.org/10.1145/3139131.3139134
Meng, X., He, Y., and Kunze, K. (2023). Towards Enhancing a Recorded Concert Experience in Virtual Reality by Visualizing the Physiological Data of the Audience. Proceedings of the Augmented Humans International Conference 2023, 330–333. https://doi.org/10.1145/3582700.3583709
Meshi, A., and Forbes, A. G. (2020). Stepping inside the Classification Cube: An Intimate Interaction with an AI System. In ACM SIGGRAPH 2020 Art Gallery (SIGGRAPH ’20). New York, NY, USA. ACM. 387–393. https://doi.org/10.1145/3386567.3388566
Miller, M. R., et al. (2020). Personal identifiability of user tracking data during observation of 360-degree VR video. Scientific Reports 10, 17404. https://doi.org/10.1038/s41598-020-74486-y
Schnädelbach, H., et al. (2008). Performing thrill: designing telemetry systems and spectator interfaces for amusement rides. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Florence Italy. ACM. 1167–1176. https://doi.org/10.1145/1357054.1357238
Shi, C., et al. (2021). Face-Mic: Inferring Live Speech and Speaker Identity via Subtle Facial Dynamics Captured by AR/VR Motion Sensors. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking (MobiCom ’21). New York, NY, USA. ACM. 478–490. https://doi.org/10.1145/3447993.3483272
Smith-Nunes, G. and Shaw, A. (2021). Doctoral Colloquium-Pandemic Pirouettes: AR Ballet Exploring Data Ethics for the Computing Classroom. In 2021 7th International Conference of the Immersive Learning Research Network (iLRN), 1–3. https://doi.org/10.23919/iLRN52045.2021.9459242
Sparrow, L. A., et al. (2024). Heart and Soul: The Ethics of Biometric Capture in Immersive Artistic Performance. https://doi.org/10.1145/3613904.3642309
Von Suchodoletz, A., and Hepach, R. (2021). Cultural values shape the expression of self-evaluative social emotions. Scientific Reports, 11(13169). https://doi.org/10.1038/s41598-021-92652-8
Tanaka, A. (2019). “Sarah Nicolls performs Suspensions, by Atau Tanaka”. YouTube. https://www.youtube.com/watch?v=HumBMPJUroQ
Thomas, P. (2018). Biometrics: An AFTRS Applied Innovation Research Project. https://www.aftrs.edu.au/wp-content/uploads/2020/09/White-Paper-3-Biometrics_Update_2020.pdf
Recommended citation
Sparrow, L et al (March, 2024). The body as data: Immersive Technology and Biometric Artistic Performance. Critical Augmented and Virtual Reality Research Network, https://cavrn.org/the-body-as-data-immersive-technology-and-biometric-artistic-performance/