Dying in VR/AR/XR: Rehearsing to get it right
Virtual Reality (VR), Augmented Reality (AR) and Extended Reality (XR) are mediating technologies uniquely positioned to explore and experience death and where death might be rehearsed. Like an illusionist’s suitcase of magic tricks, what you see may not be real, but imagination of the intangibility of death is possible with ‘artificial’ design propositions through technological mediation.
Practising one’s own death is a custom that has not been revealed through an available wealth of tangible outcomes. Rather, this is a richly philosophical proposition, where there is no end, nor any beginning. This topic of interest – dying in VR, AR and, or XR is popular in the games and entertainment industry and academia, as well as in some quarters of the field of human-computer interaction (HCI) with attention in the health and palliative care settings. We frame the current exploration as one’s own death, instead of the death of your avatar, which has already been richly investigated.
This post emerges from our involvement on “Passing Electrical Storms” (Gladwell, 2023) an interactive exhibit at the National Gallery of Victoria’s (NGV) Melbourne Now show, that launched five years after the analogous “Second Chance” (Lava Saga, 2018), hosted at the Reimagine (End of Life Week) Festival in San Francisco. Reinforced by a lifetime of video gaming where your avatars death can be rehearsed as many times as you like, the below grapples with some of the literature, ideas and advancements in this area introducing the flourishing potential of digital tools (VR/AR/AI/XR) to enable exploration of death and highlights fundamental questions for further research.
An Immersive Practice of One’s Own Death: Designing Death, Dying and Being Dead
In “Passing Electrical Storms” (Gladwell, 2023) visitors to Melbourne Now, at the NGV were presented with the opportunity to experience their own death. This involved being ushered into a specially designed space, being supplied with ‘waivers’ to proceed, then laying on a wooden bed with heart rate monitor and a VR headset applied. Multiple beds were fashioned into a circular fashioned community. One’s own death proceeded from being worked on by medical experts and attempted defibrillation. Death was virtually announced through the flatlined beeping on the heartrate monitor, and a reversed view of ‘oneself’ on the bed with medical personal announcing time of death. Proceeding through a Power of Ten framework participants tour an animated journey from the medicalised environment to a gaseous outer, outer space. Peaceful, quiet and thoughtful, the virtually artificial journey takes about 10 minutes. At the end, participants may line right up and experience their own death again and again. “Second Chance” (Lava Saga, 2018) hosted at the Reimagine (End of Life Week) Festival in San Francisco followed a similar pattern of a placement of community bodies with VR headset, and vibrant graphics to experience the action of death.
Beyond Death – Digital Afterlife
Work advances in AI to cater for when your favourite granny/lover/friend does cross over in the real. This might involve a mass data scrapping from their Google, Facebook and Insta pages, combined with uploads of the physical photos and voicemails from your significant other. Should you have previously 3D scanned your granny/lover/friend – all the better! AI offers ‘digital twins’ that let loved ones seem to ‘live on forever’. Combining XR and generative AI’s potential to reanimate data through Large Language Models (LLMs) offer ample room for criticism of technological outputs and emerging business models. It also opens more productive critique around how we are to deal with the decomposition and recomposition of dead peoples’ data in an era of generative and immersive technology.
“The inevitability of death is only matched by the inevitability of the data we give off when alive being put to use when we are not.”
On the one hand, criticism of AI-enabled memorials or avatars is framed on ethical critiques and experiential expectations around XR embodiments of the dead being too real but also not real enough; invoking a digital replacement for a no-longer-analogue human jars against common cultural understandings of both humanism and death, while relying on a techno-ideological ideal that is yet to manifest. Yet, we must ask questions about digital designs for dying and being dead as the inevitability of death is only matched by the inevitability of the data we give off when alive being put to use when we are not.
The inevitability of a digital afterlife for ‘digital ghosts’ (Sisto 2020) when our bits expire but our bytes remain shows how death can be considered when mediated across digital networks that socialise the modern world. From the macro perspective, Meta’s Facebook will accumulate more dead ‘users’ than live ones by 2070 or so (Öhman et al. 2020), offering an imagined community-graveyard rich in digital-soil, ready to be recomposed. While there are limited design choices towards how our physical bodies’ decomposing bits recompose, our bytes are more ready to be put to specific interventions.
Grief Spectacle and Service “in XR”
The idea to reanimate our latent bytes in, and with AI, and/for VR/AR/XR experiences is not new. Notable projects to do so include “The Hereafter Institute” (2015) art project, which offered a re-mediated self or loved one in immersive space. Life imitated the art of the dead soon after, with “Project Elysium” purporting to create a similar mediation in real life for recently dead users in Oculus VR Jam 2015. Such work predated the early 2020s Generative AI boom, and subsequently failed on technical grounds that might have obfuscated harder questions.
Yet, this type of work has progressed in industry and academia that by the time the “Metaverse” landed, a complex economy and culture of dealing with death in XR evolved (Kiros, 2023). XR provides novel socio-technical spaces to engage in grief. One participant in Kiros work explained that he “didn’t allow himself to process his sister’s death until he did it through VR. ‘Men, in my society, can’t be seen breaking down. […In the XR service] I was able to put the baggage down’” (Kiros 2023). This development in dealing with death is the trigger for the success of post-life grief services.
One profligate case study that shows the service and spectacle of death in XR is the virtual resurrection of the dead daughter, Jang Nayeon. A Korean TV network offered a grieving mother a 3D approximation of her (dead) daughter in VR, so that they could ‘reunite’ after three years of grieving. The uncanny/unreal experience was interpreted emotionally by the family involved and reflected upon analytically from various research perspectives. The legal ramifications around Nayeon’s ‘personality rights’ (Boothe 2022), and the psychological efficacy of the project as ‘VR interventions on grief” (Pizzolu et al. 2021) are some that have been considered. We might frame this as ‘immersive grief services’ that allow a visceral sociocultural remediation of understandings of death, yet they also open new sociotechnical ways of imagining and materialising what death is and does. Together these perspectives provide a new form of grief service that are experienced in XR/VR/AI.
Gifts from the Digital Soil
However, the race to produce digital twins of the deceased is not the only mode of decomposition and recomposition available in XR. By example, while trying to make a dead-daughter avatar for XR is non-desirable on numerous grounds, techniques of in-memoriam continue to evolve.
A different set of ethical questions is presented if, for instance, you and a loved one had shared – in writing or conversation or art – a beautiful corpus between you. How might this reanimate in a generative experiential phenomenon, not meant as a synecdoche for human life, but as contextual space for reflection and reconstitution of a shared memory? LLMs can regenerate this shared context in ostensibly meaningful new ways that recompose previously shared entanglements in an immersive form; a gift from the shared digital soil of past mediated relations, offered up in new ways. This offering opens profound discussions on the nature of life and memory, identity, and the ethical implications of digital recomposition, when analogue relations that have stopped (through death or other means) but continue to be experienced through immersive, generative media.
Another gift VR/XR may offer from death is that of memory. For example, a nascent market entry is seen by branding captured experiences in Apple’s Vision Pro as spatial video memories. Apple’s Vision Pro promises its 3D video capture to offer a “powerful way to capture and relive our memories” (Apple 2023), as the product makes “Your memories come alive” (Apple 2024). This is not technology spec, but a magic reconstitution of past life. Reviewers of early Apple hardware seem to gravitate towards the language of memory to explain the spatial video experience in the product:
“this was stuff from my own life, my own memories. I was playing back experiences I had already lived” (Stein 2023);
“[I viewed them in] an “Immersive” view where the border of the video becomes glowy and dream-like to give it characteristics of a memory” (Wong 2023).
There is, unremarkably, no mention of death in Apple’s promotional material. This promotion of the potential to select, store, and share digital ‘memories’ is a patented feature of Apple’s device. This way the memory lasts beyond the initial organic user who captured them. For clarity, the ephemeral and intangible of life’s memories are now a product: “[Apple technologies] deliver stunning spatial memories in a compact file size” (Apple 2023). These memories will live on not only when their captured subjects are dead, but also when those who captured them die. The unstated inference from the promotional material is then, that users could keep others’ memories fresh. In this process Apple speaks to novel forms of biospatial surveillance of users via “eye tracking, user mood detection, user emotion detection” (Apple 2023) through biometric characteristics that incorporate into what a memory was and can be.
Designing Better Dying?
This leads us to state that there will be no limits on the experiences designed in XR that extend death research and its associated experiences. We ask, how might rehearsals of death, visceral, intimate and immersive, be reimagined to consider what death is and could be, and could be rehearsed for? How can we move away from the mechanical taxonomies present of death and dying in digital media to consider the cultural-somatic experiences of dying through immersive media? To materialise such future research, the trajectory should consider how to equitably unpack the authority and expertise around the sociocultural considerations of designing death and dying. This means re-tooling game designers with the philosophical, and practical (?) positivistic science to other forms of knowledge creation. It requires considering co-design with communities that exist outside the normative VR/AR/XR economy that remediates old tropes of digital media.
Death in VR/AR/XR
Design in VR/AR/XR allows new ways of asking what might be the shared expectations and experiences of people’s mediated lives as they approach death? What rituals and rights might be included in VR/AR/XR immersive experiences to those who are dying? How might design work be harnessed to consider how to use VR/AR/XR to mediate other models of understanding of the needs at hand? For example, conceptualising XR as digital drugs, Barratt et al. (2022) suggest that digitally immersive sedatives could shift perceptions of treatment and care. All these projects highlight the ways processing for dying might be explored and how to design technologically mediated artefacts for those left behind.We might look to or call for design work with and for theological and spiritual communities: for instance, how do VR churches approach their teaching and spirituality? We might incorporate the debates, experiences, and policies of Medical Assistance in Dying (MAID) design. Although ‘Co-design’ with those who have passed currently proves an impossibility, palliative care knowledge should be integrated.
“A focus on in-memoriam artefacts might more ethically turn to ante-memoriam experiences”
Instead of current industry focus on XR and AI in-memoriam projects that attempt to reconstitute loved ones, we ask what it means to reconstitute the meaning of those relations after death in XR? How can we rehearse the process of death in XR? For the latter a focus on in-memoriam artefacts might more ethically turn to ante-memoriam experiences. Considering things you wish to remember for the first time, as you experience the end of your time, offers one framework to consider the mediated processes of dying, rather than producing an artefact of death. At the same time new forms of memory – mediated through generative and/or immersive technologies offer novel ethics to those still living and producing data for later use.
This entry has considered the evolving intersection of VR/AR/XR and Artificial Intelligence (AI) with concepts of death and dying. Like the biological process of decomposition, we are all beholden to, digital technologies like VR/AR/XR and AI can reconstitute data from deceased individuals, creating new experiences related to death. Immersive exhibits may be played to practice one’s own death. Our focus on mediated technology’s unique ways to experience and interact with death call for new sets of research questions and practices in the design of media for those who are in the process of dying or rehearsing their own demise. As these cultural-technical designs of experiencing death and dying continue to evolve, it is worth pointing back to Kaptelinin (2016), who suggests that iterative design improvement cycles might not be entirely suited for considering people’s perceptions of something that apparently only happens once.
References
Boothe, A. (2022). The death and life of Jang Nayeon: a case for personality rights in the digital layers of reality. International Journal of Law and Information Technology, 30(4), 398-422.
Cuerdo, M. A. M., & Melcer, E. F. (2020, April). ” I’ll Be Back”: A Taxonomy of Death and Rebirth in Platformer Video Games. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-13)
Gladwell, S. (2023). Passing Electrical Storms. Melbourne Now, National Gallery of Victoria, Australia.
Harrer, S. (2013). From losing to loss: Exploring the expressive capacities of videogames beyond death as failure. Culture Unbound, 5(4), 607-620.
Kaptelinin, V. (2016, May). Making the case for an existential perspective in HCI research on mortality and death. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (pp. 352-364).
Lava Saga, (2108) Second Chance. Reimagine – End of Life Week, San Francisco USA.
Moutogiannis, P. P., Thrift, J., Pope, J. K., Browning, M. H., McAnirlin, O., & Fasolino, T. (2023). A Rapid Review of the Role of Virtual Reality in Care Delivery of Palliative Care and Hospice. Journal of Hospice & Palliative Nursing, 25(6), 300-308.
Nicolucci, V. (2019). A death-positive video game for death education of adolescents. Italian Journal of Educational Technology, 27(2), 186-197.
Mo, J., Vickerstaff, V., Minton, O., Tavabie, S., Taubert, M., Stone, P., & White, N. (2022). How effective is virtual reality technology in palliative care? A systematic review and meta-analysis. Palliative Medicine, 36(7), 1047-1058.
Öhman, C. J., & Watson, D. (2019). Are the dead taking over Facebook? A Big Data approach to the future of death online. Big Data & Society, 6(1), 2053951719842540.
Wilson, G., & McGill, M. (2018). Violent video games in virtual reality: Re-evaluating the impact and rating of interactive experiences. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play (pp. 535-548).
Woo, O. K., & Lee, A. M. (2023). Case report: Therapeutic potential of Flourishing-Life-Of-Wish Virtual Reality Therapy on Relaxation (FLOW-VRT-Relaxation)—a novel personalized relaxation in palliative care. Frontiers in Digital Health, 5, 1228781.
Valdes, G., (2018) How virtual reality can help you face your mortality. Variety. As retrieved from https://variety.com/2018/gaming/features/death-and-vr-1202793804/
Reimagine (End of Life Week) Festival 2018 https://letsreimagine.org/sf
Recommended citation
Meyrick, T. Greuter, S., Hawker, K. & Heemsbergen, L.. (February, 2024) Title. Critical Augmented and Virtual Reality Researchers Network (CAVRN). cavrn.org/dying-in-vr-ar-xr-rehearsing-to-get-it-right/