From Calming Spaces to Superpowered Avatars: Exploring How VR and AR could Enhance Health and Well-being by Recreating Spatial, Social, and Feedback Reality
Virtual and Augmented Reality technology has seen rapid advances in the past decade. Previously, entering into a virtual environment often meant that users had to put on a heavy enclosed headset connected to a desktop PC through thick cables or wear 3D glasses and walk into a specially constructed video-theatre like room where the 3D environment is projected onto the surfaces of the room. Nowadays, there are number of affordable lightweight head mounted display devices readily available to teleport users into highly immersive online virtual environments which can be inhabited simultaneously by hundreds of other users in real-time. Even most conventional smart phone devices are now able to extend objects from the virtual space into the physical world with high fidelity. As such, it is not surprising that VR and AR technology has seen increasing use across a wide range of domains such as in education, architecture and engineering, with one of the most prominent fields being healthcare.
In this literature review, we highlight three intriguing ways in which virtual environments have been used to support physical health and enhance mental well-being. We describe 1) how physical and spatial reality could be manipulated through VR to create calm, meaningful or engaging spaces which could help improve the quality of life for people with dementia, 2) how social reality could be recreated through the embodiment of avatars with identities and characteristics that differ from oneâs own and how this form of virtual embodiment in an online virtual environment could improve self-worth for people with physical disabilities or be used to reduce biases as part of cognitive training exercises and 3) how virtual feedback created from real-world exercise interactions could be manipulated to either increase the level of challenge or sense of self-competence for people undergoing physical exercise. In each case, we examine previous work carried out in that specific domain and then highlight the opportunities, issues and critical challenges that still exists. Finally, we discuss a key trend which we believe would have a substantial impact on the future of VR/AR in healthcare: the integration of high performing generative AI models with various components of the virtual world, which would result in them becoming more intelligent and personally meaningful.
Recreating physical reality to create calm and meaningful spaces for people with dementia
People with dementia, especially those at the later stages of their condition, tend to exhibit various Behavioural and Psychological symptoms (BPSD), such as depression, aggressive behaviour and wandering which could substantially reduce their quality of life as well as that of their loved ones. A number of studies have shown that the immersive reality created within a VR headset could have the potential reduce such symptoms (Rose, et al., 2021). Through VR, people with dementia can be teleported to a calm and relaxing environment (peaceful forest, beach etc.), one which does not contain any elements that may trigger their aggressive behaviours (crowds, environments with loud noises etc.). This virtual environment serves as a private space for the individual, one which provides a brief respite from the ambiance of the care home which they are usually limited to. The potential of VR in dementia care is particularly prominent when integrated with Reminiscence therapy (Huang & Yang, 2022). In this therapy, people with dementia are usually presented with various artifacts which relate to a personally meaningful past memory, either news stories of important events which occurred when they were young or photographs and videos that relates to a particularly memorable moment from their past. Using these artifacts as a medium, participants are asked to speak out and reflect on their past experiences, a practice which has been found to help maintain or slow down the loss in their sense of identity due to dementia. The immersive reality recreated through VR, (either through 360 videos or a computer-generated VR space) provides a more salient way for people with dementia to access those memories and also has the added advantage of allowing users to quickly switch between various immersive environments, allowing them to experience and reflect on past experience in different places without the need to physically travel there.
While VR head mounted displays can be used to create an enclosed virtual space to capture the attention of users, people with dementia are usually only passive viewers of such experiences. AR on the other hand, particularly in the form of a projected virtual environment, has been used to provide an engaging experience that even people with severe levels of dementia could actively participate in. By using a projector fixed faced-down from the ceiling to project various virtual objects (flowers, fishes etc.) onto a table right in front of people with dementia and using gesture-based sensors to capture their interactions with those objects, developers were able to create simple interactive and playful virtual environments which were able to engage people with dementia for prolonged periods (Anderiesen, 2017).
Such users became captivated by playful activities such as sweeping through virtual leaves projected onto the table or playing with the projected fish that were swimming around. This in turn helped improve their mood and made it easier for them to build rapport with the care staff who often played alongside them.
Overall, these examples demonstrate the potential of VR and AR technology in creating a virtual space that provides a meaningful experience and calming distraction for people with dementia. Yet careful consideration needs be taken when designing such virtual spaces for these users. First, not all people with dementia would enjoy the experience of being enclosed in a head mounted VR display, and in several cases, users were afraid and anxious of the dark space shown in the VR device before the actual video is played (Tabbaa, et al., 2019). In addition, stereoscopic 3D videos, which generally is more immersive as they provide a sense of depth, was found to be quite disorienting to people with late-stage dementia and normal 360 videos were generally used instead (Tabbaa, et al., 2019). Careful consideration would also need to be taken when selecting content for people with dementia, as virtual environments showing too many stimuli and events (e.g. people dancing within a music festival or a busy traffic road) were distracting and tend to cause users to become more anxious and frustrated as they were not able to process all of the content. Finally, there is also the broader question of whether the use of VR/AR as a means of distraction is an ethical and appropriate practice, and whether this could have long-term adverse effects for people with dementia. Would the immersion in alternative realities through VR complicate the ability of people with dementia to cognitively recognize and process information when they return to the real world, as certain conditions of dementia could cause people to be prone to visual hallucinations? Would the effect on the quality of life of VR be strong enough to justify the cost and effort in training care staff to learn how to use this novel form of technology as opposed to traditional practices? Several ongoing longitudinal studies are being conducted to address such questions.
Recreating social reality through the embodiment of an avatar with distinct identities and characteristics
In the previous example, we highlighted how the immersive reality provided through VR has the potential to âtransportâ users from their physical reality to a virtual space, either one which invokes a calm and peaceful experience, or one filled with engaging and meaningful events. In social virtual worlds, multiple people can come together and inhabit these virtual spaces at the same time, therefore providing users with diverse opportunities for social interaction and experiences. Interestingly, similar to how the spatial realities created from VR could be distinct from oneâs physical reality, social interaction in a virtual space also offers the opportunity for individuals to embody virtual identities which are distinct from oneâs real-life self, opening up new and old possibilities for exploration and self-expression.
“Social interaction in a virtual space also offers the opportunity for individuals to embody virtual identities which are distinct from one’s real-life self…”
Social interaction in virtual environments are often mediated through virtual representations known as avatars. For people with limitations such as physical disabilities, avatars allow users to bypass various physical constraints and interact with others in ways that would otherwise be challenging in real-life. While offline, their social lives could be limited due to the difficulty in traveling to meet new people or to participate in events, within the virtual world however, they were able to connect to people from all over the world and join various online communities. Some studies even suggest that by embodying an avatar which represents an able-bodied version of themselves, users can experience a greater sense of equivalence as they are able to communicate with others more freely without feeling uncomfortable from displaying their disability (see (Ginsburg, 2020)). Yet studies have also found that despite such affordances, not all users with disabilities wished to be represented by able-bodied avatars. Some users preferred to be represented more authentically in the virtual world and purposely chose avatars that manifested symbols of their disabilities such as wheelchairs, canes and guide dogs as they still wish to embrace their real-life identity even in a virtual world (Boellstorff, 2008; Kent, Ellis, Jones, & Bennett, 2015).
The term âproteus effectâ was coined to describe the phenomenon, of how the behaviour of a user in the virtual environment could be influenced by the appearance and characteristics of their avatar (e.g. taller avatars behaved more confidently and people with attractive avatars tend to self-disclose more information and had shorter interpersonal distance during social interaction) (Yee & Bailenson, 2007). This effect might have also been shown by people with disabilities who embodied able-bodied avatars, as such users generally reported feeling more confident in themselves, more willing to self-disclose personal information and more relaxed when socializing through an avatar in past studies (Kleban & Kaye, 2015). This in turn contributed to a general feeling of well-being and self-worth. Overall, participating in such spaces has been shown to significantly lower depression and anxiety and lead to higher life satisfaction and self-esteem for people with disabilities (Gilbert, Murphy, Krueger, Ludwig, & Efron, 2013). Our own studies also found similar results with older people, who were often homebound through or conditions such as strokes or age-related disabilities. Such users were able to engage satisfactorily in highly immersive social activities, expand their social network, develop deep interpersonal relationships and continue to engage with various social activities which they could no longer do anymore in the real-life in the virtual environment (Siriaraya, Ang, & Bobrowicz, 2014).
In addition, this ability to assume a separate identity in a virtual space may prove advantageous and could be beneficial in the context of cognitive training to improve mental health and well-being. As part of a proof-of-concept therapeutic strategy to treat people with anorexia, VR was used to âbody swapâ patients into an avatar with the virtual body of a healthy-weight individual who was then asked to carry out a body-size estimation task within the virtual body. Users became able estimate their body size more accurately in the short term with the VR experience (Serino, Polli, & Riva, 2019). In another example, domestic violence offenders embodied female avatars as they entered the virtual space and experience abusive speech and violent behaviours from a virtual abuser. By experiencing what it was like as a female victim who was abused, participants were able to better recognize negative emotions form female faces and the exercise helped reduced their cognitive bias in misrecognizing fearful faces as happy faces (Seinfeld & et al, 2018). In a similar manner, implicit bias against older people were shown to be reduced when young users assumed the virtual identity of an elderly Einstein. Overall, these instances demonstrate how the body ownership illusion afforded by the embodiment of a virtual avatar whose identity and appearance are different from oneâs own, has the potential to bring about positive changes in self-perception and reduce maladaptive biases by allowing people to experience another personâs perspective and walk in their (virtual) shoe.
Of course, the use of the avatar to assume a separate identity or body appearance has been a controversial topic, particularly when this could be seen as a form of misrepresentation, or in the extreme case, deception by other users. In our own works for example, several elderly virtual world users who selected âyoungâ avatars have reported how they were marginalized against when others found out about their real age, either by their conversation partner directly informing them of their unwanted presence or leaving the virtual space abruptly (Siriaraya, Ang, & Bobrowicz, 2014). On the opposite end of this notion, there have also been several reported cases which suggested that adopting a virtual identity could also lead to an open invitation to be marginalized based on that identity. Researchers choosing avatars of young children for example, have report various forms of harassment, insults and threats targeted against their them in their assumed identity while in the metaverse (Dwivedi & et al., 2022). Such forms of âvirtual ageismâ whether against ones real or virtual identity and other forms of âvirtual marginalizationâ, does highlight several interesting issues that would need to be addressed. In virtual worlds designed mainly for entertainment such as in MMORPGs, such issues could be less prominent as there is often a shared assumption that players who chose to enter such spaces also implicitly choose to adopt the customs and rules of that playful space to immerse themselves and others in their play experience. Yet, for social virtual worlds whose purpose often goes beyond entertainment, there is an open question of how best we could go about balancing this freedom to choose different identities and what kind of regulation would be needed regarding the selecting of oneâs identity, so that users could benefit equally from the ability to embody different identities in a virtual space. Perhaps one solution would be to adopt a similar stance as in online games, with certain areas allowing for roleplaying and others regulating against such practices.
Recreating outcome reality by manipulating interaction and feedback to enhance the physical exercise experience
Another quality of virtual environments, which is beginning to be explored and utilized by VR researchers in healthcare, is the ability for the developer to freely manipulate the interaction and corresponding feedback of each user within the virtual space. Researchers designing virtual spaces to promote physical exercise, in particular, have made use of this mechanism to great effect. While we have seen various examples of VR devices being used in conjunction with exercise equipment (e.g. cycling machines or treadmills) to generate a more engaging virtual environment for users to ârunâ or âcycleâ through while at the gym, recent studies have shown that the exercise performance of the users in the virtual space does not always need to be programmed to matched their actual performance with the equipment and the discrepancies between the two could be manipulated to influence the behaviour of users in interesting ways. For instance, by manipulating the numbers on the virtual speedometer when users are riding a VR bicycle that is linked to a real world cycling machine, researchers were able to trick users into cycling 15% faster without them realizing, highlighting how a form of âperformance deceptionâ could be built into their system to frustrate users into putting more effort in their exercise (Löchtefeld, KrĂŒger, Gellersen, 2016). Alternatively, the performance of users in the virtual space could also be exaggerated to create a feeling of having superhuman strength and improve their sense of competence, thus motivating people to exercise more, such as in the case of a VR jumping exercise game (Wolf, Rogers, Kunder, Rukzio, 2020).
The disassociation of oneâs physical self while exercising, by projecting the results of their actions through a virtual avatar, might not only affect the psychological state of users, but could even influence their physiological state as well. In a weightlifting exercise experiment, researchers had found that participants reported less pain intensity and effort when they viewed themselves (i.e. their avatars) lifting weights through a head mounted display VR device, in comparison to a control group, despite lifting the same amount of weights (Matsangidou, et al., 2019). Similarly, in one of our own studies, we had developed an AR running system that allows users to compete against their past records in real-time by projecting them as a âghostâ that would run against them in a spatialized 3D virtual environment. In this way, users would hear the sound of footsteps and breathing from their ghost opponent grow louder as they catch up or slowly fade further away as they run further past their opponent. A preliminary study showed that participants running against their AR ghost opponent ran at a faster speed and reported higher levels of perceived competence without increasing their average heart rate (Kiriu, Mittal, Siriaraya, Kawai, & Nakajima, 2019). A comparable self-competition mechanism was used in a VR cycling game, which also led to an increase in exercise intensity and perceived competence without any significant effects on perceived exertion, enjoyment and heart rate (Farrow, Lutteroth, Rouse, & Bilzon, 2019).
Despite these promising results, research in this domain is still at the early stages and several questions still remain regarding the practice of manipulating exercise feedback and perceived performance through the use of VR. First and foremost, artificially enhancing or reducing one’s perceived performance could be harmful, as they could result in users overexerting themselves while exercising in VR or could create false confidence when exercising in the real world which could lead to injuries. Whether such incidences would be commonplace and how could they be best prevented are still unknown questions that would need to be addressed.
In addition, one of the joys of engaging in a prolonged exercise regime is the ability for the individual to gradually build up a sense of self-efficacy (i.e. the confidence and self-belief of being able to accomplish increasingly challenging goals) and mastery over the exercise activity. Would deceiving users about their capabilities while in VR skew self-belief in their ability as well as their perceived sense of progression when they exercise in the real world and would this damage their motivation? Furthermore, would such practices even remain effective in the long run, as once users become aware of the deception, it is less likely to be effective and users could also become accustomed to the manipulated feedback without realizing. We had seen some possible indications of this when we attempted to replicate our earlier AR running study as part of a prolonged training regime. Although, as previously mentioned, we saw an increase in running performance and perceived competence in a one-off controlled session, participants did not show significant improvements in their training results compared to a control group when they trained over a one week period, neither did participants, on average, run at a faster speed in the condition when the speed of the virtual ghost representing their best performance was increased by 10% without informing them (Siriaraya, et al., 2023). Overall, while unanswered questions such as these still remain, the potential of utilizing manipulated feedback in VR to promote exercise is a promising field that warrants further exploration.
Future trends in virtual healthcare: The fusion of AI with XR
In the past few years, we have seen several advancements that have made virtual environments more accessible (the commercial release of affordable high performing head mounted display devices), immersive (higher resolution displays and advanced sound and visual rendering algorithms) and interactive (more sophisticated tracking and haptic feedback system). However, one trend which we anticipate would have a substantial impact in the near future for VR, particularly in the domain of healthcare, is the use of AI technologies to make virtual environments more intelligent and personally meaningful. Of particular interest is the possible fusion of high performing AI models with various components of the virtual environment. With virtual agents for example, while visual graphics has advanced to a significant degree that they are able to create characters who have realistic physical appearances, past AI technology has yet been able to produce characters who are able to behave and respond flexibly and intelligently enough for users to fully believe in them. This has so far limited their use in healthcare (as a therapeutic agent or a virtual coach for health management programs etc.). However, given the rapid improvements we are recently seeing with large language models such as GPT-4, realistic looking virtual agents with a high degree of conversational and behavioural intelligence could now be created and integrated into a VR environment. This opens up a number of interesting options for VR-based healthcare. We could easily imagine for example, how, a virtual ecosystem, mixed with both intelligent AI and human characters could be created to help sensitize people with extreme social anxiety to different social situations, such as allowing them to give a public debate in front of a virtual crowd. Chatbots, integrated with this new technology, could be embedded into a relaxing VR setting for a therapeutic listening session or as a coach in a VR exercise environment controlled using real-world workout equipment to facilitate an immersive hands-on virtual exercise regime. In addition, we are already seeing promising results from several proof-of-concept AI models developed to automatically generate 3D environments from text-based user instructions. Such models would, in essence, allow users to âdictateâ past memories into immersive 3D environments on-the-fly, which would have clear practical implications for VR based reminiscence therapy. Finally, in the context of AR, we are also seeing the emergence of geospatial generative AI algorithms, which are able to produce content based on map and location information. By using, geo-tagged data from Twitter, Flicker or Open Street maps, we are able to develop AI models to generate virtual crowds based on the characteristics of the space to improve social immersion (such as people talking about how nice their picnic is at a park) or to create content specific to the features of the location (such as an Aesop story about two crabs talking next to a river bank) to be used to teach English in AR (Siriaraya, Kiriu, Kawai, & Nakajima, 2018).
“Yet for all the recent hype surrounding the capabilities of Generative AI, we should be careful of how we integrate AI into VR systems designed to support healthcare…”
Yet for all the recent hype surrounding the capabilities of Generative AI, we should be careful of how we integrate AI into VR systems designed to support healthcare, lest we also unintentionally incorporate the various problems currently encountered when AI is implemented in this domain into our VR systems as well. Of particular concern are issues related to safety. Although as a whole, researchers have found current chatbot technology to be safe, we have also seen examples of them propagating biased information and stereotypes and unfairly representing certain user groups, as such models tend to be trained based on online data that has not been curated and tend not to be equally representative (Ray, 2023). In our own works developing a social support and grief care chatbot, we have had to build in an add-on machine learning model to prevent the chatbot from giving harmful advice when participants are depressed and suicidal. If such issues are not addressed, they could result in the immersive capabilities of VR inadvertently amplifying the AI model’s worst tendencies, resulting in an outcome that could be multiple times worse.
References
Anderiesen, H. (2017). Playful Design for Activation: Co-designing serious games for people with moderate to severe dementia to reduce apathy. Doctoral dissertation, Delft University of Technology.
Dwivedi, Y. K., & et al. (2022). Metaverse beyond the hype: Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy. International Journal of Information Management 66, 102542.
Farrow, M., Lutteroth, C., Rouse, P. C., & Bilzon, J. L. (2019). Virtual-reality exergaming improves performance during high-intensity interval training. European journal of sport science, 19(6), 719-727.
Gilbert, R. L., Murphy, N. A., Krueger, A. B., Ludwig, A. R., & Efron, T. Y. (2013). Psychological benefits of participation in three-dimensional virtual worlds for individuals with real-world disabilities. . International Journal of Disability, Development and Education, 208-224.
Huang, L. C., & Yang, Y. H. (2022). The Long-term Effects of Immersive Virtual Reality Reminiscence in People With Dementia: Longitudinal Observational Study. JMIR Serious Games, 10(3), e36720.
Kiriu, T., Mittal, M., Siriaraya, P., Kawai, Y., & Nakajima, S. (2019). Development of an acoustic ar gamification system to support physical exercise. Proceedings of the 27th ACM International Conference on Multimedia , (pp. 1056-1058).
Kleban, C., & Kaye, L. K. (2015). Psychosocial impacts of engaging in Second Life for individuals with physical disabilities. Computers in Human Behavior, 45, 59-68.
Löchtefeld, M., KrĂŒger, A., & Gellersen, H. (2016). DeceptiBike: Assessing the Perception of Speed Deception in a Virtual Reality Training Bike System. Proceedings of the 9th Nordic Conference on Human-Computer Interaction, (pp. 1-10).
Matsangidou, M., Ang, C. S., Mauger, A. R., Intarasirisawat, J., Otkhmezuri, B., & & Avraamides, M. N. (2019). Is your virtual self as sensational as your real? Virtual Reality: The effect of body consciousness on the experience of exercise sensations. Psychology of sport and exercise, 218-224.
Ray, P. (2023). ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems. Internet of Things and Cyber-Physical Systems.
Rose, V., Stewart, I., Jenkins, K. G., Tabbaa, L., Ang, C. S., & Matsangidou, M. (2021). Bringing the outside in: The feasibility of virtual reality with people with dementia in an inpatient psychiatric care setting. Dementia, 20(1), 106-129.
Seinfeld, S., & et al. (2018). Offenders become the victim in virtual reality: impact of changing perspective in domestic violence. Scientific reports 8.1, 2692.
Serino, S., Polli, N., & Riva, G. (2019). From avatars to body swapping: The use of virtual reality for assessing and treating bodyâsize distortion in individuals with anorexia. Journal of clinical psychology, 75(2), 313-322.
Siriaraya, P., Ang, C., & Bobrowicz, A. (2014). Exploring the potential of virtual worlds in engaging older people and supporting healthy aging. . Behaviour & Information Technology, 33(3), 283â294.
Siriaraya, P., Kiriu, T., Kawai, Y., & Nakajima, S. (2018). Using open data to create smart auditory based pervasive game environments. Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts , (pp. 611-617).
Siriaraya, P., Kiriu, T., She, W. J., Mittal, M., Kawai, Y., & Nakajima, S. (2023). Investigating the use of Spatialized Audio Augmented Reality to enhance the outdoor running experience. Entertainment Computing, 44, 100534.
Tabbaa, L., Ang, C. S., Rose, V., Siriaraya, P., Stewart, I., Jenkins, K. G., & Matsangidou, M. (2019). Bring the outside in: providing accessible experiences through VR for people with dementia in locked psychiatric hospitals. Proceedings of the 2019 CHI conference on human factors in computing systems, (pp. 1-15).
Wolf, D., Rogers, K., Kunder, C., & Rukzio, E. (2020). Jumpvr: Jump-based locomotion augmentation for virtual reality. . Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, (pp. 1-12).
Yee, N., & Bailenson, J. (2007). The Proteus effect: The effect of transformed self-representation on behavior. Human communication research, 33(3), 271-290.
Recommended citation
Siriaraya, P. (June, 2023) From Calming Spaces to Superpowered Avatars: Exploring how VR and AR could enhance health and well-being by recreating spatial, social, and feedback reality. Critical Augmented and Virtual Reality Researchers Network (CAVRN). link
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.