How Augmented Reality Comes to Be

ByLuke Heemsbergen and Andrew Iliadis

A Midjourney generated image of an 'evolutionary tree of tech'. It is a pencil drawing of a large circular tree with different pieces of technology resting in the foliage

Evolutionary tree of tech (Authors-Midjourney)

There is no shortage of diagrams that purport to explain the evolution of technology in society. One fun diagram in the academic study of emerging technology is Geel’s (2002) “dynamic multi-level perspective on [Technology Transitions].” The diagram sure beats popular industry sources like the Gartner Hype Cycle diagram in terms of number of lines and arrows! More seriously, Geels’ diagram holds better explanatory power of how technologies are socialised. Schot and Geels (2008) reproduced the diagram with more complexity (and a lot more arrows) in 2008 and this version is the one we’ve included below (Figure 1) to help think about the socialisation of fields like Augmented Reality (AR).

Figure 1. “Multi-level perspective on transitions (adapted from Geels 2002, 1263)” (Schot & Geels, 2008).

“we use the image as inspiration to consider how technologies are conceptualised, concretised, and contextualised“

The image helps us consider three important processes; namely, how the conceptualisation, concretisation, and contextualisation of ‘tech’ is embodied in forces of markets and user preference, industry, policy, technology, culture, and science that all vie and push into configuring a ‘stable regime’ of technology use with which people identify.

Identifying tech regimes in society at a certain time and place might bring to mind expereinces like ‘renting VHS tapes’ or ‘apps on your smart phone’. Yet products, a tech regime does not make: there was not a ‘Google Glass’ regime. And there does not yet seem to be an “AR” regime. But how would such a regime emerge?

The post focuses on some of the things that happen when AR moves from niche innovation technologies (the starting point in Geels’ diagram) to a technological regime (the end point), and what the consequences are. Rather than try to explain every niche, arrow, and domain within the Schot and Geels diagram, in this post we use the image as inspiration to consider how technologies are conceptualised, concretised, and contextualised. Doing so will help track how sociotechnical imaginaries come to be, are hammered into regimes by market and other forces, and are also reconsidered in everyday use in ways that are often surprising to the makers. That is to say as technologies are ‘contextualised’ intended uses (and users) can run sideways as people use products in ways that were unintended, despite much work on market research, or regulation, etc. 

To be clear, this is about AR. We don’t think VR will be the same. VR seems destined to remain a technological niche. Yes, game engines, the metaverse and all that might be here to stay. But pulling users away from their environment segregates cyber-space by definition—AR promises a different regime.

Conceptualisation

How AR is conceptualised matters and here we want to focus on how AR is imagined as a technology. Technological imaginaries function in incredible ways to inform what is coming from what we know and dream about. Jasanoff (2015:3) sees a need for these “conceptual frameworks that situate technologies within the integrated material, moral, and social landscapes.” She notes how resources like science fiction offer this up “in such abundance;” it was the science fiction writer William Gibson who coined the word Cyberspace, Jules Verne who first took us to the moon. 

At stake for Jasanoff  (2015; 322-323) is how actors’ own technological projects are reflected in the design and fulfilment of collectively imagined forms of social life and order. She sees these imaginaries as 

  • Inviting analysis of the origin of new scientific ideas and technologies and the “social arrangements or rearrangements they help sustain.” 
  • Allowing us to consider how the “merely imagined is converted into the solidity” of routines, things, and identities; and 
  • Showing moments of resistance and power when competing imaginaries struggle to establish a “social terrain” and 
  • Offering ways to consider how unconventional ideas gain traction across time and space.

    Right now, in AR, we can see many of these imaginaries and their trajectories play out. One place to do so is FABRIC, a cultural analytics database at Isabel Pedersen’s Decimal Lab that catalogues how AR (and other wearable tech) is imagined in products and promos (Iliadis & Pedersen, 2018).

    Comparing FABRIC to a decade of Augmented Reality conferences and trade shows, Liao and Iliadis (2021) consider how AR discourses evolve through “Field Configuring Events” that offer the opportunity for numerous actors to come together to create a macro-understandings of how a set of technologies ‘work’:’ what problems do they solve, how can media tell the story, etc. 

    Another place to watch imaginaries play out is the AR research space itself. Kim et al. (2018) write the histories of AR technologies as envisioned by the research-engineers designing them. Cipresso et al. (2018) show a different set of imagined futures (and pasts) through bibliometric analysis of how clusters of AR research evolve over time. Their work, covering up to 2015, seems to suggest that as AR hardware commodified, interaction potentials in ‘clinical’ settings became the focus of AR research (in the Web of Science database at least). While the 1990s and early 2000s were about the technology (interoperability, tracking humans and objects, etc.), the cluster analysis suggest more impactful research shifted to applied settings in medical use and lab-based education studies, and by the 2010s started to reflect the shifts that mobile phones made to making AR applicable across whole cities.

    Heemsbergen and Cadman’s work (2021) borrowed Cipresso et al.’s methodology to see what the picture of AR ideas in research looked like if the next 5 years were included (up to 2020). While we saw similar clusters as citation patterns evolved (Figure 2), what was apparent is that the most recent clustering of research was focussed on AR jumping out of clinical and lab-based applications of the technology, and into peoples’ everyday practice

    A significant shift focusing on a cluster that builds from 2015 (yellow), shows AR research considering how humans (would) interact with AR in their everyday lives. This includes imaginaries from marketing to tourism, to hospitality to culture heritage. While it seems that the AR industry pivoted away from failed consumer expectations (see Google Glass and Magic Leap), various non-AR disciplines began to research their own imaginaries of how AR would mediate their industry or discipline.

    Concretisation

    We think of concretisation as the process of potential imaginaries succumbing to market forces, civic pressure, technological limits, regulation, and other constraints. These pathways of concretisation show how and where ideas that mobilise AR are finding traction and how they run up against one another as disciplines, industries, and consumers try to claim the experience of the AR media they use, and how they want to do so.

    design is a tragic process that de-forms ‘morphe’ (form) as it in-forms ‘hyle’ (matter). Design distorts idea into medium.

    We can think about the varied actors’ ideals of AR (de)forming into material systems of media. Jumping to design theory for a moment, Flusser (1999) talked about design as a tragic process that de-forms ‘morphe’ (form) as it in-forms ‘hyle’ (matter). Design distorts idea into medium. Media are a working design of an idea that people communicate. Flusser (1999: 24, 28) saw information and communication technologies media as capable of moving “past formalising a world taken for granted” to realise “forms designed to produce alternative worlds.” AR and VR offer these worlds, while AR offers it in real time in relation to our own environment.  

    Yet AR does not claim “unmediated reality” ; it affords certain properties and dynamics just like any other media does. These are ‘concretised’ by factors that include the technological, but also include the complex dance of actors and ideas that Geels (2002) sketched out across 6 dimensions at play in creating a technological regime. Ideas and constraints vie for attention and solidification into stable regimes. 

    In many respects, AR is not there yet, as a technological regime. We can look to other media to understand what emerging tech looks like when concretised into a regime. Facebook gives a pretty good example of social media as concretised regime: the economics, uses and users, and political pressures and cultures are widely undersstood. Likewise, Cable television was a thing that was technical and cultural, with an industry and entire infrastructure built around it. 

    Despite Google’s 2013, Magic Leap’s 2017, and Apple’s 2023(?) efforts at products, there is not yet this sense of formalised form which offers what AR is and is to become. How spatial computing technologies ‘extend reality’ is still up for grabs. There are imaginaries, but regulators, consumers, and other actors caught up in the socialisation of AR have not yet hardened their minds to the fluid possibilities of what “augmented publics” consist of and how their component parts balance into a future trajectory. 

    That being said, specific properties and dynamics of AR are built from a technical design constraint of invasive surveillance (Heemsbergen et al. 2021). The amount of biometric and physical-environmental surveillance required to make AR ‘work’ is unprecedented and offers contentious imaginaries that need to be concretised in and across the dimensions Geels sets out. What dimensions come into play here (e.g. regulation or market) is something to think about and shows where to intervene effectively.

    Contextualisation

    It then seems premature to speak of contextualisation if forms of AR have not yet concretised. Yet, it is important to recognise that new media, despite their intended public debut, suffer continued social and cultural appropriation (Gitelman, 2006, p. 27). Users create and relate diverse experiences to each other that afford evolving forms of mediated socialisation. In other words, how users react, engage, and reconfigure AR as media in novel practices matters and should be a site of research. 

    Such ‘unintended’ practices are afforded in context (Costa, 2018; Heemsbergen, 2019) where ideal experiences, users, and other ‘imaginaries’ give way to experimentation and varied and even subversive use. Future study of these spaces and their communities might help researchers consider how to guide potentially relevant futures of AR publics as they continue to emerge and evolve.

    One simple example of this type of contextualisation includes developers experimenting with ARKit’s occlusion feature. Apple introduced this functionality to make AR objects easily ‘disappear’ or be occluded by human forms that walk in front of the objects. This technical feat requires recognising and manipulating human forms. Some developers contextualised this feature by inverting its intended use – they made people seem to disappear from real life altogether! This was put to the test on cars, and other objects that were no longer desirable. 

    What is desirable to see or not see in a specific context of mediated reality is up for debate. The political ramifications of such work are (of course) already found in science fiction, like Black Mirror’s “Men Against Fire” episode. In this story, not only does AR make enemy combatants look like mutants, once discharged, soldiers can perceive their decrepit home towns through ‘rose coloured’ lenses (Figure 3). 

    Figure 3. Composite image by authors from Black Mirror’s “Men Against Fire” (Netflix).

    A real life culturally relevant example of this political in/visibility is explained by Indigital’s CEO, Mikaela Jade, whose acts of refusal in AR show how what is taken with us and what is not into AR culture is a political act. For instance, how should we ‘witness’ the history of massacres of First Peoples in Australian environment?  Mikaela’s answer is: was living it once not enough?Contextualisation also shows how different media systems can combine in ways that their developers/markets/regulators did not intend or imagine (see Akrich 1992). For instance, a person on Twitter decided AI art generators, AR fashion ‘mirrors,’ and voice control should come together:

    Conclusions?

    From these examples, we can see that tech like AR being conceptualised, concretised, and contextualised are not always linear or discrete phases (just look at Geels’ arrows!) but they do influence how these forms come to inform our lives. Each phase presents its own things to consider and levers to influence; research through to regulation should be aware of how to exact maximum effectiveness targeted in each phase. How AR will be socialised is complex, but within the complex mix of actors, forces, and structures that bring technological regimes into being, the conceptualisation, concretisiation, and contextualisation are all matters worth considering.

    References

    Akrich, M. 1992. The De-scription of Technical Objects. In Bijker, W. & Law, J. Eds.  Shaping Technology/Building Society. Studies in Sociotechnical Change, MIT Press, pp.205-224..

    Cipresso, P., Giglioli, I. A. C., Raya, M. A., & Riva, G. (2018). The Past, Present, and Future of Virtual and Augmented Reality Research: A Network and Cluster Analysis of the Literature [Original Research]. Frontiers in Psychology,9(2086). https://doi.org/10.3389/fpsyg.2018.02086 

    Costa, E. (2018). Affordances-in-practice: An ethnographic critique of social media logic and context collapse. New Media & Society, 20(10), 1461444818756290. https://doi.org/10.1177/1461444818756290 

    Flusser, V. (1999). The shape of things : a philosophy of design (A. Mathews, Trans.). Reaktion. 

    Geels, F.W. 2002. Technological transitions as evolutionary reconfiguration processes: a multi-level perspective and a case-study. Research Policy 31, no. 8/9: 1257–74. https://doi.org/10.1016/S0048-7333(02)00062-8

    Gitelman, L. (2006). Always already new: Media, history and the data of culture (Vol. 1). MIT press Cambridge, MA.

    Heemsbergen, L., 2019. Killing secrets from Panama to Paradise: Understanding the ICIJ through bifurcating communicative and political affordances. New Media & Society, 21(3), pp.693-711.

    Heemsbergen, L., & Cadman, S. (2021). COMMERCIAL AND RESEARCH CLUSTERING OF AUGMENTED REALITY: DISCOURSES AND DIVIDES BETWEEN AR APPS AND APPLICATIONS. AoIR Selected Papers of Internet Research, 2021. https://doi.org/10.5210/spir.v2021i0.12180

    Iliadis, A., & Pedersen, I. (2018). The fabric of digital life: Uncovering sociotechnical tradeoffs in embodied computing through metadata. Journal of Information, Communication and Ethics in Society, 16(3), 1-18. https://doi.org/10.1108/JICES-03-2018-0022

    Jasanoff, S. (2015). Future imperfect: Science, technology, and the imaginations of modernity. In S. Jasanoff & S-H Kim (Eds.), Dreamscapes of modernity: Sociotechnical imaginaries and the fabrication of power (pp. 3). University of Chicago Press.

    Kim, K., Billinghurst, M., Bruder, G., Duh, H. B.-L., & Welch, G. F. (2018). Revisiting trends in augmented reality research: a review of the 2nd decade of ISMAR (2008–2017). IEEE transactions on visualization and computer graphics, 24(11), 2947-2962. 

    Liao, T., & Iliadis, A. (2021). A future so close: Mapping 10 years of promises and futures across the augmented reality development cycle. New Media & Society, 23(2), 258-283. https://doi.org/10.1177/1461444820924623

    Schot, J., & Geels, F. W. (2008). Strategic niche management and sustainable innovation journeys: theory, findings, research agenda, and policy. Technology analysis & strategic management, 20(5), 537-554. https://doi.org/10.1080/09537320802292651

    Recommended citation

    Heemsbergen, L. & Iliadis, A. (May, 2023) How Augmented Reality Comes to Be. Critical Augmented and Virtual Reality Researchers Network (CAVRN). link