Pixel Theory

(Or, have we ever been physical?)

Still from Shinjuku Calling (Yamagami Yukihiro 2014)

Layering of the virtual on to the real is usually how Augmented Reality is described and explained, but layering is a strange metaphor to understand what is at stake when considering the physical-digital environments we live with. 

As provocation, consider when typographic ink lands on physical paper; the newspaper or a billboard exists as a real thing. But, if pixels flick on a liquid crystal substrate, a virtual experience sits apart from our physical life. There’s a clear atomic-electronic divide when we conceptualise technology. But does this divide remain useful when we consider how ‘technologies’ mediate our lives? 

---

“More useful is considering how AR can mediate our shared world in ways newsprint cannot…”

---

Similar to how the study of ‘cyberspace’ has moved on to realise embedded, embodied and interwoven digital experience (Hine, 2015), considerations of AR media need to move on from dyads that separate the electronic and atomic, and enter the real.

This post is about whether Augmented Reality is real.

Two adverts for Snap Inc., with pixels and pantones spatially augmenting a city. Which is more real?

---

Background Art

This post was, in part, inspired by Yamagami Yukihiro’s installation titled Shinjuku Calling (2014), pictured above. The, installation of ‘mixed media’ helps show how a layered divide between analogue reality and digital virtuality doesn’t explain our lived – and mediated – experience.

Currently (2023) hanging in the National Gallery of Victoria, Yukihiro’s work cleverly integrates hitherto divided atomic and electronic information. Yukihiro painstakingly pencilled a streetscape of Shinjuku Station, across about two meters of white plywood. Onto these physical pigments, pixels are projected that give a virtual ‘real life’ video of pedestrians and cars navigating the canvas; neon glows reach across the pencil lines of stencilled signs; sunset makes the buildings glow. The virtual imbues life as we expect and experience it, past the monotone sketch. It is a breathtaking work that shifts viewers into a real time perception of a distant space as they physically traverse a two dimensional – physical but unreal – pencilled detail.

One way to consider the work is the technical ‘magic’ of graphite layered on plywood, with photons dancing on top, which makes something new through layers of virtual and material representations. Yet, the work is not powerful by overlaying the virtual on the physical.

Its power to make real is by relating electrons to atoms: we perceived it as a ‘space’ in full, through the analogue line art and pixelated ghosts of movement and light. The experience of Yukihiro’s work opens up how pixels can relate to pigments; electrons to atoms. It helps focus discussion how we might think about AR as real, instead of a confusing layering of terms around virtuality, reality, physical, and digital.

The argument begins by suggesting that AR being explained as layering the digital on the physical or the virtual on the real, is less useful than it seems. This holds true to other media. Is it useful to consider the key to the medium of screens how pixels layer on otherwise inactive glass? Or for that matter, how print ink forms layers atop a physical reality of cellulose broadsheets or vinyl billboard infrastructures? This is not what we focus on when considering how humans perceive these media and act on their messages. 

Considering the novel ways how AR ‘layers’ onto extant environments, and who decides, remains an important but somewhat limited way of understanding these emerging media. In short, foregrounding the metaphor of layering serves to distinguish between layers – one virtual one real. Yet, life is not lived in layers, but in what they connect.

---

“the metaphor of layering serves to distinguish between layers – one virtual one real. Yet, life is not lived in layers, but in what they connect”

---

Otherwise, our descriptions of life suffer a latent digital dualism (Jurgenson, 2011), which cleaves our mediated perceptions of life into atomic and electronic domains to the detriment of understanding (real) life – and to the detriment of future AR research in that service. So, the remainder of this post offers some initial thoughts on what can be done to serve as a corrective to virtual-real divides in AR research.

So What?

Before jumping into the theoretical thick of it, it is fair to ask ‘so what?’. 

Simply put, understanding AR as mediating the real is crucial when forming relevant governing regimes now and in future. AR is not a virtual layer on life, it is “Real Life”. For the policy equation, it is crucial to focus regulatory power into buckets that the state (courts, parliament, regulators) understand as real life. My favourite example of this is fire code; Building design – the ‘virtual’ representations of architecture within which we gather – is regulated in a way that ensures we can’t step into a building and sign away our bodily rights with a click of a checkbox. Fire codes matter.

Yet the ways in which AR relates our bodily data to actors that might ‘burn us’ are profligate, considering who gets our data and how it’s used at the whim of a checkbox. The ‘virtual’ in AR creates bio-spatial surveillance regimes (Heemsbergen et al. 2021) that form real world consequences. Personal data available in AR is being used in ways not imagined when it was offered up, such as when gate analysis or eye movements captured by AR system predict chronic disease. Electro-atomic codes matter. 

---

“Electro-atomic codes matter. “

---

Past the policy concerns is delightful and abstruse philosophy of technology vis-a-vis these emerging media. It asks what happens when we conceptualise AR as something other than display technologies that are often defined through virtual / real divides. 

What if we conceptualise Augmented Reality media as making (a novel techne of) relations available between data-rich objects in an environment – regardless of whether those objects are made from atoms or electrons? 

With apologies to Barad (2012), my theoretical interest is in the radically relational accounts of perceptible life that AR can make visible and knowable. 

In this sense, media mediate social reality away from categories of ‘virtual’ or ‘real’ and instead serve as ‘knowledge objects’ that mediate what was previously perceptually inaccessible to humans (Bleeker et al., 2020). This mediation creates and is constrained by not just technical factors but the imagined publics (or networked publics or refractions therein) that emerge, and the economic systems that grind along in ways that might or might not produce a ‘metaverse’ of experiencing reality. As Couldry (2012; 2) points out media offer an ecology of ‘infrastructures’ that make and distribute content in forms that carry particular contexts with them.

Here, I am less concerned with product or organisational critiques, but considering the ‘augmented subjectivity’ (Rey and Boesel 2014) that references the co-production of physical and the digital to define (post-)human experience. Such intellectual concern is, for example, focussed on how AR media offer real-time computationally mediated perception (Chevalier and Kiefer, 2020). Below, I unpack these theoretical claims and offer a tentative path forward.

Divides

Atoms can present information physically, in ways that take up material form. They are real. Electrons present information unencumbered of specific form, they are virtual. 3D printers – if considered as a communication device – might sit betwixt this divide to prove the rule. Or from another perspective, 3D prints show the power of crossing divides of ‘physical’ information with ‘digital’ information and thus imply the power AR or VR might have as they interact with the physical world in their own way.

To explain this power, foundational work on Augmented Reality conceptualised a virtuality continuum (Milgram et al. 1995) that splits the “Virtual Environment” across a spectrum from the “Real Environment”. ‘Virtual’ environments are computationally created and have endless programmable possibilities through virtual objects in virtual worlds. On the other hand, the ‘real’ is a world governed by Newtonian physics. 

Ironic to much scholarship that cites it, the details of Milgram et al.’s 1995 paper seeks to move past the ‘comparatively obvious distinctions between the terms “real” and “virtual”’ (1995: 291) through three dimensions that measure towards (or away from) the illusion of an unmediated ideal ‘reality’. This unmediated ideal is explicitly referenced via Naimark’s (1991) taxonomy of ‘Realspace Imagery’, which forecasts the capacity to record and display sensory information indistinguishable from the unmediated reality of human perception. While this was useful to come to terms with new display technologies, it seems a poor way to think about media, in relation to our lived reality.

When we consider the human experience, assumptions on what is virtual or not, what is reality or not, can seem misplaced. Starting from a point that equates reality to the physical is not very accurate. The physical world and reality are not interchangeable concepts; differences between reality and our perception of it range from biological to sociological mediations. Are glyphs virtuality or reality when carved in rock? layered on paper with graphite? Shone through liquid crystal? How do any of these media make reality?

We have never lived unmediated lives

The divide between atoms and electrons, physical and ‘not’ is less than accurate in understanding the human experience – as explained from a variety of disciplines. 

From neuro-evolutionary perspectives (Hoffman 2019), physical reality itself is only perceived in particularly useful ways to keep life going; neuro-biological reality is mediated in ways that provide best ‘fit’ to succeed. From another perspective, Harari (2014; 41) suggests that sapiens are unique in their capacity to “transmit information about things that do not really exist, such as … nations, limited liability companies, and human rights”. This has been done through verbal language, written documents, social institutions, etc. We have never been merely physical.

Or, putting it another way, meaning in human experience must be mediated – reality is the media that we make up; we consensually hallucinate them into being, or they would otherwise not exist. 

The classical hallucination of the digital age comes from William Gibson (1984; 51):

“Cyberspace. A consensual hallucination experienced daily by billions of legitimate operators, in every nation…Lines of light ranged in the nonspace of the mind, clusters and constellations of data”

More recently the duality of a cyber-space, differentiated and ghettoised from other ‘spaces’ of interaction has been called into question (Jurgenson 2011; Rey et al 2014). The ‘technology’ of cyberspace is receding into the background as terms like ‘spatial computing’ become redundant for the augmented subjectivity (Rey et al 2014) of how we experience real life. 

Consider that contextually relevant data (whether measured in geospatial or temporal variables) is endlessly integrated into your mobile phone experience all day without a second thought to phones creating ‘virtual’ ‘cyber’ spaces.

Real and Virtual need not be mirrors of themselves. They instead form complex intra-relations: ‘nowadays’ the physical is infiltrated by computational surveillance of space and bodies in generative feedback loops that allow you to find a good restaurant from your car, feel reminders on your wrist while in a meeting, or automatically alert authorities to a car crash.

Generating novel relations through computational surveillance of space and bodies in generative feedback loops is what Augmented Reality does; AR relates computed environmental information back into physical space.

---

“AR relates computed environmental information back into physical space.”

---

It is not a layering these data, but data-based relations made knowable in new ‘real’ ways. Usually these relations are presented visually, and meant to be perceived by humans as part of reality. If presented aurally, we assume such media are very much part of reality – lest we consider ‘virtual’ the doppler of an approaching mechanical siren on the street, or for that matter the music in your ears from noise cancelling headphones, or other hearables that augment the perceived environment.

AR is Real

And here is where I have trouble with the utility of differentiating what makes newspaper media more real, or more part of the real world then AR media. AR media suffer a ‘virtual’ designation that has to ‘mix’ with ‘physical’ reality. Both newspaper and AR offer a medium and spit information onto it; AR’s information is just less picky or sticky. Through real-time computation, AR media indeed prove more spatially aware that they are part of reality than instantly out of date newsprint. 

More useful is considering the ways that AR mediates, which newsprint cannot. This leads us to better consider how AR’s speciality is building new and radical relations between computable knowledge and the physical world that in unison we perceive as our environment. 

---

“AR media offer radically relational
accounts of perceptible life“

---

A great AR example of this is Hu.ma.ne. Humane provides a handy (sorry) reminder of the power of focussing on how AR media offer radically relational accounts of perceptible life. Just like newsprint, Humane’s patents, prototypes and pitches ditch the technology of ‘displays’ to directly present symbols onto surfaces of the environment. I admit their work happens to spit up photons rather than ink, but I’m not sure why one (ink) would be any more ‘real’ than the other (photons). Both offer made up trickeries that we hallucinate into meaning something; speaking to a spectrum of real or virtual does not add much value in future.

Considering how Hu.ma.ne mediates relations of data-objects in the world shows the new relations available through AR’s mediation of environment with data; ephemeral and realtime – it offers a different reality to life than one dictated by newsprint and billboards. I would not describe it as virtual, however, or as layering information on the physical. It is mediating environmental features together (data and form) that we previously could not perceive. Their work considers the way AR can mediate that other media cannot.

I will end this post by stealing from Gibson’s classic consensual hallucination to help better project how AR is real. In how I propose we think about augmented reality, we might say:

“Augmented Reality Media. Consensual hallucinations experienced daily …[where] Lines of light are ranged in the space we perceive, opening the mind to clusters and constellations of data as environment”

“Cyberspace. A consensual hallucination experienced daily by billions of legitimate operators, in every nation…Lines of light ranged in the nonspace of the mind, clusters and constellations of data”

AR media are then better described as relations between computing-data and environment made perceptibly real. That’s AR. The physicality and virtuality of each bit/byte of AR media, or trying to consider how one layers atop the other, doesn’t really get at the most important points. Physical media and ‘non-physical’ media break down in AR space. These divides are subsumed with radically relational accounts of what is perceptible in our mediated reality.

But you don’t have to take my word for it. In 1999 as the first extended reality wave was forming around Virtual Reality hype, the scholar Nicola Green defined to “become virtual” as not merely having “access [to] a wholly ‘other’ space and becom[ing] digital” but rather, the processes of “making connections between programmed and nonprogrammed spaces in specific locales, and power-laden social, cultural, and economic relationships” (1999; 410-11). How fitting that Augmented Reality Media are now showing how that reality can be made perceivable out and about the lived world.

---

References

Barad, K. (2007). Meeting the universe halfway: Quantum physics and the entanglement of matter and meaning. Duke University Press.

Bleeker, M., Verhoeff, N., & Werning, S. (2020). Sensing data: Encountering data sonifications, materializations, and interactives as knowledge objects. Convergence, 26(5-6), 1088-1107.

Chevalier, C., & Kiefer, C. (2020). What does augmented reality mean as a medium of expression for computational artists?. Leonardo, 53(3), 263-267.

Couldry, N. (2012). Media, society, world: Social theory and digital media practice. Polity.

Jurgenson, N. (2011). Digital dualism versus augmented reality. The Society Pages, 24.

Gibson, William (1984). Neuromancer. New York: Ace Books.

Green, N. (1999). Disrupting the field: Virtual reality technologies and “multisited” ethnographic methods. American Behavioral Scientist, 43(3), 409-421.

Harari, Y. N. (2014). Sapiens: A brief history of humankind. Random House.

Heemsbergen, L., Bowtell, G., & Vincent, J. (2021). Conceptualising Augmented Reality: From virtual divides to mediated dynamics. Convergence, 27(3), 830-846.

Hine, C. (2020). Ethnography for the internet: Embedded, embodied and everyday. Routledge.

Hoffman, D. (2019). The case against reality: Why evolution hid the truth from our eyes. WW Norton & Company.

Milgram, P., Takemura, H., Utsumi, A., & Kishino, F. (1995, December). Augmented reality: A class of displays on the reality-virtuality continuum. In Telemanipulator and telepresence technologies (Vol. 2351, pp. 282-292). Spie.

Naimark, M. (1991, August). Elements of real-space imaging: a proposed taxonomy. In Stereoscopic displays and applications II (Vol. 1457, pp. 169-179). SPIE.Rey, P. J., & Boesel, W. E. (2014). The Web, digital prostheses, and augmented subjectivity. PJ Rey and Whitney Erin Boesel//Routledge handbook of science, technology, and society.–NY: Routledge, 173-188.

---

Recommended citation

Heemsbergen, L. (September, 2023) Pixel Theory. Critical Augmented and Virtual Reality Researchers Network (CAVRN).

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.