Pushing the Limits of Existence: Immersive Technology Sparks New Questions on Personhood

Assistant Professor Joshua A. Fisher on interactive design, immersive technology, and how posthumous personhood may come to define our legacies for the next generation.

Aaron Adler saw the world in pieces. From where he was stationed in England during World War Two, he witnessed firsthand what it was like to live amidst a population suffering from dangerously low rations. He saw what it meant for women not to be able to clothe their families, to wonder when and if their husbands might come back from the front, to live in desperate fear that the Axis bombers overhead might drop their payload on schools, or their homes. Adler saw these things, but knew he couldn’t write about them, or that if he did, censors would turn his words into a game of guesswork for his reader. But he also knew that he could write love letters to the woman he left behind in America. And so, he did. He wrote pages-long letters three or four times a week. The letters kept his love alive.

Decades later, Adler’s grandson, Interactive Arts and Media Assistant Professor Joshua A. Fisher, an expert in poetry and technology who had declared at the age of thirteen that he’d like to be a professor one day, got a devastating phone call while teaching abroad. His grandfather had died. The rich tome of Adler’s love letters, which had survived the war, had been left to Fisher. “I was like, ‘I have to do something with these letters, I have to do something with them,’” he remembers. “Not because it’s just a way for me to mourn, but because the language in them is so beautiful, and they’re written in a way that we don’t write anymore, and their visual aesthetic and materiality is something I wanted to share.”

A still from What We Mean

Above: A blackout poem alongside the original in What We Mean.

For Fisher, the project of building a format within which to hold one last conversation between himself and his grandparents and share the letters in their entirety led him to creating a mobile app titled What We Mean. In it, Fisher scanned in the over-500 pages written by his grandfather, and then, as he says, “I acted like a romantic war censor, and I blacked out parts of his text to make love poems out of his love letters…and then, in the app, you could go from the blackout poem to his letter.”

What We Mean proved to be so successful that it was the launchpad for a business for Fisher, running a mobile app development house out of 1871 at the Merchandise Mart. For four years after What We Mean was released, he worked on emerging and evolving technologies for new apps and received a patent for location-based media technology. For Fisher, it was also a professional growth period in which he learned how to program and take ideas from the germination to completion stage. In particular, he became interested in participatory design, and examining how such design methodologies can increase access and literacy in service of a more equitable future.

Teaching Immersive Environments and Serious Games and Simulations courses at Columbia is another way for Fisher to work in service of that future better world. As he says, “I get such joy out of seeing students ‘get it,’ and then seeing them make something from what they have learned.” His students have delivered: there is palpable joy when Fisher discusses a game his students created with the intent of educating people on neurodivergence and what it is like to be someone with ADHD. This is just one of many examples of the ways in which students have marshalled emerging technology under Fisher’s direction to solve real-world conundrums and to create simulations of various environmental and, in this case, psychological ecosystems.

There are challenges, of course. “If you want to work in the emerging media space, you have to be dedicated to a life of learning and re-learning…[I tell my students,] ‘You must think of emerging media as artistic interventions for doing research,’ Fisher says. After all, “We’re just at the Lumiere moment right now of coming up with grammars for storytelling.” The good news, Fisher will be the first to say, is that the ideas will always be there. “The ideas on how to design digital and computational media embedded in physical objects and spaces will always be growing. And those very human desires for storytelling and play will always be there. However, the technical parts will always be evolving.” It’s a reality that keeps the discipline fresh, challenges invigorating, and the future wide-open.

Fisher himself continues to work at the nexus of storytelling, artistic expression, and AR/VR participatory models for media production. Soon, he will be giving a paper at the HCI International 2021 Conference (HCII 2021) Conference where he will be talking about the ethical challenges of posthumous personhood as exemplified by mixed reality interactive archival representations of holocaust survivors presented at various museums around the world.

These mixed reality agents, for now relegated to the experience of well-funded institutions will likely permeate our world writ-large in the next decade, and Fisher is at the forefront of discussions of what it means to have a seemingly-thinking hologram living on after the person represented by that hologram passes away. “What does it mean that fifty years from now there’s an entity that’s claiming to be you that doesn’t necessarily think in the same way as you?” It’s a new question for a new world. Fisher is bold enough to ask it, brave enough to tackle the ethical challenges it poses, and innovative enough to engage with his students who are also at the forefront of how technology is changing what it means to live as a human and how it will define our personhood after we die to the people we leave behind.

Fisher’s new book, which examines mixed reality and posthumous personhood, Augmented and Mixed Reality for Communities, will be published by Routledge on May 15, 2021.


Rhiannon Koehler
Communications Manager