If storytelling was once confined to a single medium — a tale told, for instance, in the pages of a book — the past few decades have seen narratives spill into different platforms and media, spreading across everything from comic books to films to fan-created websites. Scholars such as Henry Jenkins called this “transmedia,” referring to the way a single fictional universe might expand into multiple formats in the digital age. Now, with the rise of virtual and augmented reality (VR and AR), the transformative possibilities of these all-enveloping story worlds has only grown.
The Transmedia Storytelling Initiative (TSI), a project led by Professor Caroline A. Jones, an art historian, critic, and curator in the History, Theory, Criticism section of the Department of Architecture, is dedicated to providing a hub where MIT students and faculty can explore the technical, aesthetic, philosophical, and ethical questions posed by transmedia — particularly with the advent of immersive virtual environments. Now in its third year, with a generous grant from David and Nina Fialkow, the initiative has produced courses and workshops for students, and is now hosting its first-ever exhibition at MIT — with a related symposium — that explores the new opportunities presented by the media of the future.
Telling new stories
From a rudimentary cell phone video to the most cutting-edge AR production, the exhibition, "Unbounded: Transmedia Storytelling @ MIT 2019-2021," features video loops documenting 3D and other projects, along with installations and immersive extended reality (XR) and projection works by over 40 MIT students.
In the exhibition, which took place in the physical gallery in October and November and remains on view online, MIT students use film, VR and AR, interactive web experience, publications, and art installations to tell multidimensional stories. The use of multiple media forms, Jones says, allows students to tell parts of stories, or tell similar stories in different ways — creating different entrance points into the work. Conceived by students in a range of MIT departments — from architecture to comparative media studies to urban planning to computer science — the work explores everything from road-tripping across America to the claustrophobia of lockdown.
In some pieces, students meditated on the nature of virtual existence and the ways in which we live and create alongside machines. Yaara Jacoby developed an eerie split-screen work depicting her “digital twin” in a small workspace during the pandemic lockdown. Vijay Gautham Rajkumar’s psychedelic abstract film, generated from AI-generated maps and scored to a melancholy jazz soundtrack, is projected from the gallery at night. Kwan “Queenie” Li created a VR installation and accompanying booklet about the connections between machine dreaming and the very human experience of insomnia.
Li also worked with Megan Prakash, Wu Li, and Kii Kang to create an AR piece about the 2020 Occupy City Hall protests, as part of the Transmedia-sponsored course Virtuality and Presence taught by Cagri Zaman, director of the School of Architecture and Planning Virtual Experience Design Lab, and Professor D. Fox Harrell, director of the Center for Advanced Virtuality.
For Li, the benefit of using augmented reality to portray something like the City Hall demonstrations is that it delivers the experience of civic protest into a viewer’s private space. “You can see, in your very familiar bedroom, protest signage that actual people hold in the streets, and this will also overcome a certain stasis of the archive,” she says. “How can we make an archive interactive? If we can make it more embodied, people can participate in it.”
Other works also used the possibilities of virtual production to illuminate social issues. Lauren Gideonse and Adriana Giorgis traveled together during the pandemic, producing a complex narrative about mapping, history, erasure, and what survives over time in the American landscape, while Melissa Teng and Lily Xie collaborated with community members in Boston’s Chinese-American community to make an interactive website that highlights the conversations that take place around the intimacy of the kitchen table.
Creating virtual community
The Transmedia Storytelling Institute provided funding and support for students to develop their projects. Over the course of the summer, student grantees met with faculty and reviewers, who offered constructive criticism on their projects as they were developing. Through these workshops, students from different disciplines were encouraged to experiment in modes outside their normal comfort zones.
“I'd say, ‘You're making a comic book, which is great, but I'm going to challenge you to make a video on your process. And you've never done that, but I think it's a useful skill for you to stretch into,'” says Jones. Especially during the isolation of the pandemic, these regular sessions offered an important virtual community for transmedia production. The overarching goal of the Transmedia Storytelling Initiative, Jones says, is to create a kind of vibrant ecosystem, a place where practitioners from many disciplines can collaborate and learn from one another.
Inside the metaverse
In addition to the exhibition, the symposium Boundary Conditions: Architecture, Simulation, Cinema explored the connections, past and present, between virtual production and architectural simulation. Since the beginning of cinematic history, the film medium has traded in facsimiles. While 19th century audiences were mesmerized by George Méliès’s fantastical short film “A Trip to the Moon,” contemporary viewers are similarly introduced to brand new forms of perception presented by metahumans, virtual reality, and deepfakes.
Bringing together filmmakers, scholars, scientists, and artists working in XR, the symposium investigated how these immersive story worlds blur the lines between documentary and fiction, the simulation and the real. In the panels, participants discussed how elements of narrative such as time, space, character — even the idea of “liveness” itself — might be radically redefined with emerging technologies.
Hacking transmedia for the 21st century
Today, Jones believes, transmedia storytelling has transcended its original definition. Whereas before the idea of transmedia was predominately tied to commercial industries — Jenkins used the example of the “Matrix” franchise as an example of this form of storytelling — Jones believes that these narrative modes can be creatively reappropriated to probe today’s pressing questions. “We're now at a point where [transmedia] means that you take the existing software, the existing platforms, and the existing games and you hack them,” she says.
As we delve further into the metaverse, the Transmedia Storytelling Initiative provides the crucial opportunity for MIT students, faculty, and researchers to think critically together about how and why we tell the stories we do. “For me, transmedia storytelling is an open question. It's a question about where is it going on, and how is the platform getting tweaked to tell a more authentic story?” Jones says. “The ‘trans’ for me is definitely a phrase that I want to mean ‘transformation.’”
