At a recent on-campus symposium titled “VR, Sound and Cinema: Implications for Storytelling and Learning,” MIT Open Learning explored the future of storytelling and learning through virtual reality (VR) and augmented reality (AR).
The event featured a panel of faculty and industry experts in VR/AR, cinema, and storytelling, showcasing the power of these tools and their potential impact on learning. Speakers included Sanjay Sarma, vice president for Open Learning; Fox Harrell, a professor of digital media and artificial intelligence at MIT; Academy Award-winning director Shekhar Kapur; Berklee College of Music Professor Susan Rogers; Academy Award-winning sound designer Mark Mangini; and Edgar Choueiri, a professor of applied physics at Princeton University.
Harrell, who is currently working on a new VR/AR project with MIT Open Learning, studies new forms of computational narrative, gaming, social media, and related digital media based in computer science. His talk focused on answering the question: “How do virtual realities impact our learning and engagement?” He also screened a preview of Karim Ben Khelifa’s “The Enemy,” a groundbreaking virtual reality experience that made its American premiere at the MIT Museum in December 2017.
In “The Enemy,” participants embody a soldier avatar, who encounters and interacts with enemy soldiers. Participants can ask their enemies questions, who can then adjust their responses based on the participants’ own lived experiences as well as their real-time physiological responses. The intended result is to create empathy between supposed enemies, whose hopes, dreams, and nightmares are more similar than their biases would have them believe.
“This can be a really powerful teaching tool,” Harrell said, explaining that it could be used in war zones and with child soldiers.
Next, film director and producer Shekhar Kapur spoke about storytelling in the age of infinite technological resources. Kapur pondered why people tend to watch the same movie over and over.
“We don’t always watch a movie again because it’s great, but because we can reflect upon ourselves and how we’ve changed even if the movie content hasn’t,” he said. In this sense, Kapur argued, stories have always been virtual, because they have always been filtered through each person’s subjective and shifting perspective.
“We are the stories we tell ourselves,” said Kapur, who believes that technology has always dictated the storytelling format. “If I don’t learn the new storytelling technologies, I’ll become a dinosaur.” Kapur insists that the three-act narrative dictated by past technologies will have to become more flexible, user-centric, and open-ended as VR becomes more commonplace. “We should be driven by the things we want. For example, I want to see my father again but he passed away several years ago. Can I retell his story with technologies that will make him seem real again? I don’t know.”
Finally, Susan Rogers, a professor of music production and engineering and an expert in music cognition at Boston’s Berklee College of Music, took the floor to talk about how technology is influencing our daily lives.
“Our behavior is becoming further from reality the more our technology imitates reality,” she said.
Rogers’ assessment focused on reality versus truth, examining what would happen to VR once it becomes so close to reality that it no longer seemed virtual.
“Scientists worship the truth — so how can scientists appreciate virtual reality?” she asked. “It isn’t truth.”
Following the panel, Professor Sarma invited guests to participate in a deeper dive into the day’s discussions. Academy Award-winning sound designer Mark Mangini and Edgar Choueiri, a professor of engineering physics at Princeton and director of the university's Electric Propulsion and Plasma Dynamics Laboratory (EPPDyL), led in-depth talks on how sound enhances learning and storytelling.
Mangini spoke of the need for sound designers to embrace artistry and narrative in their work.
“If we live in technique, we live on the boundaries of creativity,” he said. While technology has come a long way, he argued, there is still more to be done with 3-D.
“Our ancestors told stories around a fire,” he said. “Today, we still sit around in the dark watching a flickering light.”
Choueiri ended the event with a special interactive presentation, first asking aloud, “Why has spatial development been neglected for so long?” and then asserting that people’s emotional reactions are inherently spatial. To demonstrate the visceral nature of 3-D sound, Choueiri chose a volunteer and projected 3-D sound directly to him, by measuring and targeting his head-related transfer function (HRTF).
The sold-out event garnered an impressive level of interest from the public and students from MIT and Berklee College, who made up almost half of the audience. As VR/AR technology applications continue to grow, MIT Open Learning officials say they hope to hold more events that explore the intersection of science, media, and learning.
