There are certain basics about designing anything. The product has to be safe. It has to be reliable. It should be affordable. But it also has to function for the user. More than giving a cool experience, the design should augment a person’s performance and skill. That is the focus of Cagri Zaman’s work.
As the director of the MIT Virtual Experience Design Lab, he looks to use immersive media to help people do everything from learning to play piano to learning how to handle heavy machinery. As a lecturer in the Department of Architecture, he looks to train students and scholars in using technology to craft stories. With both, the key is that the person is able to interact with the material, and because of that, the learning takes a stronger hold.
“Humans are spatially wired,” Zaman says. “We engage with information most efficiently when it’s presented in space.”
Making use of a keyboard
Zaman is exploring the potential of mixed reality to facilitate the transfer of expert knowledge to novices, specifically in the context of learning complex skills, such as piano playing. His team recorded an expert piano player’s hand movements and projected them onto a keyboard using mixed-reality technology, creating a virtual environment for students to practice alongside the master.
The virtual instructor provides explanations for each hand position and movement, enabling a deeper understanding of the techniques. It means that detailed information has to be embedded, an aspect that’s still being worked on, he says, but the initial results are promising, because just putting hands in space increases a person’s engagement and “makes them more excited about learning.” The other upside is that this technology means greater accessibility to learning, since physical proximity to an expert is no longer necessary.
Although piano learning serves as a fascinating and enjoyable medium, Zaman views this project as a “petri dish” for a much broader application. He envisions that mixed reality could revolutionize employee training, particularly with tasks that require heavy equipment, which can be expensive, and chemicals, which can be risky.
With this technology, experts can provide remote training, while employees can learn in simulated environments before transitioning to real-world situations. This immersive learning experience allows for more direct communication, “You’re talking to the brain more directly,” he says.
Creating a digital replica
Another project involves digital twinning, which creates a replica of a physical space and allows people to engage with it. For the construction industry, for example, a building project could be monitored, and problems could be identified and addressed in their early stages, Zaman says.
But another use would be to assist visually impaired people in better navigating the world. The digital replica could be of any place, such as their home, a market, or a train station. The technology would add information that’s particularly useful to them: the location of electrical outlets, the position of counters, the whereabouts of their medicine, or any potential hazards. Once in the space, the person can access and search for specific information through their smartphone.
The next step would be enabling real-time streaming of this information, where a remote supervisor could guide the person and communicate directly in the digital space. This setup could also easily apply to the business world. Zaman says that with so many companies having global teams, it’s not always feasible or possible for an expert to travel to guide, teach, or inspect. But with this technology, “you can make the expert teleport to the physical space,” he says.
However, the challenge lies in the technology. The 3D models need to be digitized, and large amounts of data need to be streamed in a fast and reliable way. The former is expensive, while the latter requires compression. Zaman says these issues are par for the course, but the field is “open and exciting,” and the potential for a person to be in an entire environment and also be able to choose a specific perspective or section offers significant upside. This ability certainly applies to a building project, but it could carry over to something like a sporting event. Whatever the setting, it means having a more dynamic experience
“You can have the complete image of what’s going on,” he says.
Stepping into history
Zaman is first and foremost a teacher, and one of his classes is in immersive media. The goal, he says, is to help students understand how to communicate their idea effectively — in essence, how to tell a good story. One project he has does this by reconstructing historical events depicted in films. It’s particularly helpful by giving designers, architects, film scholars, and historians what they need most, which is the ability to explore and analyze a physical space.
But creating immersive stories also would help with another facet of employee education, such as when workers need to interact with customers. Doing that on the job brings quick experience, but it also carries risk since mistakes are unavoidable. This technology allows them to role-play, with the trial-and-error done before it counts, and because people are actively performing, not merely watching a video, the learning is more resonant, he says.
At the core of his work, Zaman emphasizes the importance of understanding the user’s experience in any project, whether it’s building a home, car, or virtual environment. The design process itself should integrate storytelling, making it a central aspect of the creative journey. While a designer may come up with the initial concept, it can’t end there, he says. Users need to test and experiment with prototypes, as their experiences may differ from the original vision. This feedback loop allows the designer to create a more responsive, intuitive, and desirable product. Regardless of the technology involved, whether spatial sensing, computing, virtual reality, or the metaverse, the end result should enhance a person’s abilities and experiences, which is only possible with a strong story at its foundation.
“Everyone needs to learn design, regardless of what they do,” Zaman says. “It’s a way of thinking.”
