UBC and Microsoft collaboration yields floating, interactive holographic brain

It might look like something straight out of a sci-fi movie, but the Holographic Brain Project is the real deal.

UBC researchers Dr. Tamara Bodnar, Dr. Claudia Krebs and Parker Holman worked together with interns at Microsoft Garage to develop the Holographic Brain Project, which was designed as a neuroanatomy learning tool to help teach a wide range of students about the brain.

This includes students from the life sciences, psychology and medicine. Bodnar and Holman are both supported by the Mitacs Accelerate Internship.

The Holographic Brain Project is an application that is run on Microsoft Hololens, an augmented reality headset that looks like a large pair of sunglasses and contains a holographic computer. After the user puts on the headset, technology inside the Hololens projects an interactive 3D brain into the user’s physical environment with brain structures that can be enlarged, isolated or rotated, among many other commands.

Currently, the app contains a set of brain structures called the basal ganglia.

Hololens allows for a unique look into the brain that is hard to obtain with other techniques like dissections. Holman described how more superficial or exterior structures are often damaged in order to expose inner structures, which is not the case with the holographic technology.

“The brain is particularly difficult to teach,” said Holman. “It’s a really compact structure. It’s also very complex. Unlike when we do dissections in the gross anatomy lab — [where] we can maintain the overall context of the body and the different structures — when we dissect the human brain, it’s a really destructive process and we lose a lot of the context.”

How the Holograhic Brain works

Unlike virtual reality, where a wholly virtual world is created, augmented reality incorporates computer-generated input into the actual environment around the user.

Sensors on the headset relay information from the environment into the Hololens, which then creates a spatial map of the surrounding environment and projects a holographic image onto it. Although the application runs on Microsoft Hololens, Microsoft Garage interns coded the programming logic such as voice and gesture controls on Unity, an engine used primarily for game development.

A unique characteristic of the app’s 3D brain is that it is based off of a real person, rather than a conceptual or abstract approximation like many other brain models.

The 3D model in the app was initially traced from a magnetic resonance imaging (MRI) scan, which is an image that looks like a flat brain slice. Building on that MRI data, the team built the 3D brain structures to overlap with corresponding 2D MRI brain structures.

One of the key strengths of the hologram app is exactly that: its ability to interface a 3D brain onto a 2D MRI scan simultaneously in the real world. Other technologies can interface 2D brain photos and 3D models together on-screen, but the app’s ability to marry both in an actual hologram allows students to more easily form a representation of 3D brain structures, and understand the relative positions of different brain structures.

The researchers compared using the Hololens to commanding your dog: it responds promptly to both predefined voice commands and hand gestures, which are its inputs.

“Rotate” will cause the 3D brain to rotate on the spot, allowing you to view structures from almost every angle. “Isolate” is useful for singling out desired structures. “Expand” blows up brain structures for easy viewing, while touching your thumb to your index finger can select specific brain areas.

Into the classroom

“Students are trying to build in their mind this 3D representation of the brain,” Holman said, referring to the process of turning a 2D brain image into a 3D understanding. “And that constant back and forth is a really high cognitive load. It uses a lot of brainpower to build that image.”

Bodnar explained the practical benefits of interfacing 2D MRIs with 3D structures, specifically for medical students. “Realistically, how medical students and doctors [are] really going to be looking at the brain is an MRI,” Bodnar said. “By sticking the two [together], they can really understand the 3D structure, but then link back to how they’re going to be examining the brain in real life.”

The intended lesson plan for the Hololens is to mimic a real lab experience, with an instructor leading the session and students following along.

Since the Hololens is a shared experience, both instructors and students, who will all wear the Hololens, can see the holographic brain in real time.

[Sorry, video not found. You can contact webmaster@ubyssey.ca to fix the issue]

After opening the session and providing a class ID, the instructor will then manipulate aspects of the hologram through voice and gesture commands, such as enlarging specific brain structures. The adjustments from the instructor’s Hololens will then be sent to all students in the room.

The researchers clarified that the app is meant to complement rather than replace physical in-lab experiences like dissections. The value of playing with brain tissue and having that tactile feedback, they said, cannot be substituted by holograms.

The hands behind the brain app

Mehrdad Ghomi, a software engineer at UBC currently working on the project, described the app’s user-friendliness, which caters well to students and instructors. “If you’re comfortable with voice commands ... your hands are free, you can interact, you can show the next person, say ‘Look at this please,’ and then say a command and the system will respond to it.”

He added that voice command recognition generated from Microsoft Cortana data allows the application to recognize various accents, a practical attribute for teaching at a school as diverse as UBC.

Kevin Jacyna, a Microsoft Garage intern and UBC student, worked on the project for four months from beginning to completion as a developer on a team of six people, with five developers and one designer. The interns were pitched different ideas from Microsoft, one of them being the Hololens project proposed by Krebs.

“We started from basically ground zero, where none of us had done any kind of AR [augmented reality] development or had worked with Unity,” Jacyna said. He added that Krebs provided the interns with a neuroanatomy lecture prior to starting the project to help them identify difficulties students might have in learning the material.

Using the Agile Method, the team of interns worked on weekly tasks and made weekly improvements based on feedback from UBC and students.

“Basically we worked right until the last week of continuously building our application,” Jacyna said. “All the way up from week one, where it was just some cubes that we were figuring out how to move around, to the end brain model and all the core features that we wanted to have finished.”

As for future plans, the research team and developers have many.

A more immediate research project involves recruiting students to assess the educational value of the app, and to gauge its teaching effectiveness in the classroom. The team has also proposed the Holographic Brain Project as a capstone design project, an upper-level intensive project similar to an honours thesis, to UBC’s undergraduate electrical and computer engineering department.

Further tasks include possibly implementing a gaze-tracking system, reducing potential rendering time lags as the app is developed, and incorporating a feature where the user will be able to see the of effect various manipulations on regional brain cell movements.

First online Oct. 2, 2017, 11:33 p.m.