Mixed Reality and 3D Printing, Imperfections to Perfection?

Dec 16, 2019

Want to write a piece for 3DHEALS Expert Corner? Email us: info@3dheals.com

Recently, we briefly reviewed a few publications examining the clinical values of mixed reality and 3D printing in our “Academia” section of the 3DHEALS blogs, and in particular, if combining the two technologies shows promises for a better overall product for pre- and intra-operative planning. This blog will dive deeper and hope to synthesize an updated overview of where the things are, shed light on future trends in a no-nonsense fashion, and perhaps provide an entry point for early adopters to these technologies.

Virtual reality, Augmented reality, Mixed reality

First of all, it is important to clarify some terms.

At a high level, virtual reality (VR) implies a complete immersion experience that shuts out the physical world. Augmented reality (AR) adds digital elements to a live view often by using the camera on a smartphone, for example, augmented reality experiences using Snapchat lenses and the game Pokemon Go. Mixed reality not only puts digital objects in a user’s environment but also allow the users to interact with both the digital objects and his/her physical environment, including anatomy models during surgery. The Microsoft website provided a slightly more in-depth explanation of the concepts: [1]

“Mixed reality is the next evolution in human, computer, and environment interaction and unlocks possibilities that before now were restricted to our imaginations. It is made possible by advancements in computer vision, graphical processing power, display technology, and input systems. The term mixed reality was originally introduced in a 1994 paper by Paul Milgram and Fumio Kishino, “A Taxonomy of Mixed Reality Visual Displays.” Their paper introduced the concept of the virtuality continuum and focused on how the categorization of taxonomy applied to displays. Since then, the application of mixed reality goes beyond displays. It also includes environmental input, spatial sound, and location.”

The following demonstration further explains the concept that “mixed reality” is a spectrum rather than a single defined point: [1]

“Towards the left (near physical reality). Users remain present in their physical environment and are never made to believe they have left that environment.
In the middle (fully mixed reality). These experiences blend the real world and the digital world. Viewers who have seen the movie Jumanji can reconcile how the physical structure of the house where the story took place was blended with a jungle environment.
Towards the right (near digital reality). Users experience a completely digital environment and are unaware of what occurs in the physical environment around them.”

What’s intriguing is that a 3D-printed model is entirely in the physical world yet representing 100% digital data (from the virtual world).

Mixed Reality versus 3D-Printing

The reason why 3D-printing and mixed reality are often evaluated side by side is that both are increasingly incorporated in advanced visualization workflow. For example, in a recent study comparing the technologies for nephron preservation surgery for Wilm’s tumor in pediatric patients [3], the authors presented the following workflow, demonstrating the close digital footprints of both procedures [Figure 1]. Both technologies can be used for surgical planning for a variety of specialties [2-7, 9-10], and both are often constrained by existing imaging processing capabilities including [5]:

Imaging data acquisition/Resolution
Segmentation
Rendering
Software analysis

Photo Credit: Lianne M Wellens, et al [3]

Both technologies can produce models that are reproducible and provide a “shared” experience both before and during surgeries.

However, there are also unique value propositions from both technologies.

For mixed reality, the construction process of a final product is often less costly, and the user can manipulate the objects with more degrees of freedom (more adaptive). For example, the user can zoom in and out, or virtually “dissect” the model to see internal vascular structures, etc. The user can further superimpose other information onto the existing model, including functional data such as 3D tractography and finite element analysis. However, registration of the virtual model to the physical world remains a unique problem for mixed reality. Tissue deformation and tactile information from the real world are also lost for current MR technology. Additionally, significant technological hurdles include data transmission latency on current wireless networks, chunky and expensive headsets, additional hardware requirements including graphic cards, etc. 5G service is still in its infancy, particularly in the U.S. [8]

Significant increase in publications focusing on 3D printing, Augmented reality, and Mixed reality as keywords

A unique value of the 3D printed model is that it can directly interact with the physical world. For example, surgeons can use a mandibular bone to pre-bend a metal plate before surgery, but not with mixed-reality models. Another example is a 3D printed surgical guide, which can directly interact with the patient’s anatomy and the surgeon. Once a model is printed, it can be shared without the need for any additional equipment or network, since it is now entirely physical.

Mixed Reality and 3D Printing

Several recent studies have explored if these currently still imperfect technologies can be complementary to each other, indicating a trend towards a more technology-agnostic approach in problem-solving. For example, Rafael Moreta-Martinez, et al [6] proposed a workflow to allow automatic registration of the real world and mixed reality data using a 3D-printed patient-specific registration instrument. Similarly, a few months later, Peng-Fei Lei, et al [2] designed a 3D-printed patient-specific registration instrument, complimenting 3D reconstructed virtual surgical planning and mixed reality intraoperative models in a complicated case hip arthroplasty case. The combination of a 3D printed surgical guide serving as a mixed reality automatic registration tool seems promising from both clinical and cost perspectives. It would be interesting to see more clinical applications leveraging both technologies simultaneously.

Surgical Guide with AR Registration Instrument (Photo Credit: Rafael Moreta-Martinez, et al [6])

Conclusion:

Advanced visualization in healthcare has lately taken on new meanings with the advancement of several emerging technologies such as 3D printing and mixed reality. The progressive understanding of human-computer interaction is now elevating the human experiences in the operating theatres beyond visual experiences. That said, significant real technological hurdles exist for both MR and 3D printing. Additionally, the lack of outcome studies and quality control standards also limit wide-spread adoption.

Nonetheless, the optimists will say that few technological breakthroughs are accidental. The progress we are seeing in advanced visualization tools in healthcare today will not occur without past decades of innovation and improvements in imaging acquisition (i.e. CT, MR, etc.), computing power (cloud, etc), and digital manufacturing (e.g. 3D printing, digital milling, etc.). While most of these innovations appear only incremental improvements and often slower than our expectations, we may very well be on the cusp of the next quantum leap when these existing imperfect tools reach the perfect alignment.