Mixed Reality and 3D Printing, Imperfections to Perfection?

Want to write a piece for 3DHEALS Expert Corner? Email us: info@3dheals.com

Recently, we briefly reviewed a few publications examining the clinical values of mixed reality and 3D printing in our “Academia” section of the 3DHEALS blogs, and in particular, if combining the two technologies shows promises for a better overall product for pre- and intra-operative planning. This blog will dive deeper and hope to synthesize an updated overview of where the things are, shed light on future trends in a no-nonsense fashion, and perhaps provide an entry point for early adopters to these technologies.  

Virtual reality, Augmented reality, Mixed reality

First of all, it is important to clarify some terms.

At a high level, virtual reality (VR) implies a complete immersion experience that shuts out the physical world. Augmented reality (AR) adds digital elements to a live view often by using the camera on a smartphone, for example, augmented reality experiences using Snapchat lenses and the game Pokemon Go. Mixed reality not only puts digital objects in a user’s environment but also allow the users to interact with both the digital objects and his/her physical environment, including anatomy models during surgery. The Microsoft website provided a slightly more in-depth explanation of the concepts: [1]

“Mixed reality is the next evolution in human, computer, and environment interaction and unlocks possibilities that before now were restricted to our imaginations. It is made possible by advancements in computer vision, graphical processing power, display technology, and input systems. The term mixed reality was originally introduced in a 1994 paper by Paul Milgram and Fumio Kishino, “A Taxonomy of Mixed Reality Visual Displays.” Their paper introduced the concept of the virtuality continuum and focused on how the categorization of taxonomy applied to displays. Since then, the application of mixed reality goes beyond displays. It also includes environmental input, spatial sound, and location.”

The following demonstration further explains the concept that “mixed reality” is a spectrum rather than a single defined point: [1]

blank
Photo Credit: Microsoft [1]

  • “Towards the left (near physical reality). Users remain present in their physical environment and are never made to believe they have left that environment.
  • In the middle (fully mixed reality). These experiences blend the real world and the digital world. Viewers who have seen the movie Jumanji can reconcile how the physical structure of the house where the story took place was blended with a jungle environment.
  • Towards the right (near digital reality). Users experience a completely digital environment and are unaware of what occurs in the physical environment around them.”

What’s intriguing is that a 3D-printed model is entirely in the physical world yet representing 100% digital data (from the virtual world).

Mixed Reality versus 3D-Printing

The reason why 3D-printing and mixed reality are often evaluated side by side is that both are increasingly incorporated in advanced visualization workflow. For example, in a recent study comparing the technologies for nephron preservation surgery for Wilm’s tumor in pediatric patients [3], the authors presented the following workflow, demonstrating the close digital footprints of both procedures [Figure 1]. Both technologies can be used for surgical planning for a variety of specialties [2-7, 9-10], and both are often constrained by existing imaging processing capabilities including [5]:

  • Imaging data acquisition/Resolution
  • Segmentation
  • Rendering
  • Software analysis

blank
Photo Credit: Lianne M Wellens, et al [3]

Both technologies can produce models that are reproducible and provide a “shared” experience both before and during surgeries.

However, there are also unique value propositions from both technologies.

For mixed reality, the construction process of a final product is often less costly, and the user can manipulate the objects with more degrees of freedom (more adaptive). For example, the user can zoom in and out, or virtually “dissect” the model to see internal vascular structures, etc. The user can further superimpose other information onto the existing model, including functional data such as 3D tractography and finite element analysis. However, registration of the virtual model to the physical world remains a unique problem for mixed reality. Tissue deformation and tactile information from the real world are also lost for current MR technology. Additionally, significant technological hurdles include data transmission latency on current wireless networks, chunky and expensive headsets, additional hardware requirements including graphic cards, etc. 5G service is still in its infancy, particularly in the U.S. [8]

blank
Significant increase in publications focusing on 3D printing, Augmented reality, and Mixed reality as keywords

A unique value of the 3D printed model is that it can directly interact with the physical world. For example, surgeons can use a mandibular bone to pre-bend a metal plate before surgery, but not with mixed-reality models. Another example is a 3D printed surgical guide, which can directly interact with the patient’s anatomy and the surgeon. Once a model is printed, it can be shared without the need for any additional equipment or network, since it is now entirely physical.

Mixed Reality and 3D Printing

Several recent studies have explored if these currently still imperfect technologies can be complementary to each other, indicating a trend towards a more technology-agnostic approach in problem-solving. For example, Rafael Moreta-Martinez, et al [6] proposed a workflow to allow automatic registration of the real world and mixed reality data using a 3D-printed patient-specific registration instrument. Similarly, a few months later, Peng-Fei Lei, et al [2] designed a 3D-printed patient-specific registration instrument, complimenting 3D reconstructed virtual surgical planning and mixed reality intraoperative models in a complicated case hip arthroplasty case. The combination of a 3D printed surgical guide serving as a mixed reality automatic registration tool seems promising from both clinical and cost perspectives. It would be interesting to see more clinical applications leveraging both technologies simultaneously.

blank
Surgical Guide with AR Registration Instrument (Photo Credit: Rafael Moreta-Martinez, et al [6])

Conclusion:

Advanced visualization in healthcare has lately taken on new meanings with the advancement of several emerging technologies such as 3D printing and mixed reality. The progressive understanding of human-computer interaction is now elevating the human experiences in the operating theatres beyond visual experiences. That said, significant real technological hurdles exist for both MR and 3D printing. Additionally, the lack of outcome studies and quality control standards also limit wide-spread adoption.

Nonetheless, the optimists will say that few technological breakthroughs are accidental. The progress we are seeing in advanced visualization tools in healthcare today will not occur without past decades of innovation and improvements in imaging acquisition (i.e. CT, MR, etc.), computing power (cloud, etc), and digital manufacturing (e.g. 3D printing, digital milling, etc.). While most of these innovations appear only incremental improvements and often slower than our expectations, we may very well be on the cusp of the next quantum leap when these existing imperfect tools reach the perfect alignment.

References:

1. https://docs.microsoft.com/en-us/windows/mixed-reality/mixed-reality

2. Mixed Reality Combined With Three-Dimensional Printing Technology in Total Hip Arthroplasty: An Updated Review With a Preliminary Case Presentation (Peng-Fei Lei, et al) Orthop Surg, 11 (5), 914-920 Oct 2019

3. Comparison of 3-Dimensional and Augmented Reality Kidney Models With Conventional Imaging Data in the Preoperative Assessment of Children With Wilms Tumors (Lianne M Wellens, et al) JAMA Network Open, 2 (4), e192633 2019 Apr 5 PMID: 31002326

PMCID: PMC6481457 DOI: 10.1001/jamanetworkopen.2019.2633

4. Visualization Improves Supraclavicular Access to the Subclavian Vein in a Mixed Reality Simulator (Joshua Warren Sappenfield, et al) Anesthesia & Analgesia. 127(1):83–89, JULY 2018 PMID: 29200069 PMCID: PMC6774241DOI: 10.1213/ANE.0000000000002572

5. Neurosurgical Virtual Reality Simulation for Brain Tumor Using High-definition Computer Graphics: A Review of the Literature (Taichi Kin, et al) Neurol Med Chir (Tokyo), 57 (10), 513-520  2017 Oct 15 PMID: 28637947 PMCID: PMC5638778 DOI: 10.2176/nmc.ra.2016-0320

6. Augmented Reality in Computer-Assisted Interventions Based on Patient-Specific 3D Printed Reference (Rafael Moreta-Martinez, et al) Healthc Technol Lett, 5 (5), 162-166 2018 Sep 14 eCollection Oct 2018 PMID: 30464847 PMCID: PMC6222179 DOI: 10.1049/htl.2018.5072

7. Augmented Reality, Surgical Navigation, and 3D Printing for Transcanal Endoscopic Approach to the Petrous Apex (Samuel R Barber, et al) OTO Open, 2 (4), 2473974X18804492 2018 Oct 29 eCollection Oct-Dec 2018

PMID: 30719506 PMCID: PMC6348519 DOI: 10.1177/2473974X18804492

8. Virtual travel could change the world- if it gets off the ground.  Sara toth stub, dec. 12, 2019 10:00 am, Wall Street Journal

9. A Review of Simulation Applications in Temporal Bone Surgery (Tanisha S Kashikar, et al) Laryngoscope Investig Otolaryngol, 4 (4), 420-424

10. Augmented Reality and Three-Dimensional Printing in Percutaneous Interventions on Pulmonary Arteries (Jan Witowski, et al) Quant Imaging Med Surg, 9 (1), 23-29 Jan 2019 PMID: 30788243 PMCID: PMC6351817 DOI: 10.21037/qims.2018.09.08

Related Articles:

From 3D Printing to VR/AR: Simple Connection?

From Academia: 3D Printed Pills, 4D Printed Structure, and how to property 3D print Chocolate

From Academia: 3D-Printed Aligner, Bioprinting for Mouth Ulcer, Vertebroplasty Guides

From Academia: 3D Printing and Robotics, Stem cell coated Implants, Decentralized Mitigation of Pandemics

Comments