Want to write a piece for 3DHEALS Expert Corner? Email us: email@example.com
From 3D Printing to VR/AR: Simple Connection?
The same, but different:
My journey in 3D imaging began with 3D Printing as I searched for a way to show the visually complex Diffusion Tensor Imaging data obtained from my research in pediatric kidney transplants. As I began to explore technologies of Augmented Reality (AR) and Virtual Reality (VR) through my work at UCSF and HoloSurg3D, I found that while these technologies have many similarities with 3D printing, there were certain considerations that needed to be made when crossing between these mediums. In this post, I’ll describe those areas and provide some considerations when optimizing images for AR/VR.
What you don’t need to consider:
There are physical constraints that must be accounted for when preparing a file for 3D printing:
- – The model mesh must be watertight, i.e. no holes or gaps can be present in the model.
- – Edges and thickness must be considered within the limitations of the printer and the material is chosen.
- – Need for struts/supports must be factored in when printing the model as well as the model position.
- – Cost including materials needed, colors, and size.
In the world of Augmented and Virtual Reality, none of the items listed above need to be considered. As AR/VR models are gravity-defying and essentially suspended in air, the constraints of physical materials do not exist. As the character Morpheus in “The Matrix” describes the virtual world: “Some [rules] can be bent, others–can be broken”!
Models that would be very difficult to print and display as physical models at actual scale due to size, fragility, or cost can be beautifully displayed in AR/VR settings.
Screen capture of .obj file of Whole Brain Diffusion Tensor Imaging
Whole Brain Diffusion Tensor Imaging displayed in Augmented Reality (click for video)
Multi-part cardiac model displayed in Augmented Reality (click for video)
An anatomically full-scale model can be created that can be peeled through without the need for cutting or other post-processing as would a physical model.
Full-scale CT-derived model from lower pelvis to tibia displayed in Augmented Reality with angle measurement being used for assessment of the degree of femoral varus bowing (click for video).
Further, a model can be enlarged to the size of the room allowing a user to interact with structures in ways otherwise not possible. As there is no cost for modeling in AR/VR aside from the initial hardware purchase, multiple iterations can be made without concern for material use or cost. Using mobile-based AR solutions, one can share cases with colleagues thousands of miles away only requiring a smartphone to interact with these models. 4D/animation can also be incorporated. Further, using complementary objects such as the Merge Cube, iOS users can have the ability to manipulate objects in the same intuitive way as holding a physical model.
Animated cardiac model derived from a multiphase CT, displayed via iOs using the Merge Cube (click for video).
CT-derived cardiac model displayed via iOs using the Merge Cube (click for video).
What you do need to consider:
There are considerations unique to both Augmented Reality and Virtual Reality. Virtual Reality considerations include the type of hardware to be used including the specific capabilities of the devices, ranging from phone-based to head-mounted devices and the computer to which the device is tethered. The display background must be created. Display frame rates must be considered as well. Mesh models intended for 3D printing can be used and displayed for VR. The 3D files can be of a larger size in comparison to stand-alone AR devices, since VR devices may be tethered to a separate computer.
Augmented Reality, which is my true area of expertise, has unique considerations that I have learned over the course of developing my initial RadHA (Radiology with Holographic Augmentation) app for the HoloLens and iOS and creating numerous models. Models must be optimally sized to account for the processing capabilities of the hardware device, either HoloLens, iPad, or iPhone. This may require decimation of file size by 50% or greater. Once this is done, appropriate smoothing algorithms must be applied to remove the rougher/pixelated look, while preserving the level of detail needed to appreciate anatomy and pathology.
Model without (right) and with (left) smoothing and model optimization.
This includes avoidance of smoothing over details such as hairline/non-displaced fractures. Furthermore, since AR is displayed on the real-world background, color shades, brightness, and degree of translucency should be considered to make the model robust for varying background lighting conditions.
At HoloSurg3D, through our extensive work specializing in AR-specific medical modeling, we have optimized these parameters through our proprietary algorithms, taking all of these factors into consideration.
In summary, while there are many similarities between model creation for 3D printing and AR/VR applications, there are unique considerations that must be made. I view AR/VR technologies as complementary to 3D printing, allowing for rapid prototyping, intra-operative guidance, and broadening the overall reach and scalability of 3D imaging. I am truly excited about the potential for these 3D imaging technologies to transform the way we practice medicine and am proud to be part of a team working on innovation in this burgeoning new field!
About the Author:
Jesse Courtier, MD, is an Associate Clinical Professor in Pediatric Radiology at the University of California, San Francisco. He is responsible for the interpretation of pediatric imaging using multiple modalities including CT, MRI, ultrasound, plain film and fluoroscopy for the thoracic, abdominal/pelvic, and musculoskeletal systems in Pediatric patients. Dr. Courtier obtained his medical degree from the University of Iowa College of Medicine, Iowa City in 2003, and completed his residency in Diagnostic Radiology from the University of Kansas, Wichita in 2008, followed by a fellowship in Abdominal Imaging and Pediatric Radiology from UCSF.
Dr. Courtier is involved in medical education, serving on the Resident Education Committee as a pediatric radiology representative as well as the Resident Selection Committee and the newly formed Resident Clinical Competency Committee. Also, he serves in departmental committees involving several aspects of the UCSF radiology and biomedical imaging section. He has published 16 peer-review journal articles as well as several abstracts and educational exhibits. Dr. Courtier has published peer-reviewed articles.