Medical Simulation Using Augmented Reality, Virtual Reality, 3D Printing

Category: Blog,From Academia
blank blank Feb 06, 2021

The medical 3D imaging data can be used for a variety of 3D technologies ranging from 3D Printing to lately, virtual reality and augmented reality. In this issue, we feature four publications investigating various applications and clinical outcomes of using medical VR/AR for medical simulation. The first article connects 3D printing, AR/VR to the management of kidney and prostate cancer. The second article focuses on diagnostic accuracies in congenital heart disease among conventional visualization, immersive, and non-immersive virtual reality. The winner of this small sample study was immersive virtual reality, especially when the case was complex. However, as we dive deeper into the VR/AR space, the human-computer interaction does not limit to simple visualization. The third article explores two existing haptic VR interfaces, a vibrotactile VR interface, and a kinesthetic VR interface, for medical diagnosis and planning on volumetric medical images. The final publication tackles the VR interface further by creating a prototype with suturing simulation in a VR laparoscopic surgery simulator with haptic force.

From Academia” features recent, relevant, close to commercialization academic publications. Subjects include but not limited to healthcare 3D printing, 3D bioprinting, and related emerging technologies.

Email: Rance Tino (tino.rance@gmail.com) if you want to share relevant academic publications with us.

3D Printing, Augmented Reality, and Virtual Reality for the Assessment and Management of Kidney and Prostate Cancer: A Systematic Review

Authored by Nicole Wake, Jeffrey E. Nussbaum, Marie I. Elias, Christine V. Nikas, Marc A. Bjurlin. Urology. September 2020

Figure 1. Examples of 3D printing (top), AR (middle), and VR (bottom) technologies. Top row: Left: Stratasys J750 Digital Anatomy Printer (image courtesy of Stratasys, Eden Prairie, MN), Middle: 3D printed kidney tumor model with the kidney in clear, collecting system semi-transparent, lesion in purple, renal artery in pink, renal vein in light blue, and collecting system in dark blue, Right: 3D printed prostate cancer model with the prostate clear, lesion – blue, neurovascular bundles – yellow, rectal wall – white, bladder neck and urethra – pink. Middle row: Left: HoloLens (Microsoft, Redmond, WA) AR headset, Middle: AR kidney tumor model shown projected in a room with the kidney – pink, tumor – gray, artery –red, vein –blue, collecting system –yellow, Right: AR prostate cancer model shown projected in a room with the prostate transparent, lesions – blue, neurovascular bundles – purple, bladder neck and collecting system – yellow. Bottom row: Left: Person wearing HTC Vive VR headset, Middle: Photograph of VR kidney tumor model overlaid onto CR images, Right: VR prostate cancer model with the same color scheme as above with the arterial supply-red. Copyright Urology
Figure 1. Examples of 3D printing (top), AR (middle), and VR (bottom) technologies. Top row: Left: Stratasys J750 Digital Anatomy Printer (image courtesy of Stratasys, Eden Prairie, MN), Middle: 3D printed kidney tumor model with the kidney in clear, collecting system semi-transparent, lesion in purple, a renal artery in pink, renal vein in light blue, and collecting system in dark blue, Right: 3D printed prostate cancer model with the prostate clear, lesion – blue, neurovascular bundles – yellow, rectal wall – white, bladder neck and urethra – pink. Middle row: Left: HoloLens (Microsoft, Redmond, WA) AR headset, Middle: AR kidney tumor model shown projected in a room with the kidney – pink, tumor – gray, artery –red, vein –blue, collecting system –yellow, Right: AR prostate cancer model shown projected in a room with the prostate transparent, lesions – blue, neurovascular bundles – purple, bladder neck and collecting system – yellow. Bottom row: Left: Person wearing HTC Vive VR headset, Middle: Photograph of VR kidney tumor model overlaid onto CR images, Right: VR prostate cancer model with the same color scheme as above with the arterial supply-red. Copyright Urology

Abstract: 

Three-dimensional (3D) printing, augmented reality, and virtual reality technologies have an increasing presence in the management of prostate and kidney cancer. To assess the utility of 3D printing augmented reality, and virtual reality for (1) quantitative outcomes, (2) surgical planning, (3) intraoperative guidance, (4) training and simulation, and (5) patient education for patients with kidney and prostate cancer a systematic literature review was performed. Existing evidence demonstrates improvement in clinical outcomes, surgical planning, and intra-operative guidance, as well as training. Future studies are needed to assess the impact of 3D technologies on long-term patient-related outcomes.

A Novel Virtual Reality Medical Image Display System for Group Discussions of Congenital Heart Disease: Development and Usability Testing

Authored by Byeol Kim, Yue-Hin Loke, Paige Mass, Matthew R Irwin, Conrad Capeland, Laura Olivieri, Axel Krieger, JMIR Publications. 31 May 2020

Medical images of the congenital heart disease cases: 2D (top) and 3D (bottom). Arrows represent the anatomical regions to scrutinize for correct diagnosis. ASD: atrial septal defect; CoA: coarctation of aorta; MAPCA: major aortopulmonary collateral artery. Copyright JMIR Publications
Medical images of the congenital heart disease cases: 2D (top) and 3D (bottom). Arrows represent the anatomical regions to scrutinize for correct diagnosis. ASD: atrial septal defect; CoA: coarctation of aorta; MAPCA: major aortopulmonary collateral artery. Copyright JMIR Publications 

Abstract: 

The objective of the study was to evaluate and compare the diagnostic accuracies and preferences of various display systems, including the conventional 2D display and a novel group VR software, in group discussions of CHD.

A total of 22 medical trainees consisting of 1 first-year, 10 second-year, 4 third-year, and 1 fourth-year resident and 6 medical students, who volunteered for the study, were formed into groups of 4 to 5 participants. Each group discussed three diagnostic cases of CHD with varying structural complexity using conventional 2D display and group VR software.

A group VR software, Cardiac Review 3D, was developed by our team using the Unity engine. By using different display hardware, VR was classified into non-immersive and full-immersive settings. The discussion time, diagnostic accuracy score, and peer assessment were collected to capture the group and individual diagnostic performances. The diagnostic accuracies for each participant were scored by two experienced cardiologists following a predetermined answer rubric. At the end of the study, all participants were provided a survey to rank their preferences of the display systems for performing group medical discussions.

Diagnostic accuracies were highest when groups used the full-immersive VR compared with the conventional and non-immersive VR (χ22=9.0, P=.01) displays. Differences between the display systems were more prominent with increasing case complexity (χ22=14.1, P<.001) where full-immersive VR had accuracy scores that were 54.49% and 146.82% higher than conventional and non-immersive VR, respectively.

The diagnostic accuracies provided by the two cardiologists for each participant did not statistically differ from each other (t=–1.01, P=.31). The full-immersive VR was ranked as the most preferred display for performing group CHD discussions by 68% of the participants. The most preferred display system among medical trainees for visualizing medical images during group diagnostic discussions is full-immersive VR, with a trend toward improved diagnostic accuracy in complex anatomical abnormalities. Immersion is a crucial feature of displays of medical images for diagnostic accuracy in collaborative discussions.

Diagram of group discussion format for each display system (from the top: conventional, nonimmersive virtual reality, and full-immersive virtual reality displays). VR: virtual reality. Copyright JMIR Publications
Diagram of group discussion format for each display system (from the top: conventional, nonimmersive virtual reality, and full-immersive virtual reality displays). VR: virtual reality. Copyright JMIR Publications 

Evaluation of haptic virtual reality user interfaces for medical marking on 3D models

Authored by Zhenxing Li, Maria Kiiveri, Jussi Rantala, Roope Raisamo. International Journal of Human-Computer Studies. 25 March 2021

(A) The vibrotactile VR interface using a Vive controller and a Vive VR headset. A close-up image of the trigger button of the controller is shown at the bottom left corner. The additional Samsung LCD 2D display (not visible to the user) shows the brain model. (B) The kinesthetic VR interface using a Geomagic Touch X force-feedback device and a Vive VR headset. A close-up image of the device button is shown at the bottom left corner. The additional 2D display shows the hipbone model. (C) The traditional 2D interface using a standard mouse with the Samsung LCD 2D display. A close-up image of the mouse buttons is shown at the bottom left corner. The screen shows the sternum model. With all three user interfaces, the space key on a standard keyboard was used to create the markers on the marking positions. Subjective results of the study. Copyright International Journal of Human-Computer Studies
(A) The vibrotactile VR interface using a Vive controller and a Vive VR headset. A close-up image of the trigger button of the controller is shown at the bottom left corner. The additional Samsung LCD 2D display (not visible to the user) shows the brain model. (B) The kinesthetic VR interface using a Geomagic Touch X force-feedback device and a Vive VR headset. A close-up image of the device button is shown at the bottom left corner. The additional 2D display shows the hipbone model. (C) The traditional 2D interface using a standard mouse with the Samsung LCD 2D display. A close-up image of the mouse buttons is shown at the bottom left corner. The screen shows the sternum model. With all three user interfaces, the space key on a standard keyboard was used to create the markers on the marking positions.Subjective results of the study. Copyright International Journal of Human-Computer Studies

Abstract: 

Three-dimensional (3D) visualization has been widely used in computer-aided medical diagnosis and planning. To interact with 3D models, current user interfaces in medical systems mainly rely on the traditional 2D interaction techniques by employing a mouse and a 2D display.

There is promising haptic virtual reality (VR) interfaces that can enable intuitive and realistic 3D interaction by using VR equipment and haptic devices. However, the practical usability of the haptic VR interfaces in this medical field remains unexplored. In this study, we propose two haptic VR interfaces, a vibrotactile VR interface, and a kinesthetic VR interface, for medical diagnosis and planning on volumetric medical images.

The vibrotactile VR interface used a head-mounted VR display as the visual output channel and a VR controller with vibrotactile feedback as the manipulation tool. Similarly, the kinesthetic VR interface used a head-mounted VR display as the visual output channel and a kinesthetic force-feedback device as the manipulation tool.

We evaluated these two VR interfaces in an experiment involving medical marking on 3D models, by comparing them with the present state-of-the-art 2D interface as the baseline. The results showed that the kinesthetic VR interface performed the best in terms of marking accuracy, whereas the vibrotactile VR interface performed the best in terms of task completion time. Overall, the participants preferred to use the kinesthetic VR interface for the medical task.

Subjective results of the study. Copyright International Journal of Human-Computer Studies
Subjective results of the study. Copyright International Journal of Human-Computer Studies

Real‐time suturing simulation for virtual reality medical training

– Authored by Peng Yu  Junjun Pan  Hong Qin  Aimin Hao  Haipeng Wang. Computer Animation and Virtual Worlds. 1 September 2020

Suturing simulation of the hand. Copyright. Computer Animation and Virtual Worlds
Suturing simulation of the hand. Copyright. Computer Animation and Virtual Worlds

Abstract: 

At present, virtual reality (VR) ‐based medical simulators provide an efficient and cost‐effective alternative without exposing risk to the traditional training approaches. As an essential and indispensable task in fundamental surgical skills training, the research of suturing simulation still remains insufficient in the field of virtual surgery.

In this paper, we present a real‐time suturing simulation framework that can handle the complex interactions between surgical instruments and soft tissue. The simulation consists of two stages: external interaction and internal coupling.

External interaction involves the interplay between needle/suture and the soft tissue, which are both deformed by position‐based dynamics (PBD) with different constraints. At the internal coupling stage, once the force exceeds a threshold, the needle tip will puncture and penetrate into the soft tissue and generate a path. To guarantee the needle/suture accurately following the path inside the soft tissue, we propose a novel coupling method by matching and generating the constraints among needle, suture, and penetration path.

We have applied this suturing simulation into a VR laparoscopic surgery simulator with haptic force. Our experimental results demonstrate that our approach can achieve real‐time performance with a high degree of visual realism and haptic fidelity.

Interface of prototyped virtual reality laparoscopic surgery simulator. Copyright. Computer Animation and Virtual Worlds
Interface of prototyped virtual reality laparoscopic surgery simulator. Copyright. Computer Animation and Virtual Worlds

3D Printing In Orthopedics: Implants, Drug Delivery, Bone Regeneration

3D Bioprinting for Wound Healing and Skinlike Sensors
Automated Bioprinting

3D Printed Drug Delivering Medical Devices

3D Printing Ceramic Implants

3D Bioprinting Knee, Aneurysm Model, 3D Organization Using Microfluidics

3D Printing Pharmaceuticals and Drug Delivery Devices

3DHEALS Guides (Collective) – This is where we dive deep into subjects that you will find helpful for your projects and career.

3DEALS Expert Corner (Collective) – This is where we invite field experts to write their perspectives in a first-person narrative. To write for this column, please email: info@3dheals.com

3DHEALS From Academia (Collective) – This section features recent, relevant, close to commercialization academic publications in the space of healthcare 3D printing, 3D bioprinting, and related emerging technologies.

Other similar articles

Comments