The SARS-CoV-2 virus has forced our global community to practice social distancing and work remotely. This has increased the adoption of cloud-based productivity and collaboration tools and promoted the development of creative new solutions. Scientists have had to find new ways to continue their critical research to study the novel coronavirus and develop drugs. Among these new tools is Nanome, a virtual reality software for molecular modeling and structure-based drug design.

Like all scientists, Dr. Michael Kuiper and his colleagues at CSIRO, an Australian research organization, have had to adapt to the Covid-19 pandemic and find ways to continue their essential work. Dr. Kuiper has been examining the SARS-CoV-2 spike protein in Nanome and recently met with the company’s CEO Steve McCloskey in a virtual workspace. This conversation was recorded and can be viewed on YouTube.      

As a biomolecular modeler, Dr. Kuiper has been collaborating with researchers who monitor the evolution of the SARS-CoV-2 virus and work on vaccine development. During the immersive real-time meeting in Nanome, Dr. Kuiper showed McCloskey his molecular dynamics simulation of the spike protein and highlighted some of the mutations in the UK, South Africa, and Brazil SARS-CoV-2 variants. 

Dr. Kuiper loaded his molecular dynamics simulation of the spike protein into the virtual environment, gripped the 3D structure with his virtual hands, and pulled it apart to enlarge it. He pointed out the amino acid lysine Lys484 in the spike protein seen in the South African variant, which mutated from glutamine (E484K mutation). “We see the human receptor ACE2 bind to the spike protein receptor-binding domain, and all of a sudden this mutation starts to make a lot more sense. It’s now creating a salt bridge between the lysine in the spike protein and glutamate in the human ACE2 receptor,” said Dr. Kuiper.

By looking at mutations like the E484K mutation, Dr. Kuiper and his collaborators look for changes in the SARS-CoV-2 virus that may strengthen the interaction and the specificity of its spike protein receptor-binding domain with the human receptor ACE2.
Next, Dr. Kuiper shifted gears away from visualization and modeling of molecular structures to experimental work. He pulled up the journal article “Deep Mutational Scanning of SARS-CoV-2 Receptor Binding Domain Reveals Constraints on Folding and ACE2 Binding” published in Cell on September 2020.

In this scientific article, Dr. Kuiper zoomed in on a figure of a heat map in which every small square represents a recombinant protein. Blue colors represent mutations that resulted in tighter ACE2 binding. Red colors represent mutations that resulted in detrimental binding. Dr. Kuiper pointed out position 501 which is blue and cross-referenced it with the table of mutations. This experiment predicted the UK, South Africa, and Brazil lineages, aka B.1.1.7, B.1.351, and P.1.

Infections, immunity, and vaccinations add selection pressure to the SARS-CoV-2 virus, causing it to mutate. This is not necessarily a bad thing, Dr. Kuiper explained, because the virus can also mutate to become less virulent. 

“We’re a giant petri dish right now,” remarked McCloskey.

This experiment also predicted the mink mutation. For an in-depth analysis of this mutation, take a look at the “COVID-19 in VR: Spike Protein Mink Mutations” video. Dr. Kuiper expressed that thanks to high-quality research like this, scientists have already uncovered many valuable insights. However, because there is so much information out there it is difficult to bring it all together.   

“That’s what I love about VR. When you’re sharing that experience with someone it’s no longer just a little blip on a graph. You can actually point at residues. For instance, you can highlight two residues that seem to have an important interaction,” said Dr. Kuiper. 

This model does not take into account solvation effects. (The solvation effect features an interaction between a solute and a solvent, resulting in stabilization of the solute particles in the solution. When an ion in the solution is in the solvated state, it is surrounded or complexed by the molecules of the solvent.) Water molecules affect the interactions between charges and are therefore important for overall interactions. Also, some interactions might be “neutral” and not affect the ACE2 binding but will still make the SARS-CoV-2 virus better at evading our immune systems. Experiments are required to understand how much certain mutations can evade a particular antibody or therapeutic monoclonal antibody. 

Dr. Kuiper has been using Nanome to examine antibody receptor binding by looking at structural information. In virtual reality, he is able to zoom in and look at how the antibody interacts with a particular protein, usually its receptor-binding domain. Changes in residues at the interface are likely to affect the kinetics of the antibody receptor binding. Conducting an initial screen in Nanome helps Dr. Kuiper decide when to proceed to experiments. This is an example of how Nanome helps with drug design decision-making. 

Lastly, Dr. Kuiper played a molecular dynamics simulation and measured the distance between hydrogens at different frames. “Molecular dynamics simulations allow us to highlight some potential interactions which we can then further explore with experiments,” Dr. Kuiper explained. The experimental results will either confirm or refute a particular hypothesis or suggest how to improve a model. 

“Maybe we can suggest other experiments to really get to the bottom of what’s going on in a particular interaction very much like this Cell deep mutational paper that is really a tour de force of experimental work and that can feed directly into our understanding of interactions that are crucial to Covid-19 pandemic,”  Dr. Kuiper said.

Check out  Nanome YouTube channel for more “Covid-19 in VR” episodes and other remote real-time meetings in virtual reality. 

Be sure to check out our webinar “ Advanced Visualization for Healthcare” where the COO and co-founder of Nanome Keita Funakawa will be presenting.

About the Author:

Maria Karpenko

Maria leads marketing at Nanome, a virtual reality application for molecular modeling and structure-based drug design. Maria has a background in biomedical sciences and over a decade of experience in marketing and communications, digital health research, medical software product design, and editorial direction for an international arts magazine. She is passionate about increasing science and tech literacy and promoting critical thinking. 

Related Article:

Medical Simulation Using Augmented Reality, Virtual Reality, 3D Printing

Medical Applications of Augmented Reality

Mixed Reality and 3D Printing, Imperfections to Perfection?

From Academia: Mixed Reality, Augmented Reality, and 3D Printing in Healthcare

From 3D Printing to VR/AR: Simple Connection?

The post Virtual Reality Software For Molecular Modeling and Structure-Based Drug Design appeared first on 3DHeals.

The medical 3D imaging data can be used for a variety of 3D technologies ranging from 3D Printing to lately, virtual reality and augmented reality. In this issue of “From Academia” , we feature four publications investigating various applications and clinical outcomes of using medical VR/AR for medical simulation. The first article connects 3D printing, AR/VR to the management of kidney and prostate cancer. The second article focuses on diagnostic accuracies in congenital heart disease among conventional visualization, immersive, and non-immersive virtual reality. The winner of this small sample study was immersive virtual reality, especially when the case was complex. However, as we dive deeper into the AR/VR space, the human-computer interaction does not limit to simple visualization. The third article explores two existing haptic VR interfaces, a vibrotactile VR interface, and a kinesthetic VR interface, for medical diagnosis and planning on volumetric medical images. The final publication tackles the VR interface further by creating a prototype with suturing simulation in a VR laparoscopic surgery simulator with haptic force.

“From Academia” features recent, relevant, close to commercialization academic publications. Subjects include but not limited to healthcare 3D printing, 3D bioprinting, and related emerging technologies.

Email: Rance Tino (tino.rance@gmail.com) if you want to share relevant academic publications with us.

3D Printing, Augmented Reality, and Virtual Reality for the Assessment and Management of Kidney and Prostate Cancer: A Systematic Review

Authored by Nicole Wake, Jeffrey E. Nussbaum, Marie I. Elias, Christine V. Nikas, Marc A. Bjurlin. Urology. September 2020

A Novel Virtual Reality Medical Image Display System for Group Discussions of Congenital Heart Disease: Development and Usability Testing

Authored by Byeol Kim, Yue-Hin Loke, Paige Mass, Matthew R Irwin, Conrad Capeland, Laura Olivieri, Axel Krieger, JMIR Publications. 31 May 2020

Evaluation of haptic virtual reality user interfaces for medical marking on 3D models

Authored by Zhenxing Li, Maria Kiiveri, Jussi Rantala, Roope Raisamo. International Journal of Human-Computer Studies. 25 March 2021

Real‐time suturing simulation for virtual reality medical training

Authored by Peng Yu  Junjun Pan  Hong Qin  Aimin Hao  Haipeng Wang. Computer Animation and Virtual Worlds. 1 September 2020

Related Articles:

3D Printing In Orthopedics: Implants, Drug Delivery, Bone Regeneration

3D Bioprinting for Wound Healing and Skinlike Sensors
Automated Bioprinting

3D Printed Drug Delivering Medical Devices

3D Printing Ceramic Implants

3D Bioprinting Knee, Aneurysm Model, 3D Organization Using Microfluidics

3D Printing Pharmaceuticals and Drug Delivery Devices

3DHEALS Guides (Collective) – This is where we dive deep into subjects that you will find helpful for your projects and career.

3DEALS Expert Corner (Collective) – This is where we invite field experts to write their perspectives in a first-person narrative. To write for this column, please email: info@3dheals.com

3DHEALS From Academia (Collective) – This section features recent, relevant, close to commercialization academic publications in the space of healthcare 3D printing, 3D bioprinting, and related emerging technologies.

Other similar articles

The post From Academia: Medical Simulation Using Augmented Reality, Virtual Reality, 3D Printing appeared first on 3DHeals.

Want to write a piece for 3DHEALS Expert Corner? Email us: info@3dheals.com

Recently, we briefly reviewed a few publications examining the clinical values of mixed reality and 3D printing in our “Academia” section of the 3DHEALS blogs, and in particular, if combining the two technologies shows promises for a better overall product for pre- and intra-operative planning. This blog will dive deeper and hope to synthesize an updated overview of where the things are, shed light on future trends in a no-nonsense fashion, and perhaps provide an entry point for early adopters to these technologies.  

Virtual reality, Augmented reality, Mixed reality

First of all, it is important to clarify some terms.

At a high level, virtual reality (VR) implies a complete immersion experience that shuts out the physical world. Augmented reality (AR) adds digital elements to a live view often by using the camera on a smartphone, for example, augmented reality experiences using Snapchat lenses and the game Pokemon Go. Mixed reality not only puts digital objects in a user’s environment but also allow the users to interact with both the digital objects and his/her physical environment, including anatomy models during surgery. The Microsoft website provided a slightly more in-depth explanation of the concepts: [1]

“Mixed reality is the next evolution in human, computer, and environment interaction and unlocks possibilities that before now were restricted to our imaginations. It is made possible by advancements in computer vision, graphical processing power, display technology, and input systems. The term mixed reality was originally introduced in a 1994 paper by Paul Milgram and Fumio Kishino, “A Taxonomy of Mixed Reality Visual Displays.” Their paper introduced the concept of the virtuality continuum and focused on how the categorization of taxonomy applied to displays. Since then, the application of mixed reality goes beyond displays. It also includes environmental input, spatial sound, and location.”

The following demonstration further explains the concept that “mixed reality” is a spectrum rather than a single defined point: [1]

• “Towards the left (near physical reality). Users remain present in their physical environment and are never made to believe they have left that environment.
• In the middle (fully mixed reality). These experiences blend the real world and the digital world. Viewers who have seen the movie Jumanji can reconcile how the physical structure of the house where the story took place was blended with a jungle environment.
• Towards the right (near digital reality). Users experience a completely digital environment and are unaware of what occurs in the physical environment around them.”

What’s intriguing is that a 3D-printed model is entirely in the physical world yet representing 100% digital data (from the virtual world).

Mixed Reality versus 3D-Printing

The reason why 3D-printing and mixed reality are often evaluated side by side is that both are increasingly incorporated in advanced visualization workflow. For example, in a recent study comparing the technologies for nephron preservation surgery for Wilm’s tumor in pediatric patients [3], the authors presented the following workflow, demonstrating the close digital footprints of both procedures [Figure 1]. Both technologies can be used for surgical planning for a variety of specialties [2-7, 9-10], and both are often constrained by existing imaging processing capabilities including [5]:

• Imaging data acquisition/Resolution
• Segmentation
• Rendering
• Software analysis

Both technologies can produce models that are reproducible and provide a “shared” experience both before and during surgeries.

However, there are also unique value propositions from both technologies.

For mixed reality, the construction process of a final product is often less costly, and the user can manipulate the objects with more degrees of freedom (more adaptive). For example, the user can zoom in and out, or virtually “dissect” the model to see internal vascular structures, etc. The user can further superimpose other information onto the existing model, including functional data such as 3D tractography and finite element analysis. However, registration of the virtual model to the physical world remains a unique problem for mixed reality. Tissue deformation and tactile information from the real world are also lost for current MR technology. Additionally, significant technological hurdles include data transmission latency on current wireless networks, chunky and expensive headsets, additional hardware requirements including graphic cards, etc. 5G service is still in its infancy, particularly in the U.S. [8]

A unique value of the 3D printed model is that it can directly interact with the physical world. For example, surgeons can use a mandibular bone to pre-bend a metal plate before surgery, but not with mixed-reality models. Another example is a 3D printed surgical guide, which can directly interact with the patient’s anatomy and the surgeon. Once a model is printed, it can be shared without the need for any additional equipment or network, since it is now entirely physical.

Mixed Reality and 3D Printing

Several recent studies have explored if these currently still imperfect technologies can be complementary to each other, indicating a trend towards a more technology-agnostic approach in problem-solving. For example, Rafael Moreta-Martinez, et al [6] proposed a workflow to allow automatic registration of the real world and mixed reality data using a 3D-printed patient-specific registration instrument. Similarly, a few months later, Peng-Fei Lei, et al [2] designed a 3D-printed patient-specific registration instrument, complimenting 3D reconstructed virtual surgical planning and mixed reality intraoperative models in a complicated case hip arthroplasty case. The combination of a 3D printed surgical guide serving as a mixed reality automatic registration tool seems promising from both clinical and cost perspectives. It would be interesting to see more clinical applications leveraging both technologies simultaneously.

Conclusion:

Advanced visualization in healthcare has lately taken on new meanings with the advancement of several emerging technologies such as 3D printing and mixed reality. The progressive understanding of human-computer interaction is now elevating the human experiences in the operating theatres beyond visual experiences. That said, significant real technological hurdles exist for both MR and 3D printing. Additionally, the lack of outcome studies and quality control standards also limit wide-spread adoption.

Nonetheless, the optimists will say that few technological breakthroughs are accidental. The progress we are seeing in advanced visualization tools in healthcare today will not occur without past decades of innovation and improvements in imaging acquisition (i.e. CT, MR, etc.), computing power (cloud, etc), and digital manufacturing (e.g. 3D printing, digital milling, etc.). While most of these innovations appear only incremental improvements and often slower than our expectations, we may very well be on the cusp of the next quantum leap when these existing imperfect tools reach the perfect alignment.

References:

1. https://docs.microsoft.com/en-us/windows/mixed-reality/mixed-reality

2. Mixed Reality Combined With Three-Dimensional Printing Technology in Total Hip Arthroplasty: An Updated Review With a Preliminary Case Presentation (Peng-Fei Lei, et al) Orthop Surg, 11 (5), 914-920 Oct 2019

3. Comparison of 3-Dimensional and Augmented Reality Kidney Models With Conventional Imaging Data in the Preoperative Assessment of Children With Wilms Tumors (Lianne M Wellens, et al) JAMA Network Open, 2 (4), e192633 2019 Apr 5 PMID: 31002326

PMCID: PMC6481457 DOI: 10.1001/jamanetworkopen.2019.2633

4. Visualization Improves Supraclavicular Access to the Subclavian Vein in a Mixed Reality Simulator (Joshua Warren Sappenfield, et al) Anesthesia & Analgesia. 127(1):83–89, JULY 2018 PMID: 29200069 PMCID: PMC6774241DOI: 10.1213/ANE.0000000000002572

5. Neurosurgical Virtual Reality Simulation for Brain Tumor Using High-definition Computer Graphics: A Review of the Literature (Taichi Kin, et al) Neurol Med Chir (Tokyo), 57 (10), 513-520  2017 Oct 15 PMID: 28637947 PMCID: PMC5638778 DOI: 10.2176/nmc.ra.2016-0320

6. Augmented Reality in Computer-Assisted Interventions Based on Patient-Specific 3D Printed Reference (Rafael Moreta-Martinez, et al) Healthc Technol Lett, 5 (5), 162-166 2018 Sep 14 eCollection Oct 2018 PMID: 30464847 PMCID: PMC6222179 DOI: 10.1049/htl.2018.5072

7. Augmented Reality, Surgical Navigation, and 3D Printing for Transcanal Endoscopic Approach to the Petrous Apex (Samuel R Barber, et al) OTO Open, 2 (4), 2473974X18804492 2018 Oct 29 eCollection Oct-Dec 2018

PMID: 30719506 PMCID: PMC6348519 DOI: 10.1177/2473974X18804492

8. Virtual travel could change the world- if it gets off the ground.  Sara toth stub, dec. 12, 2019 10:00 am, Wall Street Journal

9. A Review of Simulation Applications in Temporal Bone Surgery (Tanisha S Kashikar, et al) Laryngoscope Investig Otolaryngol, 4 (4), 420-424

10. Augmented Reality and Three-Dimensional Printing in Percutaneous Interventions on Pulmonary Arteries (Jan Witowski, et al) Quant Imaging Med Surg, 9 (1), 23-29 Jan 2019 PMID: 30788243 PMCID: PMC6351817 DOI: 10.21037/qims.2018.09.08

Related Articles:

From 3D Printing to VR/AR: Simple Connection?

From Academia: 3D Printed Pills, 4D Printed Structure, and how to property 3D print Chocolate

From Academia: 3D-Printed Aligner, Bioprinting for Mouth Ulcer, Vertebroplasty Guides

From Academia: 3D Printing and Robotics, Stem cell coated Implants, Decentralized Mitigation of Pandemics

The post Mixed Reality and 3D Printing, Imperfections to Perfection? appeared first on 3DHeals.

In today’s issue, we share with you a collection of latest publications that explores a diverse take on digital imaging-modelling and 3D printing in Healthcare. Here, mixed reality (MR), augmented reality (AR), and virtual reality (VR) utilized for better simulation and visualization for pre-surgical and educational applications, and the combination of 3D printing technology with MR/AR/VR to further enchance modelling and simulation workflows as well as patient education. Stay tuned for a more in-depth discussion on the technical and clinical insights on our upcoming Expert Corner blog soon.

“From Academia” features recent, relevant, close to commercialization academic publications. Subjects include but not limited to healthcare 3D printing, 3D bioprinting, and related emerging technologies.

Email: Rance Tino (info@3dheals.com) if you want to share relevant academic publications with us.

Mixed Reality Combined With Three-Dimensional Printing Technology in Total Hip Arthroplasty: An Updated Review With a Preliminary Case Presentation

Authored by Peng-fei Lei, Shi-long Su, Ling-yu Kong, Cheng-gong Wang, Da Zhong, Yi-he Hu. Orthopaedic Surgery, October 2019

A Review of Simulation Applications in Temporal Bone Surgery

Authored by Tanisha S. Kashikar, Thomas F. Kerwin, Aaron C. Moberly, Gregory J. Wiet. Laryngoscope Investig Otolaryngol, 7 June 2019

Comparison of 3-Dimensional and Augmented Reality Kidney Models With Conventional Imaging Data in the Preoperative Assessment of Children With Wilms Tumors

Authored by Lianne M. Wellens, Jene Meulstee, Cornelis P. van de Ven. Quant Imaging Med Surg, 5 April 2019

Augmented Reality and Three-Dimensional Printing in Percutaneous Interventions on Pulmonary Arteries

Authored by Jan Witowski et al. Quant Imaging Med Surg, January 2019

Patient-specific 3D Printed and Augmented Reality Kidney and Prostate Cancer Models: Impact on Patient Education

Authored by Nicole Wake, Andrew B. Rosenkrantz, Richard Huang, Katalina U. Park, James S. Wysock, Samir S. Taneja, William C. Huang, Daniel K. Sodickson & Hersh Chandarana. 3D Printing in Medicine, 19 February 2019

Augmented Reality, Surgical Navigation, and 3D Printing for Transcanal Endoscopic Approach to the Petrous Apex

Authored by Samuel R. Barber, Kevin Wong, Vivek Kanumuri, Ruwan Kiringoda, Judith Kempfle, Aaron K. Remenscheider, Elliott D. Kozin, and Daniel J. Lee. OTO Open, 29 October 2018

Visualization Improves Supraclavicular Access to the Subclavian Vein in a Mixed Reality Simulator

Authored by Joshua Warren Sappenfield, William Brit Smith, Lou Ann Cooper, David Lizdas, Drew B. Gonsalves, Nikolaus, Gravenstein, Samsun Lampotang, Albert R. Robinson III. Anesthesia & Analgesia. July 2018

Neurosurgical Virtual Reality Simulation for Brain Tumor Using High-definition Computer Graphics: A Review of the Literature

Authored by Taichi Kin, Hirofumi Nakatomi, Kaoyuki Shono, Seiji Nomura, Toki Saito, Hiroshi Oyama, Nobuhito Saito. Neurologia medico-chirurgica. 15 October 2017

Augmented Reality in Computer-Assisted Interventions Based on Patient-Specific 3D Printed Reference

Authored by Rafael Moreta-Martinez, David Garcia-Mato, Monica Garcia-Sevilla, Ruben Perez-Mananes, Jose Calvo-Haro, Javier Pascau. Healthcare Technology Letters. September 2018

Related Articles:

From 3D Printing to VR/AR: Simple Connection?

From Academia: 3D Printed Pills, 4D Printed Structure, and how to property 3D print Chocolate

From Academia: 3D-Printed Aligner, Bioprinting for Mouth Ulcer, Vertebroplasty Guides

From Academia: 3D Printing and Robotics, Stem cell coated Implants, Decentralized Mitigation of Pandemics

The post From Academia: Mixed Reality, Augmented Reality, and 3D Printing in Healthcare appeared first on 3DHeals.

Want to write a piece for 3DHEALS Expert Corner? Email us: info@3dheals.com

AR/VR and 3D Printing are emerging technologies because of the advancement in computer science and our deepened understanding of human-computer interaction. The power of 3D printing is that digital design can now make a significant impact in the real world, either in the form of personalized medicine or architectural plans. However, the ultimate origin of the digital manufacture blueprints is the human mind. The uprising of VR/AR technologies will serve to expand the human minds, providing new multi-sensory environments to improve 3D printing design and to inspire 3D printing innovations. The result is a 3D printed product that will interact with the physical world more efficiently.

However, the possibilities of AR/VR and 3D printing working together to improve human lives are not limited to the digital manufacture workflow but revolutionize future innovation and product cycle (Figure 1).

Figure 1. VR/AR empowered 3D printing design/product cycle

Trust me; everyone is keeping an eye on this. Happy 2017!

The Augmented Mind: How AR/VR will empower 3D Printing technology in bettering the real world. appeared first on 3DHeals.