- Augmented Reality and Virtual Reality for Medical Applications
- Visualization of Medical Data
- User Interaction in XR and 3DUI
Don't miss out the homepage of the project: http://www.artekmed.de/
The goal of ARTEKMED is to provide paramedics and young doctors the possibility to allow a remote expert to help them in unfamiliar situations. By using Augmented Reality and head-mounted displays (i.e. Microsoft HoloLens), they can see 3D annotations and live guidance from the remote expert. The remote expert sees the 3D reconstruction of the local environment including patient data and real-time sensor data, in order to provide conscientious teleconsultation and annotations. The 3D reconstruction is made possible by using a multi-camera setup at the emergency site, either using the built-in cameras of the head-mounted displays itself or cameras inside the ambulance, to reconstruct the environment of the patient and send this data to the remote expert.
ArtekMed is currently in mid-development stage and is operated in tight collaboration with the NarVis group (http://campar.in.tum.de/Main/NarvisLabNew) and the Institute for Emergency Medicine (https://www.inm-online.de/de/)
Development of a "Surgical Cockpit", including UI and annotations:
Please contact me, if you are interested in this project. Preferably, you have experience with Unity / C# or AR/VR.
If you have cool ideas for AR/VR in a clinical environment, you are welcome to discuss it with us. We can provide HoloLens, 3D Printers, clinical partners, and more.
|Thomas Wegele||Bachelor's Thesis||Development of a Concept for an Augmented Reality Telecommunication Application for Medical Purposes|
Credits to Thomas Wegele.
IEEE VR 2021 was super exciting and well-done in nearly all aspects. I am happy to announce that our demo on the work of Magnoramas has received the Honorable Mention Demo Award. I want to thanks everyone who was involved in this project and hope to achieve much more in the future.
Exciting news! Two papers from the ArtekMed project are accepted to this years IEEE VR 2021 conference.
Please do not miss this chance and visit the online presentation of our work.
- Conference Paper:
Magnoramas: Magnifying Dioramas for Precise Annotations in Asymmetric 3D Teleconsultation
- 3DUI Contest:
Real-time Mixed Reality Teleconsultation for Intensive Care Units in Pandemic Situations
If you are not convinced yet, please take a look at the teaser video.
The focus of the Medical Augmented Reality Summer School Zurich is to teach the new generation of scientists the basics, challenges and recent advantages of Augmented Reality as well as to strengthen the connection between medicine, science and industry. In the first week, leading scientists of the field will present keynotes and insights from a medical and technical point of view. A condensed version of the TU Munich and JHU Baltimore lecture “Medical Augmented Reality” will provide profound theoretical knowledge of the topic. In the second week, the participants will develop their own Medical AR solutions using cutting-edge technologies in interdisciplinary teams.
At this amazing event, my group developed ARISE during the one-week hackathon. ARISE is an AR parkour for the physical rehabilitation of patients after a stroke incident. It combines the SLAM capability of the HoloLens 2 with the mobile motion tracking system Xsens. The result is a highly powerful system for generating large and adaptiv AR parkours with integrated patient feedback.
- Yu K, Winkler A, Pankratz F, Lazarovici M, Wilhelm D, Eck U, Roth D, Navab N. Magnoramas: Magnifying Dioramas for Precise Annotations in Asymmetric 3D Teleconsultation IEEE VR 2021, 2021 Mar, Honorable Mention Demo Award
- Yu K*, Gomez A*, Winkler A*, Roth D, Eck U, Navab N. Augmented Mirrors IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2020 Nov (*Shared Authorship)
- Yu K, Wegele T, Ostler D, Wilhelm D, Feußner H. EyeRobot: Enabling Telemedicine Using a Robot Arm and a Head-Mounted Display. Current Directions in Biomedical Engineering. 2020 Sep 17;6(1).
- Yu K*, Unberath M*, Barmaki R, Johnson A, Navab N Augment Yourself: Mixed Reality Self-Augmentation Using Optical See-through Head-mounted Displays and Physical Mirrors arXiv preprint arXiv:2007.02884 (*Shared Authorship)
- Yu K*, Held JPO*, Pyles C, Veerbeek JM, Bork F, Heining SM, Navab N, Luft AR Augmented Reality–Based Rehabilitation of Gait Impairments: Case Report. JMIR Mhealth Uhealth 2020;8(5):e17804, URL: https://mhealth.jmir.org/2020/5/e17804, DOI: 10.2196/17804, PMID: 32452815 (*Shared Authorship)
- Guo, Z., Yu, K., Pearlman, R., Navab, N., & Barmaki, R. (2019). Collaboration Analysis Using Deep Learning. arXiv preprint arXiv:1904.08066.
- Barmaki, R., Yu, K., Pearlman, R., Shingles, R., Bork, F., Osgood, G. M., & Navab, N. (2019). Enhancement of Anatomical Education Using Augmented Reality: An Empirical Study of Body Painting. Anatomical sciences education.
- Deib, G., Johnson, A., Unberath, M., Yu, K., Andress, S., Qian, L., ... & Gailloud, P. (2018). Image Guided Percutaneous Spine Procedures Using an Optical See-Through Head Mounted Display: Proof of Concept and Rationale. Journal of neurointerventional surgery, 10(12), 1187-1191.
- Andress, S., Johnson, A., Unberath, M., Winkler, A., Yu, K., Fotouhi, J., ... & Navab, N. (2018, March). On-the-fly Augmented Reality for Orthopaedic Surgery Using a Multi-Modal Fiducial. In Medical Imaging 2018: Image-Guided Procedures, Robotic Interventions, and Modeling (Vol. 10576, p. 105760H). International Society for Optics and Photonics.
- Yu, K., Barmaki, R., Unberath, M., Mears, A., Brey, J., Chung, T. H., & Navab, N. (2018, May). On the Accuracy of Low-Cost Motion Capture Systems for Range of Motion Measurements. In Medical Imaging 2018: Imaging Informatics for Healthcare, Research, and Applications (Vol. 10579, p. 105790G). International Society for Optics and Photonics.
- Qian, L., Unberath, M., Yu, K., Fuerst, B., Johnson, A., Navab, N., & Osgood, G. (2017). Towards Virtual Monitors for Image Guided Interventions-Real-Time Streaming to Optical See-Through Head-Mounted Displays. arXiv preprint arXiv:1710.00808.
- Bork, F., Barmaki, R., Eck, U., Yu, K., Sandor, C., & Navab, N. (2017, October). Empirical Study of Non-Reversing Magic Mirrors for Augmented Reality Anatomy Learning. In 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (pp. 169-176). IEEE.
Google Scholar: https://scholar.google.de/citations?user=DXhMcasAAAAJ
- Virtual Display in HoloLens with Real-Time Streaming
- HoloLens Volume Rendering
- Collaborative Pre-Operative Planning Using Augmented Reality
- Magic Mirror for Muscle Learning (also refer to Felix Bork)
- Rehabilitation Using Augmented Reality
A playlist of my (side-) projects can be found here: Youtube