Kevin Yu
Research Focus
- Augmented Reality and Virtual Reality for Medical Applications
- Visualization of Medical Data
- User Interaction in XR and 3DUI
Current Projects
ARTEKMED
Don't miss out the project homepage: http://www.artekmed.de/
The goal of ARTEKMED is to provide paramedics and young doctors the possibility to allow a remote expert to help them in unfamiliar situations. By using Augmented Reality and head-mounted displays (i.e. Microsoft HoloLens), they can see 3D annotations and live guidance from the remote expert. The remote expert sees the 3D reconstruction of the local environment including patient data and real-time sensor data, in order to provide conscientious teleconsultation and annotations. The 3D reconstruction is made possible by using a multi-camera setup at the emergency site, either using the built-in cameras of the head-mounted displays itself or cameras inside the ambulance, to reconstruct the environment of the patient and send this data to the remote expert.
ArtekMed is currently in mid-development stage and is operated in tight collaboration with the NarVis group (http://campar.in.tum.de/Main/NarvisLabNew) and the Institute for Emergency Medicine (https://www.inm-online.de/de/)
| Open topics: Development of a "Surgical Cockpit", including UI and annotations:
|
Please contact me, if you are interested in this project. Preferably, you have experience with Unity / C# or AR/VR.
If you have cool ideas for AR/VR in a clinical environment, you are welcome to discuss it with us. We can provide HoloLens, 3D Printers, clinical partners, and more.
Augmented Mirror
A recurrent problem in egocentric Augmented Reality (AR) applications is the misestimation of depth. Providing alternative views from non-egocentric perspectives can convey useful information for applications that require the correct judgment of depth as it is in the case of placement and alignment of virtual and real content, but also for exploration and visualization tasks.In this paper, we introduce Augmented Mirrors. Through the integration of a real mirror, our approach is capable to reflect changes of the real and virtual content of an AR application while users benefit from the perceptual advantages of using mirrors. Our concept, simple yet effective, only requires tracking the user and mirror poses with the accuracy demanded by a specific application. To showcase the potential and flexibility of the Augmented Mirrors, we present and discuss multiple examples ranging from alignment, exploration, spatial understanding, and selective content visualization using different AR-enabled devices and tracking technologies. We envision the Augmented Mirrors as a new and valuable concept that can be used in applications that benefit from additional viewpoints and require the simultaneous visualization of real and virtual content.
This project is with collaboration with the Chair of Computer Aided Medical Procedures by Prof. Nassir Navab.
Open topics:
This can be topic of a bachelor or master thesis. Knowledge in Computer Vision and coding are required.
Visualization of Sterility
Open Topics:
This can be topic of a bachelor or master thesis. Knowledge in coding (C#) and Unity is recommended.
Students
| Student | Type | Topic |
|---|---|---|
| Thomas Wegele | Bachelor's Thesis (finished) | Development of a Concept for an Augmented Reality Telecommunication Application for Medical Purposes |
| Alexander Burghardt | Bachelor's Thesis (finished) | Efficient Real-time Point Cloud Transfer and Virtual Reality Rendering System |
| Thomas Nibler | IDP (finished) | Comparison of RTSP Implementations for Data-Driven Applications |
| Vanessa Kern | Bachelor's Thesis (finished) | Zooming Augmented Mirrors: Applying the Paradigm of Magnifying Real and Virtual Augmentations on Planar and Curved Mirrors |
| Ulli Harlass | MA (external) | tbd |
| Ihsan Berkan Balaban | MA (ongoing) | tbd |
News
ArtekMed @ MEDICA 2021
The team of ArtekMed was invited to MEDICA 2021 and present the current results of the research project. We received great feedback and even greater contacts for future collaborations and partners. Thanks to everyone supporting us so far!
Don't miss our talks at ISMAR 2021!
I am proud to announce that my journal paper together with Gleb Gorbachev has been accepted to ISMAR 2021! We will present our work on the virtual conference on the 5th of October at 4pm CEST during the avatar-session.
Honorable Mention Demo Award for work on Magnoramas!
IEEE VR 2021 was super exciting and well-done in nearly all aspects. I am happy to announce that our demo on the work of Magnoramas has received the Honorable Mention Demo Award. I want to thanks everyone who was involved in this project and hope to achieve much more in the future.
Don't miss our talks on IEEE VR 2021!
Exciting news! Two papers from the ArtekMed project are accepted to this years IEEE VR 2021 conference.
Please do not miss this chance and visit the online presentation of our work.
- Conference Paper:
Magnoramas: Magnifying Dioramas for Precise Annotations in Asymmetric 3D Teleconsultation
- 3DUI Contest:
Real-time Mixed Reality Teleconsultation for Intensive Care Units in Pandemic Situations
If you are not convinced yet, please take a look at the teaser video.
Aug 5th - Aug 16th: Medical Augmented Reality Summer School 2019 in Balgrist, Zurich
The focus of the Medical Augmented Reality Summer School Zurich is to teach the new generation of scientists the basics, challenges and recent advantages of Augmented Reality as well as to strengthen the connection between medicine, science and industry. In the first week, leading scientists of the field will present keynotes and insights from a medical and technical point of view. A condensed version of the TU Munich and JHU Baltimore lecture “Medical Augmented Reality” will provide profound theoretical knowledge of the topic. In the second week, the participants will develop their own Medical AR solutions using cutting-edge technologies in interdisciplinary teams.
At this amazing event, my group developed ARISE during the one-week hackathon. ARISE is an AR parkour for the physical rehabilitation of patients after a stroke incident. It combines the SLAM capability of the HoloLens 2 with the mobile motion tracking system Xsens. The result is a highly powerful system for generating large and adaptiv AR parkours with integrated patient feedback.
Publications
2022
- Yu K, Eck U, Pankratz F, Lazarovici M, Wilhelm D, Navab N. Duplicated Reality for Co-located Augmented Reality Collaboration IEEE VR 2022, 2022 Mar.
- [Poster] Yu K, Eck U, Navab N. Spatial Exploration with a WiM for Capturing 3D Dioramic Snapshots IEEE VR 2022 3DUI Contest, 2022 Mar.
2021
- Ostler D, Steger J, Bernhard L, Yu K, Hartwig R, Feussner H, Wilhelm D. “Hybrid” scientific conference: lessons learned from the digital annual meeting of the CARS international conference during the Covid-19 pandemic. Innovative Surgical Sciences. 2021 Nov 9.
- Strak R, Yu K, Pankratz F, Lazarovici M, Sandmeyer B, Reichling J, Weider S, Kraetsch C, Roegele B, Navab N, Eck U, Roth, D. Comparison Between Video-mediated and Asymmetric 3D Teleconsultation During a Preclinical Scenario Mensch und Maschinen, 2021 Sep
- Yu K, Ostler D, Fuchtmann J, Zapaishchykova A, Berlet M, Navab N, Feussner H, Wilhelm D. Clean-AR: Using Augmented Reality for Reducing the Risk of Contamination from Airborne Disease Agents on Surfaces Current Directions in Biomedical Engineering, 2021 Aug
- Yu K*, Gorbachev G*, Eck U, Pankratz F, Navab N, Roth D. Avatars for Teleconsultation: Effects of Avatar Embodiment Techniques on User Perception in 3D Asymmetric Telepresence IEEE Transactions on Visualization and Computer Graphics 2021, 2021 Aug, (*Shared Authorship)
- [Utility Model] Yu K, Gomez A, Navab N, Winkler A (all authors equally own this patent). Vorrichtung eines Spiegels mit Erweiterter Realität.
- [Poster] Roth D, Yu K, Pankratz F, Gorbachev G, Keller A, Lazarovici M, Wilhelm D, Weidert S, Navab N, Eck U. Real-time Mixed Reality Teleconsultation for Intensive Care Units in Pademic Situations, 3DUI Contest, IEEE VR 2021, 2021 Mar,
- Yu K, Winkler A, Pankratz F, Lazarovici M, Wilhelm D, Eck U, Roth D, Navab N. Magnoramas: Magnifying Dioramas for Precise Annotations in Asymmetric 3D Teleconsultation 2021 IEEE Virtual Reality and 3D User Interfaces (VR) (pp. 392-401). IEEE. Honorable Mention Demo Award
2020
- Yu K*, Gomez A*, Winkler A*, Roth D, Eck U, Navab N. Augmented Mirrors IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2020 Nov (*Shared Authorship)
- Yu K, Wegele T, Ostler D, Wilhelm D, Feußner H. EyeRobot: Enabling Telemedicine Using a Robot Arm and a Head-Mounted Display. Current Directions in Biomedical Engineering. 2020 Sep 17;6(1).
- Yu K*, Unberath M*, Barmaki R, Johnson A, Navab N Augment Yourself: Mixed Reality Self-Augmentation Using Optical See-through Head-mounted Displays and Physical Mirrors arXiv preprint arXiv:2007.02884 (*Shared Authorship)
- Yu K*, Held JPO*, Pyles C, Veerbeek JM, Bork F, Heining SM, Navab N, Luft AR Augmented Reality–Based Rehabilitation of Gait Impairments: Case Report. JMIR Mhealth Uhealth 2020;8(5):e17804, URL: https://mhealth.jmir.org/2020/5/e17804, DOI: 10.2196/17804, PMID: 32452815 (*Shared Authorship)
- Hartwig R, Ostler D, Feußner H, Berlet M, Yu K, Rosenthal J, Wilhelm D. COMPASS: Localization in Laparoscopic Visceral Surgery Current Directions in Biomedical Engineering 6.1 (2020)
2019
- Guo, Z., Yu, K., Pearlman, R., Navab, N., & Barmaki, R. (2019). Collaboration Analysis Using Deep Learning. arXiv preprint arXiv:1904.08066.
- Barmaki, R., Yu, K., Pearlman, R., Shingles, R., Bork, F., Osgood, G. M., & Navab, N. (2019). Enhancement of Anatomical Education Using Augmented Reality: An Empirical Study of Body Painting. Anatomical sciences education.
2018
- Deib, G., Johnson, A., Unberath, M., Yu, K., Andress, S., Qian, L., ... & Gailloud, P. (2018). Image Guided Percutaneous Spine Procedures Using an Optical See-Through Head Mounted Display: Proof of Concept and Rationale. Journal of neurointerventional surgery, 10(12), 1187-1191.
- Andress, S., Johnson, A., Unberath, M., Winkler, A., Yu, K., Fotouhi, J., ... & Navab, N. (2018, March). On-the-fly Augmented Reality for Orthopaedic Surgery Using a Multi-Modal Fiducial. In Medical Imaging 2018: Image-Guided Procedures, Robotic Interventions, and Modeling (Vol. 10576, p. 105760H). International Society for Optics and Photonics.
- Yu, K., Barmaki, R., Unberath, M., Mears, A., Brey, J., Chung, T. H., & Navab, N. (2018, May). On the Accuracy of Low-Cost Motion Capture Systems for Range of Motion Measurements. In Medical Imaging 2018: Imaging Informatics for Healthcare, Research, and Applications (Vol. 10579, p. 105790G). International Society for Optics and Photonics.
2017
- Qian, L., Unberath, M., Yu, K., Fuerst, B., Johnson, A., Navab, N., & Osgood, G. (2017). Towards Virtual Monitors for Image Guided Interventions-Real-Time Streaming to Optical See-Through Head-Mounted Displays. arXiv preprint arXiv:1710.00808.
- Bork, F., Barmaki, R., Eck, U., Yu, K., Sandor, C., & Navab, N. (2017, October). Empirical Study of Non-Reversing Magic Mirrors for Augmented Reality Anatomy Learning. In 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (pp. 169-176). IEEE.
Google Scholar: https://scholar.google.de/citations?user=DXhMcasAAAAJ
ResearchGate: https://www.researchgate.net/profile/Kevin_Yu22