Artificial Intelligence driven multimodal imaging for tumor diagnosis and therapy

Intraoperative tumor resection is based on the surgeon’s experience, supported by macroscopic, endoscopic, microscopic or robotic-assisted examination of the tumor. However, this approach does not allow for a precise definition of the tumor borders and can lead to incomplete tumour resection and thus poor survival of the patient. Biophotonic imaging approaches could play a crucial role in addressing this unmet medical need, as they provide besides morphological information also molecular information. Here, we will report on the investigation of novel multimodal labelfree spectroscopic instrumentation in combination with innovative artificial intelligence (AI) approaches. Labelfree multimodal spectroscopic imaging visualizes the tissue morphology and molecular structure. The multimodal images can be automatically analyzed using AI-based image analysis approaches into diagnostic information. To take full advantage of multimodal imaging approaches a major step forward would be the implementation of spectroscopic-guided femtosecond ablation in a “seek and treat manner”. We will introduce a multimodal nonlinear microendoscope, which also allows for the ablation of biological tissue with femtosecond lasers. The interplay between multimodal imaging in combination with innovative AI approaches for real-time spectral analysis and fs-laser ablation opens exciting new ways for an intraoperative histopathological tumor analysis and selective removal. An outlook to also include haptic feedback towards sensorised surgery for precise surgical guidance and intraoperative histopathological examination of tissue under in-vivo or near in-vivo conditions will be given. Acknowledgements Financial support of the EU, the ”Thüringer Ministerium für Wirtschaft, Wissenschaft und Digitale Gesellschaft”, the ”Thüringer Aufbaubank”, the Federal Ministry of Education and Research, Germany, the German Science Foundation, and the Carl-Zeiss Foundation are greatly acknowledged