Visual Tracking for Surgical Augmented Reality

In the research team MIMESIS, we develop interactive simulations allowing surgery training, planning and guidance. These simulations permits the modelling, in a realistic manner, the behavior of soft tissues (organs and other anatomical structures) during interactions with the medical instruments, while ensuring real-time computation.

Within our partnership with the IHU of Strasbourg ( www.ihu-strasbourg.eu ) we are developing new tools for Augmented Reality during minimally invasive liver surgery. The purpose is to superimpose in the intra-operative endoscopic images, the pre-operative 3D data (tumors and vascular networks) computed from CT scans to guide surgeon gestures. In order to etablish a reliable and robust augmented reality system in a complex environement (specular reflection, deformations, respiratory motion), we developed an approach which combines the extracted information from image data (acquired using endoscopic camera), with a biomechanical model of the liver. This internship will be dedicated to the development of a hybrid approach which will consist in the visual detection and the temporal tracking of points of interest (using fluorescence filter) on the surface of a kidney with the managing of lost points. This visual tracking will be used as first stage towards the establishing of a Augemnted Reality framework dedicated to partial nephrectomy (tumor resection in kidney).

Keywords: Features detectors and descriptors, Visual tracking, Interactive Simulation, Partial Nephrectomy.

Skills:
– C/C++ programming, Unix/Linux, some knowledge on OpenCV library will be highly appreciated.
– Motivation !

References

[1] Li-Ming Su, Balazs P. Vagvolgyi, Rahul Agarwal, Carol E. Reiley, Russell H. Taylor, and Gregory D. Hager. Augmented Reality During Robot-assisted Laparoscopic Partial Nephrectomy: Toward Real-Time 3D-CT to Stereoscopic Video Registration. Technology and Engineering 2008.

[2] Dogu Teber and Selcuk Guven and Tobias Simpfendörfer and Mathias Baumhauer and Esref Oguz Güven and Faruk Yencilek and Ali Serdar Gözen and Jens Rassweiler. Augmented Reality: A New Tool To Improve Surgical Accuracy during Laparoscopic Partial Nephrectomy? Preliminary In Vitro and In Vivo Results. European Urology 2009.

[3] Puerto-Souza, G.A. and Mariottini, G.L. Toward long-term and accurate Augmented-Reality display for minimally-invasive surgery. ICRA 2013.

[4] Haouchine, Nazim and Dequidt, Jeremie and  Peterlik, Igor and Kerrien, Erwan and Berger, Marie-Odile and Cotin, Stephane. Image-guided Simulation of Heterogeneous Tissue Deformation for Augmented Reality during Hepatic Surgery . TVCG 2014

Version Française

Dans l’équipe de recherche MIMESIS, nous développons des simulations interactives permettant l’apprentissage, la planification pré-opératoire et l’aide au geste intra-opératoire. Ces simulations doivent permettre de modéliser de manière réaliste le comportement des tissus mous (organes et autres structures anatomiques) lors d’interactions avec les instruments médicaux, tout en respectant une contrainte de calcul en temps-réel.

Dans le cadre de notre partenariat avec l’IHU de Strasbourg (www.ihu-strasbourg.eu) nous développons de nouveaux outils pour la réalité augmentée lors de chirurgie hépatique minimalement invasive. Le but étant d’integrer aux images endoscopiques intra-operatoires les données 3D pré-operatoires (tumeurs et réseaux vasculaires) issues de scans CT afin de guider les gestes chirurgicaux. Afin d’etablir un système de realité augmentée fiable et robuste dans un environement complexe (réflection spéculaire, déformations, mouvement réspiratoire), nous développons actuellement une approche qui combine les informations extraites de l’image a un modèle bioméchanique du foie. C’est dans ce context que se situe ce stage qui sera consacré au développement d’une approche hybride qui consistera a extraire des points d’intérêts fluorescent de la surface du foie et de les tracker temporellement d’une manière robuste en gérant la perte de points et l’appartion de nouveaux.

Mots clés: Features detectors and descriptors, Visual tracking, Interactive Simulation, Partial Nephrectomy.

Compétences:

– Programmation C/C++, Unix/Linux, la connaissance de la librairie OpenCV serait un plus.
– Motivation.

References

[1] Li-Ming Su, Balazs P. Vagvolgyi, Rahul Agarwal, Carol E. Reiley, Russell H. Taylor, and Gregory D. Hager. Augmented Reality During Robot-assisted Laparoscopic Partial Nephrectomy: Toward Real-Time 3D-CT to Stereoscopic Video Registration. Technology and Engineering 2008.
[2] Dogu Teber and Selcuk Guven and Tobias Simpfendörfer and Mathias Baumhauer and Esref Oguz Güven and Faruk Yencilek and Ali Serdar Gözen and Jens Rassweiler. Augmented Reality: A New Tool To Improve Surgical Accuracy during Laparoscopic Partial Nephrectomy? Preliminary In Vitro and In Vivo Results. European Urology 2009.
[3] Puerto-Souza, G.A. and Mariottini, G.L. Toward long-term and accurate Augmented-Reality display for minimally-invasive surgery. ICRA 2013.
[4] Haouchine, Nazim and Dequidt, Jeremie and  Peterlik, Igor and Kerrien, Erwan and Berger, Marie-Odile and Cotin, Stephane. Image-guided Simulation of Heterogeneous Tissue Deformation for Augmented Reality during Hepatic Surgery . TVCG 2014

Stéphane Cotin

Research Director