Building extended 2D and 3D field of views using images including poor information: application to endoscopy

Endoscopy is the reference examination to visualize and diagnose diseases in hollow organs (bladder, stomach, etc.). However, the small field of view (FoV) of endoscopes and the 2D nature of the data complicate the examinations made by endoscopists. The degree of chronic inflammation in the stomach (gastroscopy) or the extend of cancerous tumors in the bladder are not easily determined in video sequences. The latter are also rarely archived because they are very difficult to evaluate without guiding the endoscope. Mosaics (“maps” corresponding to a wide FoV) can be calculated from 2D video sequences. These mosaics facilitate the diagnosis, allow to archive a panoramic view of the organ (e.g., for a confirmation of the diagnosis by a second physician) and give a representation of the evolution of diseases (by visually comparing two mosaics or by registering them).

An accurate and robust mosaicing of endoscopic data is challenging for several reasons. On the one hand, very little texture and structure information is available in the images and strong illumination variations affect the data depending on the viewpoint of the endoscope. On the other hand, the trajectory and viewpoints of an endoscope are difficult to control and the image sensor is close to the internal organ surface.

This talk first focusses on descriptor-based illumination invariant optical flow techniques [1] allowing to find homologous points between pairs of complex medical images. Then, 2D cartography [2] results are shown to illustrate the interest of optical flow in gastroscopy. Finally, an improved “Structure-from-Motion” (SfM) technique is detailed. This approach can manage strong illumination variations, is applicable to various endoscopic examinations, and is suitable for different types of light source colors (white light, green-blue narrow band imaging and fluorescence). SfM is an approach that is widely used in numerous fields (e.g. archaeology), mainly when the acquisition conditions are controllable and rich texture information is present in the images. This talk shows how the SfM approach can be adapted to complex scenes [3]. References

[1] Dinh-Hoan Trinh and Christian Daul, “On illumination-invariant variational optical flow for weakly textured scenes”, Computer Vision and Image Understanding, Vol.179, Issue C, pages 1-18, 2019.
[2] O. Zenteno, D.-H. Trinh, S. Treuillet, Y. Lucas, T. Bazin, D. Lamarque, C. Daul, “Optical biopsy mapping on endoscopic image mosaics with a marker-free probe”, Computers in Biology and Medicine, Vol. 143, 2022.
[3] T.-B. Phan, D.-H. Trinh, D. Wolf, C. Daul, “Optical flow-based structure-from-motion for the reconstruction of epithelial surfaces”, Pattern Recognition, Vol. 105, 2020.