Enhancing surgeon action detection in robot-assisted minimally invasive surgery

Abstract

Although minimally invasive surgeries have achieved impressive results during the last decade, more complex surgeries require higher technology, such as robotics. Smart Autonomous Robotic Assistant Surgeon (SARAS) project has developed a novel challenge,

named Endoscopic Surgeon Action Detection (ESAD), with the goal of making minimally invasive surgery safer. In the context of this project, a deep learning algorithm has been developed to identify and localize the surgical actions performed by the main surgeon. This study aims to investigate different approaches to improve the baseline model provided by SARAS project. The model consists of a ResNet based Feature Pyramid Network connected to a RetinaNet algorithm. Towards this end, an ablation study was performed through the implementation of different processing techniques: removal of the surrounding black borders of the images, data augmentation of underrepresented surgical actions, and hyperparameter optimization regarding the loss function, batch size, number of iterations, initialization weights and optimizer. Also, a novel deep learning framework for real-time spatiotemporal action classification and localization was implemented in order to explore temporal connections among the detected actions, and evaluate their impact on the results of the former model. This model consists of a Visual Geometry Group network connected to a Single Shot Detector algorithm. Our results suggest that the experiment with better performance of the ablation study

is more accurate than the novel deep learning framework. Its performance on the identification and localization of surgical actions obtained a mean Average Precision (mAP) of

15.27, four points below the best team of SARAS-ESAD challenge with a mAP of 19.28.