Spatiotemporal autocalibration of multi-segment robotic medical instruments

A method, comprising:

• receiving, by one or more processors coupled with memory, from a shape sensor system, pose information related to a tip of one or more segments along a flexible body of an instrument system used to perform a medical procedure;
• receiving, by the one or more processors, from a camera, a video stream related to the medical procedure;
• detecting, by the one or more processors, during the medical procedure, an out-of-calibration error between one or more frames of the video stream and the pose information;
• calibrating, by the one or more processors, responsive to detection of the out-of-calibration error during the medical procedure, the pose information with the video stream using a compensation matrix;
• and
• providing, by the one or more processors, for display via a display device, a location of the tip aligned based on application of the compensation matrix.

A non-transitory computer-readable medium storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to:

• receive, from a shape sensor system, position and orientation information related to a tip of one or more segments along a flexible body of an instrument system used to perform a medical procedure;
• receive, from a sensor, a video stream related to the medical procedure;
• detect, during the medical procedure, an out-of-calibration error between one or more frames of the video stream and the position and orientation information;
• align, responsive to detection of the out-of-calibration error during the medical procedure, the position and orientation information with the video stream using a compensation matrix;
• and
• perform an action related to the medical procedure using the aligned position and orientation information with the video stream.