Surgical field camera system that only uses images from cameras with an unobstructed sight line for tracking

A method comprising:

• a processor in communication with memory storing instructions for tracking an instrument within a surgical field, which when executed by the processor, cause the processor to:
• receive synchronized image captures from a plurality of cameras in a mesh of cameras, wherein the plurality of cameras includes at least two pairs of infrared cameras and an augmented reality (AR) headset camera distributed around the surgical field, the received synchronized image captures from all cameras within the mesh of cameras including synchronized image captures from the AR headset camera, wherein at least a portion of the synchronized image captures include information indicative of a position and orientation of a tracked instrument within the surgical field;
• analyze the synchronized image captures, to detect that a sight line from at least one camera in the mesh of cameras to the tracked instrument is at least partially obstructed to generate a non-obstructed group of images;
• generate tracking data for the tracked instrument, wherein the tracking data is generated from information extracted from the non-obstructed group of images including at least a portion of the synchronized image captures from the AR headset camera;
• determine, based on the tracking data, a position and an orientation of the tracked instrument within a virtual three-dimensional coordinate system;
• display the position and the orientation of the tracked instrument within a display of the AR headset;
• and
• utilize an image of the non-obstructed group of images including the tracked instrument to augment a view of the tracked instrument on the display of the AR headset when the AR headset camera is obstructed.

At least one non-transitory machine-readable medium including instructions for tracking an object within a surgical field, which when executed by a processor, cause the processor to:

• receive synchronized image captures from a plurality of cameras in a mesh of cameras, wherein the plurality of cameras includes at least two pairs of infrared cameras and an augmented reality (AR) headset camera distributed around the surgical field, the received synchronized image captures including a first image from a first camera in a first pair of the at least two pairs of infrared cameras, a second image from a second camera in the first pair of the at least two pairs of infrared cameras, and a third image from the AR headset camera, wherein at least a portion of the synchronized image captures include information indicative of a position and orientation of a tracked instrument within the surgical field;
• analyze the synchronized image captures, including the first image, the second image, and the third image, to generate tracking data for the tracked instrument, wherein the tracking data is generated from information extracted from at least two of the first image, the second image, and the third image;
• determine, based on the tracking data, a position and an orientation of the tracked instrument within a virtual three-dimensional coordinate system;
• display the position and the orientation of the tracked instrument on a display of the AR headset and utilizes images of the tracked instrument captured by the AR camera to highlight visible portions of the tracked instrument on the display of the AR headset;
• and
• utilize a non-obstructed image from the first image, the second image, and the third image including the tracked instrument to augment a view of the tracked instrument on the display of the AR headset when the AR headset camera is obstructed.