Extended reality headset camera system for computer assisted navigation in surgery

An extended reality (XR) camera assembly for a head-mounted display device comprising:

• a rigid mounting element configured to be coupled to a head-wearable frame;
• a right-side visible light camera coupled to the rigid mounting element;
• a left-side visible light camera coupled to the rigid mounting element;
• a right-side near-infrared (NIR) camera coupled to the rigid mounting element;
• a left-side NIR camera coupled to the rigid mounting element;
• and
• at least one NIR light-emitting diode (LED) configured to illuminate a region within a field of view of the right-side NIR camera and the left-side NIR camera,
• wherein the right-side visible light camera and the left-side visible light camera are configured to capture stereoscopic visible light images within a field of view of a user when the user is wearing the frame, and
• wherein the right-side NIR camera and the left-side NIR camera are configured to capture stereoscopic NIR images within the field of view of the user when the user is wearing the frame,
• a tracking system adapted to determine a pose of dynamic reference arrays attached to a patient and to surgical instruments based on stereoscopic visible light images received through the right-side and left-side visible cameras, the tracking system further adapted to determine hand gesture based commands based on the stereoscopic NIR images received through the right-side and left-side NIR cameras.

A method of generating extended reality (XR) imagery comprising:

• capturing a first plurality of visible light stereoscopic images of a scene within a field of view of a pair of visible light cameras coupled to a rigid mounting element during a time period;
• capturing a second plurality of near-infrared (NIR) stereoscopic images of a scene within a field of view of a pair of NIR cameras coupled to the rigid mounting element during the time period;
• determining a pose of dynamic reference arrays attached to a patient and to a surgical instrument based on the first plurality of visible light stereoscopic images within a three-dimensional space;
• and
• determining a hand gesture based commands based on the second plurality of NIR stereoscopic images within the three-dimensional space.