Methods and systems for imaging a scene, such as a medical scene, and tracking objects within the scene

A camera array, comprising:

• a support structure having a center,
• a depth sensor mounted to the support structure proximate to the center, wherein the depth sensor has a focal plane;
• a plurality of cameras mounted to the support structure radially outward from the depth sensor, wherein the cameras are configured to capture image data of a scene,
• wherein the cameras each have a focal axis, and wherein the focal axes of the cameras converge at a point below the focal plane of the depth sensor, and
• a plurality of trackers mounted to the support structure radially outward from the cameras,
• wherein the trackers are configured to capture positional data of a tool within the scene.

A mediated-reality system, comprising:

• a camera array including
• a support structure having a central region;
• a depth sensor mounted to the support structure at the central region;
• a plurality of cameras mounted to the support structure outward from the depth sensor away from the central region, wherein the cameras are configured to capture image data of a scene;
• and
• a plurality of trackers mounted to the support structure radially outward from the cameras away from the central region, wherein the trackers are configured to capture positional data of a tool within the scene;
• an input controller configured to control a position and orientation of a virtual perspective of the scene;
• a processing device communicatively coupled to the camera array and the input controller, wherein the processing device is configured to synthesize a virtual image corresponding to the virtual perspective based on the image data from at least two of the cameras;
• and
• determine a position of the tool based on the positional data from at least two of the trackers;
• and
• a display device communicatively coupled to the processing device, wherein the display device is configured to display a graphical representation of the tool at the determined position in the virtual image.

A method of imaging a subject within a scene, the method comprising:

• aligning a depth sensor with a portion of interest of the subject such that the portion of interest is positioned proximate a focal depth of the depth sensor;
• capturing, with a plurality of cameras, image data of the scene including the portion of interest of the subject;
• capturing, with a plurality of trackers, positional data of a tool within the scene, wherein the depth sensor, the cameras, and the trackers are commonly mounted to a support structure of a camera array;
• receiving input regarding a selected position and orientation of a virtual perspective of the scene;
• synthesizing a virtual image corresponding to the virtual perspective of the scene based on the image data;
• determining a position of the tool based on the positional data;
• and
• displaying, at a display device, a graphical representation of the tool at the determined position in the virtual image.

A method of imaging a subject within a scene, the method comprising:

• aligning a depth sensor of a camera array with a portion of interest of the subject such that the portion of interest is positioned proximate a focal depth of the depth sensor;
• capturing, with a plurality of cameras of the camera array, image data of the scene including the portion of interest of the subject, wherein the cameras each have a focal axis, and wherein the focal axes of the cameras converge at a point below the focal depth of the depth sensor
• capturing, with a plurality of trackers of the camera array, positional data of a tool within the scene,
• receiving input regarding a selected position and orientation of a virtual perspective of the scene;
• synthesizing a virtual image corresponding to the virtual perspective of the scene based on the image data;
• determining a position of the tool based on the positional data;
• and
• displaying, at a display device, a graphical representation of the tool at the determined position in the virtual image.

A method for determining a position of a tip of a tool within a scene, the method comprising:

• receiving positional data of the tool from at least two trackers mounted to the camera array;
• estimating a three-dimensional (3D) position of the tip of the tool based on the positional data;
• receiving an image of the scene from each of one or more cameras mounted to the camera array;
• for the image from each of the one or more cameras
• defining a region of interest in the image based on the estimated 3D position of the tip of the tool;
• and
• processing the image in the region of interest to determine a position of the tip of the tool in the region of interest;
• and
• determining an updated 3D position of the tip of the tool based on the determined position of the tool in the region of interest of the one or more images.

A method for determining a position of a tip of a tool within a scene, the method comprising:

• receiving positional data of the tool from at least two trackers;
• estimating a three-dimensional (3D) position of the tip of the tool based on the positional data;
• receiving an image from one camera;
• for the image from the camera
• defining a region of interest in the image based on the estimated 3D position of the tip of the tool;
• and
• processing the image in the region of interest to determine a position of the tip of the tool in the region of interest;
• and
• determining an updated 3D position of the tip of the tool based on the determined position of the tool in the region of interest of the image by
• projecting the position of the tip of the tool in the region of interest into a 3D line based on a calibration of the camera;
• and
• determining the updated 3D position of the tip of the tool based on an intersection of the 3D line and the primary axis of the tool.