Systems and methods for identifying and tracking physical objects during a robotic surgical procedure

A method of operating a surgical system to remove material from a bone at a surgical site including a retractor, the surgical system including a robotic device comprising a robotic arm configured to support and move a cutting instrument along a tool path to remove the material from the bone, a vision device attached to the robotic arm or the cutting instrument such that the vision device has a field of view including the surgical site and such that movement of the vision device relative to the surgical site occurs as a result of movement of the cutting instrument along the tool path to remove the material from the bone, a tracker coupled to the robotic device or the cutting instrument, a localizer configured to detect poses of the tracker, and a control system coupled to the localizer, the robotic device and the vision device, the method comprising the control system:

• controlling the robotic device to move the cutting instrument along the tool path for removing the material from the bone;
• acquiring, from the vision device, vision data sets captured from multiple perspectives of the surgical site enabled by the vision device being moved in a plurality of degrees of freedom as a result of the from the moving the cutting instrument along the tool path to remove the material from the bone;
• tracking a pose of the vision device at each location of the vision device in which the vision data sets are generated based on the detected poses of the tracker and a known spatial relationship between the tracker and the vision device;
• detecting the retractor from the vision data sets based on the pose of the vision device at each location of the vision device in which the vision data sets are generated;
• and
• generating an output for preventing collision of the cutting instrument with the retractor.