Inverting neural radiance fields for part and scene pose estimation

A method that generates an application specific pose estimation responsive to a camera pose, a scene pose and at least one object pose, the method comprising:

• generating an observed image (Iobs) of at least one object and a scene using a camera;
• processing the observed image to generate a current pose estimate having a current object pose and a current scene pose, the processing including segmenting the observed image using a 2D segmentation technique to obtain from the observed image a first segmented component representing a scene image of the current scene pose and a second segmented component representing an object image representing the current object pose that is an isolated from the first segmented component:
• generating an object render by applying an-the object image to a first Neural Radiance Field (NeRF), and
• generating a scene render by applying, independently from the object image, the scene image to a second NeRF that is different from the first NeRF;
• generating a final rendered output (Irndr) by combining the a object render and the scene render;
• comparing the observed image (Iobs) with the final rendered output (Irndr) to obtain a loss metric, wherein the loss metric is responsive to a difference between the observed image (Iobs) and the final rendered output (Irndr);
• processing the final rendered output (Irndr) and the observed image (Iobs) to generate a final pose estimate, wherein the final pose estimate matches the observed image (Iobs) when the loss metric is equal or greater than a loss metric threshold,
• wherein the object render and the scene render are generated independently based on the segmented components, and
• wherein the object render and the scene render are composited to generate the final rendered output (Irndr).