Structure annotation

A computer system comprising one or more computers programmed or otherwise configured to:

• receive an image to be annotated;
• determine a relative pose between the image and a 3D object model;
• raycast through each pixel of the plane of the image and determining if and where the ray intersects a surface of the 3D model, thereby determining one or more raycasted values for the pixels;
• and
• determine an instance segmentation mask as substantially optimizing an energy function, wherein the instance segmentation mask assigns, to each pixel of a set of pixels in the image plane, an instance classification label, and wherein the energy function has:
• a unary term, which aggregates a unary function over the instance segmentation mask, the unary function applied to the instance classification label of each pixel individually, wherein the unary function is defined by one or more raycasted values, as determined in said projecting step by raycasting through that pixel and determining if and where the ray intersects a surface of the 3D model, and
• a non-unary term, which aggregates a non-unary function over the instance segmentation mask, the non-unary function applied to the instance classification labels of at least two of the pixels.

One or more computer programs embodied on non-transitory computer-readable media for programming a computer system to carry out the steps of:

• receiving at the annotation computer system at least one captured frame comprising a set of 3D structure points, in which at least a portion of a structure component is captured;
• computing a reference position for the structure component within the at least one captured frame;
• generating a 3D model for the structure component by selectively extracting 3D structure points of the at least one captured frame based on the reference position;
• computing a projection of the 3D model into an image plane;
• and
• storing 2D annotation data of the computed projection in persistent computer storage for annotating the structure component within the image plane.