Optimizations for Dynamic Object Instance Detection, Segmentation, and Structure Mapping

A system comprising:

• one or more processors and one or more computer-readable non-transitory storage media coupled to one or more of the processors, the one or more computer-readable non-transitory storage media comprising instructions operable when executed by one or more of the processors to cause the system to perform operations comprising:
• accessing an image;
• generating a feature map for the image using a first neural network;
• identifying a plurality of regions of interest in the feature map;
• generating a plurality of regional feature maps for the plurality of regions of interest, respectively, by sampling the feature map for the image;
• processing the plurality of regional feature maps using a second neural network to:
• detect at least one regional feature map in the plurality of regional feature maps that corresponds to a person depicted in the image;
• and
• generate a target region definition associated with a location of the person using the regional feature map;
• generating, based on the target region definition associated with the location of the person, a target regional feature map by sampling the feature map for the image;
• and
• generating:
• a keypoint mask associated with the person by processing the target regional feature map using a third neural network;
• or
• an instance segmentation mask associated with the person by processing the target regional feature map using a fourth neural network.

One or more computer-readable non-transitory storage media embodying software that is operable when executed to cause one or more processors to perform operations comprising:

• accessing an image;
• generating a feature map for the image using a first neural network;
• identifying a plurality of regions of interest in the feature map;
• generating a plurality of regional feature maps for the plurality of regions of interest, respectively, by sampling the feature map for the image;
• processing the plurality of regional feature maps using a second neural network to:
• detect at least one regional feature map in the plurality of regional feature maps that corresponds to a person depicted in the image;
• and
• generate a target region definition associated with a location of the person using the regional feature map;
• generating, based on the target region definition associated with the location of the person, a target regional feature map by sampling the feature map for the image;
• and
• generating:
• a keypoint mask associated with the person by processing the target regional feature map using a third neural network;
• or
• an instance segmentation mask associated with the person by processing the target regional feature map using a fourth neural network.