Localization of objects encoded in image data in accordance with natural language queries

A computing system, comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable media that store instructions that when executed by the one or more processors, cause the computer system to perform operations, the operations comprising:
• generating a set of image features tensors based on image data encoding an image and an image features model, wherein the set of image features tensors includes first image features that encode a high-level features map for the image and second image features that encode a low-level features map for the image;
• generating a set of textual features tensors, which includes at least a first textual features tensor, based on a text-based query and a textual features model, wherein at least a first object depicted in the image corresponds to the text-based query;
• generating a set of fused features tensors, which includes at least a first fused features tensor, based on a feature fusion model, the set of image features tensors, and the set of textual features tensors;
• determining a set of bounding boxes, which includes at least a first bounding box, based on a decoder model and the set of fused features tensors, wherein when the first bounding box is positioned on the image, the first bounding box encompasses the first object that corresponds to the text-based query;
• and
• providing an indication of at least the first bounding box.

One or more tangible non-transitory computer-readable media storing computer-readable instructions that when executed by one or more processors cause the one or more processors to perform operations, the operations comprising:

• generating a set of image features tensors based on image data encoding an image and an image features model, wherein the set of image features tensors includes first image features that encode a high-level features map for the image and second image features that encode a low-level features map for the image;
• generating a set of textual features tensors, which includes at least a first textual features tensor, based on a text-based query and a textual features model, wherein at least a first object depicted in the image corresponds to the text-based query;
• generating a set of fused features tensors, which includes at least a first fused features tensor, based on a feature fusion model, the set of image features tensors, and the set of textual features tensors;
• determining a set of bounding boxes, which includes at least a first bounding box, based on a decoder model and the set of fused features tensors, wherein when the first bounding box is positioned on the image, the first bounding box encompasses the first object that corresponds to the text-based query;
• and
• providing an indication of at least the first bounding box.