Dense Video Object Captioning from Disjoint Vision

One or more non-transitory computer-readable media that store computer-readable instructions that, when executed by a computing system, cause the computing system to perform operations, the operations comprising:

• obtaining, by the computing system, a video comprising a plurality of image frames, wherein the video depicts a plurality of objects;
• respectively processing, by the computing system, each image frame with a machine-learned object detection model to extract one or more sets of feature data respectively for one or more of the plurality of objects that are depicted in the image frame and to generate a set of bounding boxes and detection scores for the plurality of objects;
• processing, by the computing system, the sets of feature data extracted for the plurality of image frames with a machine-learned tracking model to generate a plurality of trajectories respectively for the plurality of objects, wherein the trajectory for each object identifies the sets of feature data that correspond to the object;
• obtaining, by the computing system, a textual query;
• and
• identifying, by the computing system based on the set of bounding boxes and detection scores and using a machine-learned text generation model, one or more bounding boxes with a highest weighted likelihood for the textual query.

A computing system comprising one or more processors and one or more non-transitory computer-readable media that store computer-readable instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:

• obtaining, by the computing system, a video comprising a plurality of image frames, wherein the video depicts a plurality of objects;
• respectively processing, by the computing system, each image frame with a machine-learned object detection model to extract one or more sets of feature data respectively for one or more of the plurality of objects that are depicted in the image frame;
• processing, by the computing system, the sets of feature data extracted for the plurality of image frames with a machine-learned tracking model to generate a plurality of trajectories respectively for the plurality of objects, wherein the trajectory for each object identifies the sets of feature data that correspond to the object;
• respectively processing, by the computing system, at least some of the respective sets of feature data that correspond to each object with a machine-learned text generation model to generate a textual caption for each object;
• and
• providing, by the computing system, the textual caption for each object as an output.