Object in-painting in a video stream

One or more non-transitory computer readable media having instructions stored thereon for performing a method, the method comprising:

• determining a bounding box for an object represented in a designated frame of a live video stream, the live video stream being associated with two or more object tracks tracking two or more objects across successive frames in the live video stream;
• applying an appearance-based object tracking model to the designated frame to determine a first correspondence between the bounding box and two or more comparison bounding boxes in the two or more object tracks;
• applying a motion-based object tracking model to the designated frame to determine a second correspondence between the bounding and the two or more comparison bounding boxes in the two or more object tracks;
• determining a plurality of performance metrics for the appearance-based object tracking model and the motion-based object tracking model based at least in part on one or more visual features within the designated frame;
• selecting an object track of the two or more object tracks based on the first correspondence, the second correspondence, and the plurality of performance metrics, the object tracking being associated with a first object identifier of a plurality of object identifiers;
• determining whether to in-paint the bounding box based on the first object identifier;
• and
• upon determining to in-paint the bounding box, updating the live video stream to replace the designated frame with a replacement frame, the replacement frame replacing a portion of the designated frame corresponding with the bounding box with the replacement frame.

A system including a processor and memory, the system configured to perform a method comprising:

• determining a bounding box for an object represented in a designated frame of a live video stream, the live video stream being associated with two or more object tracks tracking two or more objects across successive frames in the live video stream;
• applying an appearance-based object tracking model to the designated frame to determine a first correspondence between the bounding box and two or more comparison bounding boxes in the two or more object tracks;
• applying a motion-based object tracking model to the designated frame to determine a second correspondence between the bounding and the two or more comparison bounding boxes in the two or more object tracks;
• determining a plurality of performance metrics for the appearance-based object tracking model and the motion-based object tracking model based at least in part on one or more visual features within the designated frame;
• selecting an object track of the two or more object tracks based on the first correspondence, the second correspondence, and the plurality of performance metrics, the object tracking being associated with a first object identifier of a plurality of object identifiers;
• determining whether to in-paint the bounding box based on the first object identifier;
• and
• upon determining to in-paint the bounding box, updating the live video stream to replace the designated frame with a replacement frame, the replacement frame replacing a portion of the designated frame corresponding with the bounding box with the replacement frame.