Correlating videos and sentences

A method of providing a description of a video, the method comprising automatically performing the following steps using a processor:

• detecting one or more candidate object(s) in each of a plurality of frames of the video using image data of the plurality of frames;
• generating a candidate description including one or more participant(s) and one or more condition(s) applied to the participant(s), the conditions and participants selected from a linguistic model;
• constructing a plurality of respective component lattices corresponding to the participant(s) or condition(s), wherein at least one of the component lattices includes a node corresponding to one of the candidate objects detected in one of the frames of the video;
• producing an aggregate lattice having a plurality of nodes, each of which includes a respective factor computed from corresponding nodes in a respective plurality of corresponding ones of the component lattices;
• determining a score for the video with respect to the candidate description by determining an aggregate score for a path through the aggregate lattice;
• if the aggregate score does not satisfy a termination condition, altering the candidate description by adding to it one or more participant(s) or condition(s) selected from the linguistic model, and repeating the constructing, producing, and determining steps with respect to the altered candidate description.

A method of testing a video against an aggregate query, the video including image data of each of a plurality of frames, the method comprising automatically performing the following steps using a processor:

• receiving an aggregate query defining one or more participant(s) and one or more condition(s) with respect to the participant(s);
• detecting one or more candidate object(s) in each of the plurality of frames of the video;
• providing a unified cost function using the detected candidate object(s) for computing how closely an input combination of the candidate object(s) corresponds to one or more object track(s) and how closely the corresponding one or more object track(s) correspond to the participant(s) and the condition(s);
• determining whether the video corresponds to the aggregate query by mathematically optimizing the unified cost function to select a combination of the detected candidate object(s) that has an aggregate cost with respect to the participant(s) and the condition(s).