US 12,711,388 B2Granted

Aligning sequences by generating encoded representations of data items

AssigneeGDM Holding LLC

InventorsYusuf Aytar, Debidatta Dwibedi, Andrew Zisserman, Jonathan Tompson, Pierre Sermanet

Life of this patent

Estimating the term…

Loading patent…

Abstract

An encoder neural network is described which can encode a data item, such as a frame of a video, to form a respective encoded data item. Data items of a first data sequence are associated with respective data items of a second sequence, by determining which of the encoded data items of the second sequence is closest to the encoded data item produced from each data item of the first sequence. Thus, the two data sequences are aligned. The encoder neural network is trained automatically using a training set of data sequences, by an iterative process of successively increasing cycle consistency between pairs of the data sequences.

Claims10

Dependent claims

A method of aligning two video sequences of events using a single encoder neural network, a first video sequence of the two video sequences being a sequence of first video frames each comprising image data captured by at least one video camera, and a second video sequence of the two video sequences being a sequence of second video frames each comprising image data captured by at least one video camera, the sequence of first video frames depicting respective first events which occur in a first environment at successive first times, and the sequence of second video frames depicting respective second events which occur in a second environment at successive second times, the first video sequence depicting a first task and the second video sequence depicting a second task, the first task and the second task each having a plurality of corresponding phases that are performed in the same order in the first and second video sequences, the method comprising the steps of:

encoding the first video sequence using an encoder neural network to form, from each first video frame of the sequence of first video frames, a corresponding first encoded video frame;
attributing annotation data to one or more first video frames of the first video sequence, wherein the annotation data specifies a phase of the first task performed in an environment to which a video frame to which it is attributed relates;
encoding the second video sequence using the encoder neural network to form, from each second video frame of the sequence of second video frames, a corresponding second encoded video frame;
and
for at least one first video frame of the sequence of first video frames:
(i) for each of a plurality of the second video frames, determining a respective distance value indicative of a distance between the corresponding first encoded video frame and the corresponding second encoded video frame according to a distance measure;
(ii) determining one of the plurality of second video frames for which the corresponding distance value is lowest;
(iii) associating the first video frame and the determined one of the plurality of second video frames, to associate the corresponding first event with the corresponding second event, the first and second events being events of a same type;
(iv) based on the association of the first event and the second event, associating the annotation data with second video frames of the second video sequence;
and
(v) using the annotation data associated with the second video frames of the second video sequence as an input to a control program to determine that a first phase of the second task has been completed and to initiate a second phase of the second task.

8 more elements

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for aligning two video sequences of events using a single encoder neural network, a first video sequence of the two video sequences being a sequence of first video frames each comprising image data captured by at least one video camera, and a second video sequence of the two video sequences being a sequence of second video frames each comprising image data captured by at least one video camera, the sequence of first video frames depicting respective first events which occur in a first environment at successive first times, and the sequence of second video frames depicting respective second events which occur in a second environment at successive second times, the first video sequence depicting a first task and the second video sequence depicting a second task, the first task and the second task each having a plurality of corresponding phases that are performed in the same order in the first and second video sequences, the operations comprising:

encoding the first video sequence using an encoder neural network to form, from each first video frame of the sequence of first video frames, a corresponding first encoded video frame;
attributing annotation data to one or more first video frames of the first video sequence, wherein the annotation data specifies a phase of the first task performed in an environment to which a video frame to which it is attributed relates;
encoding the second video sequence using the encoder neural network to form, from each second video frame of the sequence of second video frames, a corresponding second encoded video frame;
and
for at least one first video frame of the sequence of first video frames:
(i) for each of a plurality of the second video frames, determining a respective distance value indicative of a distance between the corresponding first encoded video frame and the corresponding second encoded video frame according to a distance measure;
(ii) determining one of the plurality of second video frames for which the corresponding distance value is lowest;
(iii) associating the first video frame and the determined one of the plurality of second video frames, to associate the corresponding first event with the corresponding second event, the first and second events being events of a same type;
(iv) based on the association of the first event and the second event, associating the annotation data with second video frames of the second video sequence;
and
(v) using the annotation data associated with the second video frames of the second video sequence as an input to a control program to determine that a first phase of the second task has been completed and to initiate a second phase of the second task.

8 more elements

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for aligning two video sequences of events using a single encoder neural network, a first video sequence of the two video sequences being a sequence of first video frames each comprising image data captured by at least one video camera, and a second video sequence of the two video sequences being a sequence of second video frames each comprising image data captured by at least one video camera, the sequence of first video frames depicting respective first events which occur in a first environment at successive first times, and the sequence of second video frames depicting respective second events which occur in a second environment at successive second times, the first video sequence depicting a first task and the second video sequence depicting a second task, the first task and the second task each having a plurality of corresponding phases that are performed in the same order in the first and second video sequences, the operations comprising:

encoding the first video sequence using an encoder neural network to form, from each first video frame of the sequence of first video frames, a corresponding first encoded video frame;
attributing annotation data to one or more first video frames of the first video sequence, wherein the annotation data specifies a phase of the first task performed in an environment to which a video frame to which it is attributed relates;
encoding the second video sequence using the encoder neural network to form, from each second video frame of the sequence of second video frames, a corresponding second encoded video frame;
and
for at least one first video frame of the sequence of first video frames:
(i) for each of a plurality of the second video frames, determining a respective distance value indicative of a distance between the corresponding first encoded video frame and the corresponding second encoded video frame according to a distance measure;
(ii) determining one of the plurality of second video frames for which the corresponding distance value is lowest;
(iii) associating the first video frame and the determined one of the plurality of second video frames, to associate the corresponding first event with the corresponding second event, the first and second events being events of a same type;
(iv) based on the association of the first event and the second event, associating the annotation data with second video frames of the second video sequence;
and
(v) using the annotation data associated with the second video frames of the second video sequence as an input to a control program to determine that a first phase of the second task has been completed and to initiate a second phase of the second task.

8 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/295,286Filed 21 Nov 2019 · us provisional application
Priority: 21 Nov 2018
Published: US 2022/0004883 A16 Jan 2022
Granted: 18 Aug 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification29 codes

G06N 3/088
Learning methods: non-supervised learning, e.g. competitive learning
G11B 27/10
Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel: indexing; Addressing; Timing or synchronising;…
G06F 18/214
Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation: generating training…
G06F 18/24147
Based on distances to training or reference patterns: distances to closest patterns, e.g. nearest neighbour classification