US 11,106,947 B2 · System and method of classifying an action or event

Abstract

A method of classifying an action or event using an artificial neural network. The method comprises obtaining a first and a second plurality of feature responses, corresponding to point data in a first channel and a second channel respectively. Each of the first and second plurality of feature responses have associated temporal and spatial position values, the first and second plurality of feature responses relating to a plurality of objects. The method also comprises generating a third plurality of feature responses based on one of the first plurality of feature responses and one of the second plurality of feature responses, and a weighted combination of associated temporal and spatial position values of the corresponding one of the first and second plurality of feature responses; and classifying an action or event relating to the objects using the artificial neural network based on the third plurality of feature responses.

Drawings

Drawings are temporarily unavailable. Open the PDF at USPTO

Drawings

Claims16

Dependent claims

A method of classifying an action or event relating to a plurality of objects in a scene of video data using an artificial neural network, the method comprising:

obtaining a first plurality of feature responses corresponding to point data in a first channel and a second plurality of feature responses corresponding to point data in a second channel, each feature response of the first and second plurality of feature responses comprising a temporal position value that indicates a temporal position in the video data, a spatial position value that indicates a spatial position of at least one object among the plurality of objects at the temporal position value, and a response value derived at a node in a predetermined layer of the artificial neural network, the first and second plurality of feature responses relating to the plurality of objects in the scene of the video data and having associated temporal and spatial position values;
generating a third plurality of feature responses comprising a temporal position value that indicates a temporal position in the video data, a spatial position value that indicates a spatial position of at least one object among the plurality of objects at the temporal position value, and a response value derived at a node in a layer subsequent to the predetermined layer of the artificial neural network, each of the third plurality of feature responses being generated based on calculation in the layer subsequent to the predetermined layer of the artificial neural network, the calculation being performed using one of the first plurality of feature responses from the first channel, inputted to the layer subsequent to the predetermined layer of the artificial neural network, and one of the second plurality of feature responses from the second channel, inputted to the layer subsequent to the predetermined layer of the artificial neural network, and a weighted combination of associated temporal and spatial position values which correspond to one of the received first and second plurality of feature responses;
and
classifying an action or event relating to the plurality of objects in the scene using the artificial neural network based on the generated third plurality of feature responses.

1 more element

A non-transitory computer readable storage medium storing program instructions for classifying an action or event relating to a plurality of objects in a scene of video data using an artificial neural network, the program comprising:

code for obtaining a first plurality of feature responses corresponding to point data in a first channel and a second plurality of feature responses corresponding to point data in a second channel, each feature response of the first and second plurality of feature responses comprising a temporal position value that indicates a temporal position in the video data, a spatial position value that indicates a spatial position of at least one object among the plurality of objects at the temporal position value, and a response value derived at a node in a predetermined layer of the artificial neural network, the first and second plurality of feature responses relating to the plurality of objects in the scene of the video data and having associated temporal and spatial position values;
code for generating a third plurality of feature responses comprising a temporal position value that indicates a temporal position in the video data, a spatial position value that indicates a spatial position of at least one object among the plurality of objects at the temporal position value, and a response value derived at a node in a layer subsequent to the predetermined layer of the artificial neural network, each of the third plurality of feature responses being generated based on calculation in the layer subsequent to the predetermined layer of the artificial neural network, the calculation being performed using one of the first plurality of feature responses from the first channel, inputted to the layer subsequent to the predetermined layer of the artificial neural network, and one of the second plurality of feature responses from the second channel, inputted to the layer subsequent to the predetermined layer of the artificial neural network, and a weighted combination of associated temporal and spatial position values which correspond to one of the received first and second plurality of feature responses;
and
code for classifying an action or event relating to the plurality of objects in the scene using the artificial neural network based on the generated third plurality of feature responses.

1 more element

Apparatus for classifying an action or event relating to a plurality of objects in a scene of video data, comprising:

one or more processors;
and
one or more memory devices storing a software program for directing the processor to:
obtain a first plurality of feature responses corresponding to point data in a first channel and a second plurality of feature responses corresponding to point data in a second channel, each feature response of the first and second plurality of feature responses comprising a temporal position value that indicates a temporal position in the video data, a spatial position value that indicates a spatial position of at least one object among the plurality of objects at the temporal position value, and a response value derived at a node in a predetermined layer of the artificial neural network, the first and second plurality of feature responses relating to the plurality of objects in the scene of the video data and having associated temporal and spatial position values;
generate a third plurality of feature responses comprising a temporal position value that indicates a temporal position in the video data, a spatial position value that indicates a spatial position of at least one object among the plurality of objects at the temporal position value, and a response value derived at a node in a layer subsequent to the predetermined layer of the artificial neural network, each of the third plurality of feature responses being generated based on calculation in the layer subsequent to the predetermined layer of the artificial neural network, the calculation being performed using one of the first plurality of feature responses from the first channel, inputted to the layer subsequent to the predetermined layer of the artificial neural network, and one of the second plurality of feature responses from the second channel, inputted to the layer subsequent to the predetermined layer of the artificial neural network, and a weighted combination of associated temporal and spatial position values which correspond to one of the received first and second plurality of feature responses;
and
classify an action or event relating to the plurality of objects in the scene using the artificial neural network based on the generated third plurality of feature responses.

4 more elements

A system classifying an action or event relating to a plurality of objects in a scene of video data using an artificial neural network, the system comprising:

one or more processors;
and
one or more memory devices storing a software program for directing the processor to:
obtain a first plurality of feature responses corresponding to point data in a first channel and a second plurality of feature responses corresponding to point data in a second channel, each feature response of the first and second plurality of feature responses comprising a temporal position value that indicates a temporal position in the video data, a spatial position value that indicates a spatial position of at least one object among the plurality of objects at the temporal position value, and a response value derived at a node in a predetermined layer of the artificial neural network, the first and second plurality of feature responses relating to the plurality of objects in the scene of the video data and having associated temporal and spatial position values;
generate a third plurality of feature responses comprising a temporal position value that indicates a temporal position in the video data, a spatial position value that indicates a spatial position of at least one object among the plurality of objects at the temporal position value, and a response value derived at a node in a layer subsequent to the predetermined layer of the artificial neural network, each of the third plurality of feature responses being generated based on calculation in the layer subsequent to the predetermined layer of the artificial neural network, the calculation being performed using one of the first plurality of feature responses from the first channel, inputted to the layer subsequent to the predetermined layer of the artificial neural network, and one of the second plurality of feature responses from the second channel, inputted to the layer subsequent to the predetermined layer of the artificial neural network, and a weighted combination of associated temporal and spatial position values which correspond to one of the received first and second plurality of feature responses;
and
classifying an action or event relating to the plurality of objects in the scene using the artificial neural network based on the generated third plurality of feature responses.

4 more elements

Dependencies are inferred from the claim text.

Description10,580 words

Summary1,957 words

#The present invention relates to recognition of actions, events and objects, for example the recognition of human actions and events in sports, from video and other data such as tracking of player and ball positions. In particular the present invention describes a method for encoding features in an artificial neural network based on point data that includes a response value and values representing position in space and time.

#Recognition of human actions and events is typically performed using video data containing the actions. In addition, further data such as tracking information of the position of people and objects of interest, for example players and a ball in a game space may also be used to detect actions. Tracking information may be determined from video data, annotated by hand or collected by other means such as with a tracking recording system.

#Using an example of a ball sport, tracking data typically encodes the positions of players and objects as a set of coordinate values within a scene (such as a playing field), corresponding to positions at various points in time. Existing methods for classifying actions and events include artificial neural network methods. Existing artificial neural network methods are designed for operating on dense data encoded in an array or tensor, such as an array where the coordinates of the array encode position information. In typical artificial neural network methods, related information from nearby positions in the array can be processed together using convolution operators, and the convolution operator can be applied to various positions.

Brief description of the drawings11 figures

#One or more embodiments of the invention will now be described with reference to the following drawings, in which:

#FIG. 1 shows a schematic flow diagram illustrating a method of performing training and classification of the artificial neural network;

#FIG. 2 shows a schematic flow diagram illustrating a method of performing training of the artificial neural network as used in the method of FIG. 1;

Detailed description8,376 words

#A method and system for classifying data is described below.

#The present arrangements described relate to a method for classification of video and other point data. The methods described relate to allowing reduced memory usage for classification of video and related data, and effective classification using data encoded as point information. Point data is data where position values are encoded as numeric values.

#Classification of data involves applying a label or output for an instance of data, for example by applying the label for an action present in a video sequence. A set of instances of training data and labels are used to train a classification system such as an artificial neural network. The artificial neural network is trained to be able to perform classification of instances not present in the training data, as a result of learning a relationship between data and labels for the instances present in the training set. An example of an artificial neural network 900 is shown in FIG. 9.

Facts

Application

15/840,988Filed 13 Dec 2017

Published

US 2019/0180149 A113 Jun 2019

Granted

31 Aug 2021

Term ends

25 Aug 2038

Estimated · how

20 years from earliest known nonprovisional domestic filing plus recorded patent term adjustment, including 255 days of adjustment. Maintenance-fee lapses, terminal disclaimers and statutory extensions have not been checked. The continuity chain may be incomplete; this is not an enforceability determination.

Classification20 codes

G06T 7/246
Analysis of motion: using feature-based methods, e.g. the tracking of corners or segments
G06F 18/24
Analysing: classification techniques
G06N 3/045
Architecture, e.g. interconnection topology: combinations of networks
G06N 3/0464
Architecture, e.g. interconnection topology: convolutional networks [CNN, ConvNet]

Timeline

13 Dec 2017Application filedUS 15/840,988
5 Mar 2018Assigned to Canon Kabushiki Kaishaassignment of assignor's interest
13 Jun 2019PublishedUS 2019/0180149 A1
31 Aug 2021GrantedUS 11,106,947 B2
12 Oct 2026TodayWithin its term
25 Aug 2038Term endsEstimated, includes 255 days of adjustment

System and method of classifying an action or event

Life of this patent

System and method of classifying an action or event

Life of this patent

Life of this patent

Abstract

Drawings

Drawings

Claims16

Description10,580 words

Drawings

Classification20 codes

Life of this patent

Description10,580 words