# US 10,482,329 B2 · Systems and methods for identifying activities and/or events in media contents based on object data and scene data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,482,329 B2
- Document ID: US10482329B2
- Status: Granted
- Kind: B2
- Application number: 15/908,617
- Filed: 2018-02-28
- Published: 2018-07-05
- Granted: 2019-11-19
- Earliest claimed: 2016-04-26
- Assignees: Disney Enterprises, Inc.
- Applicants: Disney Enterprises, Inc.
- Inventors: Zuxuan Wu; Yanwei Fu; Leonid Sigal
- Web page: https://patentgrep.com/patent/US10482329B2

## CPC

- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06T 7/62: of area, perimeter, diameter or volume
- G06T 7/90: Determination of colour characteristics
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 20/20: in augmented reality scenes
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 20/47: {Detecting features for summarising video content}
- G11B 27/102: {Programmed access in sequence to addressed parts of tracks of operating record carriers (access by moving the head G11B3/08, G11B5/54, G11B7/085, G11B21/022; by moving the record carrier G11B15/005, G11B17/005, by driving of both record carrier and head G11B15/1816)}
- H04L 65/61: for supporting one-way streaming services, e.g. Internet radio
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06V 20/44: {Event detection}
- G06V 2201/07: Target detection
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

There is provided a system including a non-transitory memory storing an executable code and a hardware processor executing the executable code to receive a plurality of training contents depicting a plurality of activities, extract training object data from the plurality of training contents including a first training object data corresponding to a first activity, extract training scene data from the plurality of training contents including a first training scene data corresponding to the first activity, determine that a probability of the first activity is maximized when the first training object data and the first training scene data both exist in a sample media content.

## Claims

### Claim 1

A system comprising:
a non-transitory memory storing an executable code including an object data module, a scene data module, an image data module, and a plurality of fusion layers of a neural network; and
a hardware processor to:
receive a media content having video frames;
extract object data, by executing the object data module, from the video frames of the media content;
generate an average object data representation of the extracted object data;
extract scene data, by executing the scene data module, from the frames of the video media content;
generate an average scene data representation of the extracted scene data;
extract image data, by executing the image data module, from the frames of the video media content;
generate an average image data representation of the extracted image data;
feed the average object data representation, the average scene data representation and the average image data representation to the plurality of fusion layers of the neural network; and
classify an action in the video frames of the media content by executing the plurality of fusion layers using the average object data representation, the average scene data representation and the average image data representation.

### Claim 2 (depends on 1)

The system of claim 1, wherein the media content includes at least one of object annotations, scene annotations, and activity annotations.

### Claim 3 (depends on 1)

The system of claim 1, wherein first object data includes at least one of a color of the object, a shape of the object, a size of the object, and a relative size of the object.

### Claim 4 (depends on 1)

The system of claim 1, wherein the scene data includes at least one of a location of the scene, a lighting of the scene and an identifiable structure of the scene.

### Claim 5 (depends on 1)

The system of claim 1, wherein the image data includes at least one of a color of the image and a texture of the image.

### Claim 6 (depends on 1)

The system of claim 1, wherein the plurality of fusion layers include a first layer having a first number of object data neurons, a second number of scene data neurons, and a third number of image data neurons.

### Claim 7 (depends on 6)

The system of claim 6, wherein each of the first number and the third number is higher than the second number.

### Claim 8 (depends on 6)

The system of claim 6, wherein the plurality of fusion layers include a second layer having a fourth number of neurons across the first layer.

### Claim 9 (depends on 8)

The system of claim 8, wherein the plurality of fusion layers include a classifier layer across the second layer.

### Claim 10

A method for use with a system including a hardware processor and a non-transitory memory storing an executable code including an object data module, a scene data module, an image data module, and a plurality of fusion layers of a neural network, the method comprising:
receiving, using the hardware processor, a media content having video frames;
extracting object data, by executing the object data module using the hardware processor, from the video frames of the media content;
generating, using the hardware processor, an average object data representation of the extracted object data;
extracting scene data, by executing the scene data module using the hardware processor, from the frames of the video media content;
generating, using the hardware processor, an average scene data representation of the extracted scene data;
extracting image data, by executing the image data module using the hardware processor, from the frames of the video media content;
generating, using the hardware processor, an average image data representation of the extracted image data;
feeding, using the hardware processor, the average object data representation, the average scene data representation and the average image data representation to the plurality of fusion layers of the neural network; and
classifying, by executing the plurality of fusion layers using the hardware processor, an action in the video frames of the media content using the average object data representation, the average scene data representation and the average image data representation.

### Claim 11 (depends on 10)

The method of claim 10, wherein the media content includes at least one of object annotations, scene annotations, and activity annotations.

### Claim 12 (depends on 10)

The method of claim 10, wherein first object data includes at least one of a color of the object, a shape of the object, a size of the object, and a relative size of the object.

### Claim 13 (depends on 10)

The method of claim 10, wherein the scene data includes at least one of a location of the scene, a lighting of the scene and an identifiable structure of the scene.

### Claim 14 (depends on 10)

The method of claim 10, wherein the image data includes at least one of a color of the image and a texture of the image.

### Claim 15 (depends on 10)

The method of claim 10, wherein the plurality of fusion layers include a first layer having a first number of object data neurons, a second number of scene data neurons, and a third number of image data neurons.

### Claim 16 (depends on 15)

The method of claim 15, wherein each of the first number and the third number is higher than the second number.

### Claim 17 (depends on 15)

The method of claim 15, wherein the plurality of fusion layers include a second layer having a fourth number of neurons across the first layer.

### Claim 18 (depends on 17)

The method of claim 17, wherein the plurality of fusion layers include a classifier layer across the second layer.