# US 2017/0255832 A1 · Method and System for Detecting Actions in Videos

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0255832 A1
- Document ID: US20170255832A1
- Status: Published application
- Kind: A1
- Application number: 15/058,264
- Filed: 2016-03-02
- Published: 2017-09-07
- Earliest claimed: 2016-03-02
- Assignees: Mitsubishi Electric Research Laboratories, Inc.
- Applicants: Mitsubishi Electric Research Laboratories, Inc.
- Inventors: Michael J. Jones; Tim Marks; Oncel Tuzel; Bharat Singh
- Web page: https://patentgrep.com/patent/US20170255832A1

## CPC

- G06N 3/08: Learning methods
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/30232: Surveillance
- G06V 2201/06: Recognition of objects for industrial automation
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method and system detects actions of an object in a scene by first acquiring a video of the scene as a sequence of images, wherein each image includes pixels, wherein the video is partitioned into chunks. The object in the video is tracked. For each object and each chunk of the video, trajectories of the pixels within a bounding box located over the object are tracked, and cropped trajectories and cropped images for one or more images in the chunk are produced using the bounding box. Then, the cropped trajectories and cropped images are passed to a recurrent neural network (RNN) that outputs a relative score for each action of interest.

## Claims

### Claim 1

A method for detecting actions of an object in a scene, comprising steps:
acquiring a video of the scene as a sequence of images, wherein each image includes pixels, wherein the video is partitioned into chunks;
tracking the object in the video, and for each object and each chunk of the video, further comprising:
determining trajectories of the pixels within a bounding box located over the object;
using the bounding box to produce cropped trajectories and cropped images for one or more images in the chunk; and
passing the cropped trajectories and cropped images to a recurrent neural network (RNN) that outputs a relative score for each action of interest, wherein the steps are performed in a processor.

### Claim 2 (depends on 1)

The method of claim 1, wherein the RNN includes convolutional neural network layers and one or more recurrent neural network layers.

### Claim 3 (depends on 2)

The method of claim 2, wherein the convolutional neural network layers operate on multiple streams, including the cropped trajectories and the cropped images as well as trajectories and images that have an entire spatial extent of the video.

### Claim 4 (depends on 2)

The method of claim 2, wherein the recurrent neural network layers include Long Short-Term Memory (LSTM) cells.

### Claim 5 (depends on 3)

The method of claim 3, wherein the recurrent neural network layers include bi-directional Long Short-Term Memory LSTM cells.

### Claim 6 (depends on 1)

The method of claim 1, wherein the trajectories are encoded as pixel trajectories.

### Claim 7 (depends on 1)

The method of claim 1, wherein the trajectories are encodes as stacked optical flow.

### Claim 8 (depends on 1)

The method of claim 1, wherein the tracking includes selecting a bounding box that maximizes a magnitude of the stacked optical flow inside the bounding box.

### Claim 9 (depends on 8)

The method of claim 8, wherein the tracking further comprises:
updating a location of the bounding box if a magnitude of the stacked optical flow inside the bounding box is greater than a threshold.

### Claim 10 (depends on 1)

The method of claim 1, wherein the trajectories for the pixels are determined from a central image in the chunk to each of K previous and K subsequent images.

### Claim 11 (depends on 10)

The method of claim 10, wherein K is 3.

### Claim 12 (depends on 10)

The method of claim 10, wherein a motion pattern for each pixel is determined using a 1×2K convolutional kernel.

### Claim 13 (depends on 1)

The method of claim 1, wherein the method is used for fine-grained action detection in the video.

### Claim 14 (depends on 1)

The method of claim 1, wherein the method includes training the RNN prior to the detecting.

### Claim 15 (depends on 1)

The method of claim 1, wherein the RNN has been previously trained.

### Claim 16 (depends on 1)

The method of claim 1, wherein the detecting comprises temporal action detection.

### Claim 17 (depends on 1)

The method of claim 1, wherein the detecting comprises spatio-temporal action detection.

### Claim 18 (depends on 1)

The method of claim 1, wherein the video is initially acquired in some form other than a sequence of images, and is converted to a sequence of images.

### Claim 19 (depends on 1)

The method of claim 1, in which the object is a person.

### Claim 20 (depends on 1)

The method of claim 1, in which the object is a robot.

### Claim 21 (depends on 1)

The method of claim 1, in which the object is an industrial robot.

### Claim 22

A system for detecting actions of an object in a scene, comprising:
means for acquiring a video of the scene as a sequence of images, wherein each image includes pixels, wherein the video is partitioned into chunks; and
a processor configured to track the object in the video, and for each object and each chunk of the video, and wherein the processor is further configured to, for each object and each chunk of the video, determine trajectories of the pixels within a bounding box located over the object, use the bounding box to produce cropped trajectories and cropped images for one or more images in the chunk, and pass the cropped trajectories and cropped images to a recurrent neural network (RNN) that outputs a relative score for each action of interest.
