Method and system for detecting actions in videos
AssigneeMitsubishi Electric Research Laboratories, Inc.
InventorsMichael J Jones, Tim Marks, Oncel Tuzel, Bharat Singh
Life of this patent
Estimating the term…
Abstract
A method and system detects actions of an object in a scene by first acquiring a video of the scene as a sequence of images, wherein each image includes pixels, wherein the video is partitioned into chunks. The object in the video is tracked. For each object and each chunk of the video, trajectories of the pixels within a bounding box located over the object are tracked, and cropped trajectories and cropped images for one or more images in the chunk are produced using the bounding box. Then, the cropped trajectories and cropped images are passed to a recurrent neural network (RNN) that outputs a relative score for each action of interest.
Claims19
A method for detecting actions of an object in a scene, comprising steps:
- acquiring a video of the scene as a sequence of images, wherein each image includes pixels, wherein the video is partitioned into chunks;
- tracking the object in the video, and for each object and each chunk of the video, further comprising:
- determining trajectories of the pixels within a bounding box located over the object, wherein the trajectories for the pixels are determined from a central image in the chunk to each of K previous and K subsequent images;
- using the bounding box to produce cropped trajectories and cropped images for one or more images in the chunk;
- and
- passing the cropped trajectories and cropped images to a recurrent neural network (RNN) that outputs a relative score for each action of interest, wherein the RNN includes convolutional neural network (CNN) layers and one or more recurrent neural network layers, wherein the steps are performed in a processor.
A system for detecting actions of an object in a scene, comprising:
- an input interface to acquire a video of the scene as a sequence of images from a video camera, wherein each image includes pixels, wherein the video is partitioned into chunks;
- and
- a processor configured to track the object in the video, and for each object and each chunk of the video further comprising:
- determine trajectories of the pixels within a bounding box located over the object, wherein the trajectories for the pixels are determined from a central image in the chunk to each of K previous and K subsequent images;
- use the bounding box to produce cropped trajectories and cropped images for one or more images in the chunk;
- and
- pass the cropped trajectories and cropped images to a recurrent neural network (RNN) that outputs a relative score for each action of interest, wherein the RNN includes convolutional neural network (CNN) layers and one or more recurrent neural network layers.
A method for detecting actions of an object in a scene, wherein the detecting includes detecting spatio-temporal action detection, comprising steps:
- acquiring a video of the scene as a sequence of images via an input interface, wherein each image includes pixels, wherein the video is partitioned into chunks;
- tracking the object in the video, and
- for each object and each chunk of the video, further comprising:
- determining trajectories of the pixels within a bounding box located over the object, wherein the trajectories for the pixels are determined from a central image in the chunk to each of K previous and K subsequent images;
- using the bounding box to produce cropped trajectories and cropped images for one or more images in the chunk;
- and
- passing the cropped trajectories and cropped images to a recurrent neural network (RNN) that outputs a relative score for each action of interest, wherein the RNN includes convolutional neural network (CNN) layers and one or more recurrent neural network layers, the CNN layers operate on multiple streams, including the cropped trajectories and the cropped images as well as trajectories and images that have an entire spatial extent of the video, wherein the steps are performed in a processor in communication with the input interface.
Dependencies are inferred from the claim text.