# US 2021/0342924 A9 · Compressed content object and action detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342924 A9
- Document ID: US20210342924A9
- Status: Published application
- Kind: A9
- Application number: 16/728,714
- Filed: 2019-12-27
- Published: 2021-11-04
- Earliest claimed: 2017-11-20
- Assignees: A9.com, Inc.
- Applicants: A9.com, Inc.
- Inventors: R. Manmatha; Hexiang Hu; Deva Ramanan
- Web page: https://patentgrep.com/patent/US20210342924A9

## CPC

- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06F 3/04812: Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 16/7867: {using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings}
- G06F 16/951: Indexing; Web crawling techniques
- G06T 7/174: involving the use of two or more images
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Various embodiments of a framework which allow, as an alternative to resource-taxing decompression, efficient computation of feature maps using a compressed content data subset, such as video, by exploiting the motion information, such as a motion vector, present in the compressed video. This framework allows frame-specific object recognition and action detection algorithms to be applied to compressed video and other media files by executing only on I-frames in a Group of Pictures and linearly interpolating the results. Training and machine learning increases recognition accuracy. Yielding significant computational gains, this approach accelerates frame-wise feature extraction I-frame/P-frame/P-frame videos as well as I-frame/P-frame/B-frame videos. The present techniques may also be used for segmentation to identify and label respective regions for objects in a video.

## Claims

### Claim 1

A computer-implemented method, comprising:
receiving two or more compressed image frames comprising at least one complete image frame and at least one partial image frame;
using a neural network to extract a feature from the at least one complete image frame;
estimating a feature map for the at least one partial image frame;
using an iterative detector routine to detect features in the at least one partial image frame;
calculating a motion vector for a feature location in the at least one partial image frame based on the at least one complete image frame and the iterative detector routine; and
estimating the feature location in the at least one partial image frame using the calculated motion vector.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving compressed image frames for training the neural network prior to receiving the at least one complete image frame and the at least one partial image frame.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
executing one or more of an event detection module, action detection module, or video segmentation module on the feature map.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the at least one complete image frame consists of an Intracoded frame (I-frame), and the at least one partial image frame consists of a Predictive frame (P-frame) or a Bidirectional frame (B-frame).

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, further comprising:
applying an object detection network to detect the features in the at least one partial image frame.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein the object detection network generates one or more proposed locations within a bounding box and iteratively checks each of the one or more proposed locations to determine if the feature is present at the proposed location.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising:
using an existing motion vector to calculate the motion vector for the feature location in the at least one partial image frame.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, further comprising:
a training element for improving detection accuracy through machine learning.

### Claim 9

A computing system, comprising:
at least one processor; and
memory including instructions that, when executed by the at least one processor, cause the computing system to:
obtain compressed content including two or more compressed frames comprising at least one complete frame and at least one partial frame;
extract an object from the at least one complete frame using a neural network;
estimate an object map for the at least one partial frame;
detect objects in the at least one partial frame using an iterative detection algorithm;
use the at least one complete frame to calculate a motion estimation vector for the object in the at least one partial frame;
estimate the location of the object in the at least one partial frame using the calculated motion estimation vector;
propose the estimated location of the object for verification.

### Claim 10 (depends on 10)

The computing system of claim 10, wherein the instructions when executed further cause the computing system to:
obtain, prior to obtaining the compressed content, obtain compressed training content for training the neural network.

### Claim 11 (depends on 9)

The computing system of claim 9, wherein the instructions when executed further cause the computing system to:
execute at least one an event detection module, action detection module, or video segmentation module on top of the feature map.

### Claim 12 (depends on 9)

The computing system of claim 9, wherein the instructions when executed further cause the computing system to:
apply an object detection algorithm to detect the object in the at least one partial frame.

### Claim 13 (depends on 9)

The computing system of claim 9, wherein the instructions when executed further cause the computing system to:
apply an object detection algorithm that generates one or more potential object locations within a bounding box and iteratively checks each of the one or more potential object locations to determine if the feature is present at each potential object location.

### Claim 14 (depends on 9)

The computing system of claim 9, wherein the instructions when executed further cause the computing system to:
use an existing motion vector compressed with the at least one complete frame to calculate the motion estimation vector for the object in the at least one partial frame.

### Claim 15

A method, comprising:
receiving a plurality of compressed key frames comprising at least one complete key frame and at least one partial key frame;
using a neural network with machine learning algorithms to extract a feature from the at least one complete key frame;
using the at least one complete key frame to estimate a feature map for the at least one partial key frame;
using an iterative feature detection algorithm to detect features in the at least one partial key frame;
calculating a motion vector for a feature location in the at least one partial key frame based on the at least one complete key frame and the feature map; and
estimating the feature location in the at least one partial key frame using the calculated motion vector.

### Claim 16 (depends on 15)

The method of claim 15, further comprising:
using the feature map to execute at least one of an event detection module, action detection module, or video segmentation module.

### Claim 17 (depends on 15)

The method of claim 15, wherein the neural network with machine learning algorithms has been previously trained using compressed training key frames for improved feature detection accuracy.

### Claim 18 (depends on 15)

The method of claim 15, further comprising:
applying an object detection network to detect the features in the at least one partial key frame.

### Claim 19 (depends on 15)

The method of claim 15, further comprising:
generating one or more proposed feature locations within a bounding box and iteratively checking each of the one or more proposed feature locations to determine if the feature is present at the each proposed feature location.

### Claim 20 (depends on 15)

The method of claim 15, further comprising:
using an existing motion vector compressed with the at least one complete key frame to calculate the motion vector for the feature location in the at least one partial key frame.
