# US 2021/0342593 A1 · Method and apparatus for detecting target in video, computing device, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342593 A1
- Document ID: US20210342593A1
- Status: Published application
- Kind: A1
- Application number: 17/374,971
- Filed: 2021-07-13
- Published: 2021-11-04
- Earliest claimed: 2019-05-21
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Zequn JIE
- Web page: https://patentgrep.com/patent/US20210342593A1

## CPC

- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/277: involving stochastic approaches, e.g. using Kalman filters
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30232: Surveillance
- G06T 2207/30241: Trajectory
- G06V 2201/07: Target detection
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

This application discloses a method for detecting a target in a video, performed by a computing device. The method includes applying a target detection operation to a first frame in the video, to determine a first target detection result of the target in the first frame; applying a target tracking operation to a second frame after the first frame in the video, to determine changes of the target between the first frame and the second frame; and determining a second target detection result of the target in the second frame according to the first target detection result and the changes of the target between the first frame and the second frame.

## Claims

### Claim 1

A method for detecting a target in a video, performed by a computing device, the method comprising:
applying a target detection operation to a first frame in the video, to determine a first target detection result of the target in the first frame;
applying a target tracking operation to a second frame after the first frame in the video, to determine changes of the target between the first frame and the second frame; and
determining a second target detection result of the target in the second frame according to the first target detection result and the changes of the target between the first frame and the second frame.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the applying a target tracking operation to a second frame after the first frame in the video, to determine changes of the target corresponding to the first target detection result and between the first frame and the second frame comprises:
extracting semantic information of a combination of the first frame and the second frame by using a first neural network, and outputting, according to the extracted semantic information, a result indicating the changes of the target corresponding to the first target detection result and between the first frame and the second frame.

### Claim 3 (depends on 2)

The method according to claim 2, wherein an output of the first neural network comprises at least two output sets, and each output set comprises at least one output map having the same size as the second frame; and
wherein the applying a target tracking operation to a second frame after the first frame, to determine changes of the target between the first frame and the second frame comprises:
selecting a corresponding output set according to a size of the target, to determine the changes of the target between the first frame and the second frame.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the changes of the target between the first frame and the second frame are changes in a position and a shape feature of the target between the first frame and the second frame.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the determining a second target detection result of the target in the second frame according to the first target detection result and the changes of the target between the first frame and the second frame comprises:
adjusting the first target detection result according to the changes of the target between the first frame and the second frame, to determine the second target detection result.

### Claim 6 (depends on 1)

The method according to claim 1, further comprising:
applying the target tracking operation to a third frame after the second frame in the video, to determine changes of the target between the second frame and the third frame; and
determining a third target detection result of the target in the third frame according to the second target detection result and the changes of the target between the second frame and the third frame.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the applying a target detection operation to a first frame in the video, to determine a first target detection result for the first frame comprises:
performing the target tracking operation based on a frame before the first frame and the first frame, to determine a target tracking result of the target in the first frame; and
performing the target detection operation to a combination of the first frame and the target tracking result, to determine the first target detection result of the target in the first frame.

### Claim 8 (depends on 2)

The method according to claim 2, wherein the first neural network is trained by using the following operations:
determining a first training set required to train the first neural network, the first training set comprising a first training video, the first training video comprising a first training frame and a second training frame after the first training frame, and the first training frame and the second training frame comprising a to-be-detected training target;
determining a real change of the training target between the first training frame and the second training frame;
applying the target detection operation to the first training frame, to determine a first training detection result of the training target in the first training frame;
applying the target tracking operation to the second training frame by using the first neural network, to determine a first change of the training target between the first training frame and the second training frame;
determining a difference between the first change and the real change;
determining a second change of the training target between the first training frame and the second training frame according to an output of at least one intermediate layer of the first neural network;
determining a difference between the second change and the real change; and
adjusting a parameter of the first neural network to minimize the difference between the first change and the real change and the difference between the second change and the real change.

### Claim 9

An apparatus for detecting a target in a video, comprising a memory and a processor, the memory storing instructions, and the instructions, when executed by the processor, causing the processor to:
apply a target detection operation to a first frame in the video, to determine a first target detection result of the target in the first frame;
apply a target tracking operation to a second frame after the first frame in the video, to determine changes of the target between the first frame and the second frame; and
determine a second target detection result of the target in the second frame according to the first target detection result and the changes of the target between the first frame and the second frame.

### Claim 10 (depends on 9)

The apparatus according to claim 9, wherein the processor is further configured to:
extract semantic information of a combination of the first frame and the second frame by using a first neural network, and output, according to the extracted semantic information, a result indicating the changes of the target corresponding to the first target detection result and between the first frame and the second frame.

### Claim 11 (depends on 10)

The apparatus according to claim 10, wherein an output of the first neural network comprises at least two output sets, and each output set comprises at least one output map having the same size as the second frame, wherein
the processor is further configured to select a corresponding output set according to a size of the target, to determine the changes of the target between the first frame and the second frame.

### Claim 12 (depends on 9)

The apparatus according to claim 9, wherein the changes of the target between the first frame and the second frame are changes in a position and a shape feature of the target between the first frame and the second frame.

### Claim 13 (depends on 9)

The apparatus according to claim 9, wherein the processor is further configured to adjust the first target detection result according to the changes of the target between the first frame and the second frame, to determine the second target detection result.

### Claim 14 (depends on 9)

The apparatus according to claim 9, wherein the processor is further configured to apply the target tracking operation to a third frame after the second frame, to determine changes of the target between the second frame and the third frame; and
determine a third target detection result of the target in the third frame according to the second target detection result and the changes of the target between the second frame and the third frame.

### Claim 15

A non-transitory computer-readable storage medium, storing instructions, the instructions, when executed by a processor, causing the processor to perform the method for detecting a target, the method comprising:
applying a target detection operation to a first frame in the video, to determine a first target detection result of the target in the first frame;
applying a target tracking operation to a second frame after the first frame in the video, to determine changes of the target between the first frame and the second frame; and
determining a second target detection result of the target in the second frame according to the first target detection result and the changes of the target between the first frame and the second frame.

### Claim 16 (depends on 15)

The computer-readable storage medium according to claim 15, wherein the applying a target tracking operation to a second frame after the first frame in the video, to determine changes of the target corresponding to the first target detection result and between the first frame and the second frame comprises:
extracting semantic information of a combination of the first frame and the second frame by using a first neural network, and outputting, according to the extracted semantic information, a result indicating the changes of the target corresponding to the first target detection result and between the first frame and the second frame.

### Claim 17 (depends on 16)

The computer-readable storage medium according to claim 16, wherein an output of the first neural network comprises at least two output sets, and each output set comprises at least one output map having the same size as the second frame; and
Wherein the applying a target tracking operation to a second frame after the first frame, to determine changes of the target between the first frame and the second frame comprises:
selecting a corresponding output set according to a size of the target, to determine the changes of the target between the first frame and the second frame.

### Claim 18 (depends on 15)

The computer-readable storage medium according to claim 15, wherein the changes of the target between the first frame and the second frame are changes in a position and a shape feature of the target between the first frame and the second frame.

### Claim 19 (depends on 15)

The computer-readable storage medium according to claim 15, wherein the determining a second target detection result of the target in the second frame according to the first target detection result and the changes of the target between the first frame and the second frame comprises:
adjusting the first target detection result according to the changes of the target between the first frame and the second frame, to determine the second target detection result.

### Claim 20 (depends on 15)

The computer-readable storage medium according to claim 15, the method further comprising:
applying the target tracking operation to a third frame after the second frame in the video, to determine changes of the target between the second frame and the third frame; and
determining a third target detection result of the target in the third frame according to the second target detection result and the changes of the target between the second frame and the third frame.
