# US 2021/0342594 A1 · Method and apparatus for sampling training data and computer server

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342594 A1
- Document ID: US20210342594A1
- Status: Published application
- Kind: A1
- Application number: 17/378,054
- Filed: 2021-07-16
- Published: 2021-11-04
- Earliest claimed: 2018-05-23
- Assignees: Beijing Tusen Weilai Technology Co., Ltd.
- Applicants: Beijing Tusen Weilai Technology Co., Ltd.
- Inventors: Zehao Huang; Naiyan Wang
- Web page: https://patentgrep.com/patent/US20210342594A1

## CPC

- G06N 3/08: Learning methods
- G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling}
- G06F 18/2413: based on distances to training or reference patterns
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 3/091: Active learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7747: {Organisation of the process, e.g. bagging or boosting}
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 2201/07: Target detection
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The present disclosure provides a method and an apparatus for sampling training data and a computer server. The method includes: inputting a video to a target detection model to obtain a detection result for each frame of image; inputting the detection results for all frames of images in the video to a target tracking model, to obtain a tracking result for each frame of image; and for each frame of image in the video: matching the detection result and the tracking result for the frame of image, and when the detection result and the tracking result for the frame of image are inconsistent with each other, determining the frame of image as a sample image to be marked, for which processing by the target detection model is not optimal.

## Claims

### Claim 1

A method for sampling training data, comprising:
inputting a video to a target detection model to obtain a detection result for each frame of image;
inputting the detection results for all frames of images in the video to a target tracking model, to obtain a tracking result for each frame of image;
for each frame of image in the video:
matching the detection result and the tracking result for the frame of image, and when the detection result and the tracking result for the frame of image are inconsistent with each other, determining the frame of image as a sample image to be annotated, for which processing by the target detection model is not optimal; and
retraining the target detection model.

### Claim 2 (depends on 1)

The method of claim 1, wherein said matching the detection result and the tracking result for the frame of image comprises:
matching the detection result and the tracking result for the frame of image by using a bipartite graph matching algorithm.

### Claim 3 (depends on 2)

The method of claim 2, wherein the bipartite graph matching algorithm comprises Hungarian matching algorithm.

### Claim 4 (depends on 1)

The method of claim 1, further comprising, after the matching for all the frames of images in the video has completed:
sampling the determined sample images using a temporal non-maximum suppression algorithm.

### Claim 5 (depends on 4)

The method of claim 4, wherein, the temporal non-maximum suppression algorithm uses a non-maximum value N, where N is an integer, and wherein the sampling the determined sample images includes obtaining a frame from N successive frames.

### Claim 6 (depends on 1)

The method of claim 1, wherein the tracking result for a frame of image is obtained from a detection result for a previous frame of image.

### Claim 7 (depends on 1)

The method of claim 1, wherein for each frame of image, each detection target and each tracking target may be matched in a traversal manner.

### Claim 8 (depends on 1)

The method of claim 1, wherein the video comprises a plurality of neighboring frames of images that are substantially the same, wherein each of neighboring frames comprises frames of image determined as sample images to be annotated.

### Claim 9

An apparatus for sampling training data, comprising:
a target detecting unit configured to input a video to a target detection model to obtain a detection result for each frame of image;
a target tracking unit configured to input the detection results for all frames of images in the video to a target tracking model, to obtain a tracking result for each frame of image; and
a determining unit configured to, for each frame of image in the video:
match the detection result and the tracking result for the frame of image, and when the detection result and the tracking result for the frame of image are inconsistent with each other, determine the frame of image as a sample image to be annotated, for which processing by the target detection model is not optimal, wherein the frames of images in the video are in a chronological order, wherein the video comprises a plurality of neighboring frames of images that are substantially the same, and each of neighboring frames comprises frames of image determined as sample images to be annotated.

### Claim 10 (depends on 9)

The apparatus of claim 9, wherein the determining unit being configured to match the detection result and the tracking result for the frame of image comprises the determining unit being configured to:
match the detection result and the tracking result for the frame of image by using a bipartite graph matching algorithm.

### Claim 11 (depends on 10)

The apparatus of claim 10, wherein the bipartite graph matching algorithm comprises Hungarian matching algorithm.

### Claim 12 (depends on 9)

The apparatus of claim 9, further comprising a sampling unit, wherein
the determining unit is further configured to trigger the sampling unit after the matching for all the frames of images in the video has completed, and
the sampling unit is configured to sample the sample images determined by the determining unit by using a temporal non-maximum suppression algorithm.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein, the temporal non-maximum suppression algorithm uses a non-maximum value N, where N is an integer, and wherein the sampling the determined sample images includes obtaining a frame from N successive frames.

### Claim 14 (depends on 9)

The apparatus of claim 9, wherein the frames of images in the video are in a chronological order.

### Claim 15 (depends on 9)

The apparatus of claim 9, wherein the tracking result for a frame of image is obtained from a detection result for a previous frame of image.

### Claim 16

A computer server, comprising a memory and one or more processors communicatively connected to the memory, the memory storing instructions executable by the one or more processors, the instructions, when executed by the one or more processors, causing the one or more processors to perform the method comprising:
inputting a video to a target detection model to obtain a detection result for each frame of image;
inputting the detection results for all frames of images in the video to a target tracking model, to obtain a tracking result for each frame of image; and
for each frame of image in the video:
matching the detection result and the tracking result for the frame of image, and when the detection result and the tracking result for the frame of image are inconsistent with each other, determining the frame of image as a sample image to be annotated, for which processing by the target detection model is not optimal,
wherein the video comprises a plurality of neighboring frames of images that are substantially the same, and each of neighboring frames comprises frames of image determined as sample images to be annotated.

### Claim 17 (depends on 16)

The computer server of claim 16, wherein the frames of images in the video are in a chronological order.

### Claim 18 (depends on 16)

The computer server of claim 16, wherein the tracking result for a frame of image is obtained from a detection result for a previous frame of image.

### Claim 19 (depends on 16)

The computer server of claim 16, wherein for each frame of image, each detection target and each tracking target may be matched in a traversal manner.

### Claim 20 (depends on 16)

The computer server of claim 16, wherein the method comprises retraining the target detection model.
