# US 2021/0241031 A1 · Method, device, and program for sampling frame image of object to be learned in video for artificial intelligence video learning, and image learning method thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241031 A1
- Document ID: US20210241031A1
- Status: Published application
- Kind: A1
- Application number: 17/212,771
- Filed: 2021-03-25
- Published: 2021-08-05
- Earliest claimed: 2018-09-27
- Assignees: Crowdworks, Inc.
- Applicants: Crowdworks Inc.
- Inventors: Min Woo PARK
- Web page: https://patentgrep.com/patent/US20210241031A1

## CPC

- G06N 20/00: Machine learning
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 5/04: Inference or reasoning models
- G06T 7/194: involving foreground-background segmentation
- G06T 7/215: Motion-based segmentation
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06N 3/045: Combinations of networks
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06V 2201/07: Target detection
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Provided is a method, device, and program for sampling a frame image of an object to be learned in a video for artificial intelligence video learning, and an image learning method thereof. The method includes receiving a raw video for the AI image learning, extracting a predetermined number of frame images from the received raw, detecting learning target objects in each of the frame images, removing a background other than the learning target objects from each of the frame images, measuring a movement amount of each of the detected learning target objects in a n-th frame image, from which the background is removed, and selecting the n-th frame image as the learning target frame image, by comparing a result of measuring the movement amount of each of the detected one or more learning target objects in the n-th frame image with a predetermined reference.

## Claims

### Claim 1

A method of sampling a learning target frame image in a video for artificial intelligence (AI) image learning that is performed by a computer, the method comprising:
receiving a raw video for the AI image learning;
extracting a predetermined number of frame images from the received raw video per predetermined time interval;
detecting one or more learning target objects, each of which has one or more predetermined types, in each of the frame images by using an object detection algorithm;
removing a background other than the learning target objects from each of the frame images;
measuring a movement amount of each of the detected one or more learning target objects in a n-th frame image (′n′ being a natural number of 2 or more) by comparing a location of each of the detected one or more learning target objects in the n-th frame image, from which the background is removed, with a location of each of the detected one or more learning target objects in a (n−1)-th frame image, from which the background is removed and which is immediately before the n-th frame image; and
selecting the n-th frame image as the learning target frame image, by comparing a result of measuring the movement amount of each of the detected one or more learning target objects in the n-th frame image with a predetermined reference.

### Claim 2 (depends on 1)

The method of claim 1, wherein the selecting of the n-th frame image as the learning target frame image includes:
when a movement amount of each of the learning target objects, of which the number is not less than a predetermined number, from among the detected one or more learning target objects is not less than the predetermined reference, selecting the n-th frame image as the learning target frame image; and
otherwise, not selecting the n-th frame image as the learning target frame image.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
generating a learning video set by processing the selected learning target frame image; and
performing the AI image learning by using the generated learning video set.

### Claim 4 (depends on 3)

The method of claim 3, wherein the learning target frame image in the generating of the learning video set is a frame image before the one or more learning target objects are detected in the extracting of the predetermined number of the frame images, and
wherein the generating of the learning video set includes:
detecting the one or more learning target objects, each of which has the one or more predetermined types, in the learning target frame image; and
annotating the detected one or more learning target objects in the learning target frame image.

### Claim 5 (depends on 3)

The method of claim 3, wherein the learning target frame image in the generating of the learning video set is a frame image, in which the one or more learning target objects are detected in the detecting of the one or more learning target objects, and
wherein the generating of the learning video set includes:
annotating the detected one or more learning target objects in the learning target frame image.

### Claim 6

A device for sampling a learning target frame image in a video for AI image learning, the device comprising:
a raw video receiving unit configured to receive a raw video for the AI image learning;
a frame image extracting unit configured to extract a predetermined number of frame images from the received raw video per predetermined time interval;
a learning target object detecting unit configured to detect one or more learning target objects, each of which has one or more predetermined types, in each of the frame images by using an object detection algorithm;
a background removing unit configured to remove a background other than the learning target objects from each of the frame images;
an object movement amount measuring unit configured to measure a movement amount of each of the detected one or more learning target objects in a n-th frame image (′n′ being a natural number of 2 or more) by comparing a location of each of the detected one or more learning target objects in the n-th frame image, from which the background is removed, with a location of each of the detected one or more learning target objects in a (n−1)-th frame image, from which the background is removed and which is immediately before the n-th frame image; and
a learning target frame image selecting unit configured to select the n-th frame image as the learning target frame image, by comparing a result of measuring the movement amount of each of the detected one or more learning target objects in the n-th frame image with a predetermined reference.

### Claim 7 (depends on 6)

The device of claim 6, wherein, when a movement amount of each of the learning target objects, of which the number is not less than a predetermined number, from among the detected one or more learning target objects is not less than the predetermined reference, the learning target frame image selecting unit selects the n-th frame image as the learning target frame image, and
wherein, otherwise, the learning target frame image selecting unit selects does not select the n-th frame image as the learning target frame image.

### Claim 8 (depends on 6)

The device of claim 6, further comprising:
a learning video set generating unit configured to generate a learning video set by processing the selected learning target frame image; and
an AI image learning executing unit configured to perform the AI image learning by using the generated learning video set.

### Claim 9 (depends on 8)

The device of claim 8, wherein the learning target frame image is a frame image before the one or more learning target objects are detected, and
wherein the learning video set generating unit detects the one or more learning target objects, each of which has the one or more predetermined types, in the learning target frame image and annotates the detected one or more learning target objects in the learning target frame image.

### Claim 10 (depends on 8)

The device of claim 8, wherein the learning target frame image is a frame image, in which the one or more learning target objects are detected, and
wherein the learning video set generating unit annotates the detected one or more learning target objects in the learning target frame image.

### Claim 11

A program stored in a recording medium to perform a method for sampling a learning target frame image in a video for AI video learning, using a computer being hardware, the program for performing the method comprising:
receiving a raw video for the AI image learning;
extracting a predetermined number of frame images from the received raw video per predetermined time interval;
detecting one or more learning target objects, each of which has one or more predetermined types, in each of the frame images by using an object detection algorithm;
removing a background other than the learning target objects from each of the frame images;
measuring a movement amount of each of the detected one or more learning target objects in a n-th frame image (′n′ being a natural number of 2 or more) by comparing a location of each of the detected one or more learning target objects in the n-th frame image, from which the background is removed, with a location of each of the detected one or more learning target objects in a (n−1)-th frame image, from which the background is removed and which is immediately before the n-th frame image; and
selecting the n-th frame image as the learning target frame image, by comparing a result of measuring the movement amount of each of the detected one or more learning target objects in the n-th frame image with a predetermined reference.