# US 11,908,177 B2 · Updated learning of feature extraction model that extracts moving image feature amount from moving image data and still image feature amount from still image data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,177 B2
- Document ID: US11908177B2
- Status: Granted
- Kind: B2
- Application number: 17/608,201
- Filed: 2019-05-29
- Published: 2022-06-23
- Granted: 2024-02-20
- Earliest claimed: 2019-05-29
- Assignees: NEC Corporation
- Applicants: NEC Corporation
- Inventors: Shuhei Yoshida; Makoto Terao
- Web page: https://patentgrep.com/patent/US11908177B2

## CPC

- G06V 10/776: Validation; Performance evaluation
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V 10/7747: {Organisation of the process, e.g. bagging or boosting}
- G06V 10/806: {of extracted features}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The learning device 10D is learned to extract moving image feature amount Fm which is feature amount relating to the moving image data Dm when the moving image data Dm is inputted thereto, and is learned to extract still image feature amount Fs which is feature amount relating to the still image data Ds when the still image data Ds is inputted thereto. The first inference unit 32D performs a first inference regarding the moving image data Dm based on the moving image feature amount Fm. The second inference unit 34D performs a second inference regarding the still image data Ds based on the still image feature amount Fs. The learning unit 36D performs learning of the feature extraction unit 31D based on the results of the first inference and the second inference.

## Claims

### Claim 1

A learning device comprising:
a processor; and
a memory storing instructions executable by the processor to:
extract a moving image feature amount from moving image data and a still image feature amount from still image data by inputting the moving image data and the still image data into a feature extraction model that is configured to extract the moving image feature amount and the still image feature image when provided with the moving image data and the still image data;
perform a first inference regarding the moving image data based on the moving image feature amount;
perform a second inference regarding the still image data based on the still image feature amount; and
perform updated learning of the feature extraction model based on results of the first inference and the second inference.

### Claim 2 (depends on 1)

The learning device according to claim 1,
wherein the feature extraction model extracts the still image feature amount by using a parameter that the feature extraction model also used to extract the moving image feature amount, and
wherein the instructions are executable by the processor to further update the parameter based on the results of the first inference and the second inference.

### Claim 3 (depends on 1)

The learning device according to claim 1,
wherein the feature extraction model includes:
an image feature conversion block configured to perform an image-based feature conversion, the image-based feature conversion being applied to the moving image data or a feature amount of the moving image data, and to the still image data or to a feature amount of the still image data;
a time series feature conversion block configured to perform a time series-based feature conversion, the time series-based feature conversion being applied to the moving image data or to the feature amount of the moving image data; and
a point feature conversion block configured to perform an image point-based feature conversion, the image point-based feature conversion being applied at least to the still image data or to the feature amount of the still image data.

### Claim 4 (depends on 3)

The learning device according to claim 3,
wherein the feature extraction model includes a layer structure in which layers, each of which includes the image feature conversion block, the time series feature conversion block, and the point feature conversion block, are stacked.

### Claim 5 (depends on 3)

The learning device according to claim 3,
wherein the time series feature conversion block converts the feature amount of the moving image data in a time series direction and in a channel direction, and
wherein the point feature conversion block converts the feature amount of the still image data so that a number of channels of the converted feature amount of the still image data is the same as the feature amount of the moving image data converted by the time series feature conversion block.

### Claim 6 (depends on 2)

The learning device according to claim 2,
wherein the program code is executable by the processor to further:
calculate a first inference loss for the first inference based on the result of the first inference and correct answer data corresponding to the first inference;
calculate a second inference loss for the second inference based on the result of the second inference and correct answer data corresponding to the second inference; and
update the parameter of the feature extraction model based on the first inference loss and the second inference loss.

### Claim 7 (depends on 1)

The learning device according to claim 1,
wherein the feature extraction model extracts, as the still image feature amount, a feature amount relating to an object that appears in both the moving image data and the still image data.

### Claim 8 (depends on 1)

The learning device according to claim 1,
wherein the processor performs the updated learning of a first inference model that performs the first inference, based on the result of the first inference and correct answer data corresponding to the moving image data, and
wherein the processor performs the updated learning of a second inference model that performs the second inference, based on the result of the second inference and correct answer data corresponding to the still image data.

### Claim 9 (depends on 1)

The learning device according to claim 1,
wherein the processor performs the first inference based on the moving image feature amount and information relating to the second inference.

### Claim 10

A control method executed by a learning device, the control method comprising:
performing extraction of a moving image feature amount from moving image data and a still image feature amount from still image data by inputting the moving image data and the still image data into a feature extraction model that is configured to extract the moving image feature amount and the still image feature image when provided with the moving image data and the still image data;
performing a first inference regarding the moving image data based on the moving image feature amount;
performing a second inference regarding the still image data based on the still image feature amount; and
performing updating learning of the feature extraction model based on results of the first inference and the second inference.