# US 2021/0241460 A1 · Computer-readable recording medium having stored therein training program, training method, and information processing apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241460 A1
- Document ID: US20210241460A1
- Status: Published application
- Kind: A1
- Application number: 17/144,157
- Filed: 2021-01-08
- Published: 2021-08-05
- Earliest claimed: 2020-01-30
- Assignees: Fujitsu Limited; Riken; National Cancer Center
- Applicants: Fujitsu Limited; Riken; National Cancer Center
- Inventors: Akira SAKAI; Masaaki KOMATSU; Ai DOZEN
- Web page: https://patentgrep.com/patent/US20210241460A1

## CPC

- G06V 10/30: Noise filtering
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/11: Region-based segmentation
- G06T 2207/10132: Ultrasound image
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method includes: acquiring data including moving image obtained by photographing a target and annotation images each indicative of a region of the target in each of frame images in the moving image; executing a process using the data. The process includes: detecting the target in the frame images; inputting, to an auto-encoder, an image obtained by combining partial images including the target and peripheral region images of the target detected in a given number of preceding and succeeding second frame images in a time series of the moving image of a first frame image; inputting a partial image corresponding to the first frame image to a neural network performing a segmentation; updating parameter of the auto-encoder and the neural network based on a difference between an image obtained by combining images from the auto-encoder and the neural network and a partial image of the annotation image.

## Claims

### Claim 1

A non-transitory computer-readable recording medium having stored therein a training program that causes a computer to execute a process comprising:
acquiring training data including moving image data obtained by photographing a target and a plurality of annotation images each indicative of a region of the target in each of a plurality of frame images included in the moving image data; and
executing a training process using the training data, wherein
the training process comprises:
detecting the target included in the plurality of frame images;
inputting a combined image to an auto-encoder, the combined image being obtained by combining a plurality of partial images including the target and a plurality of peripheral region images of the target, the plurality of partial images and plurality of peripheral region images being detected in a given number of preceding and succeeding second frame images in a time series of the moving image data of a first frame image from among the plurality of frame images;
inputting a partial image, in the plurality of partial images, corresponding to the first frame image to a neural network that performs a segmentation process for an image; and
performing parameter update of the auto-encoder and the neural network, based on a difference between a combination output image obtained by combining an output image from the auto-encoder and an output image from the neural network and a partial image of the annotation image indicative of a region of the target in the first frame image.

### Claim 2 (depends on 1)

The non-transitory computer-readable recording medium according to claim 1, wherein
the training process comprises:
inputting a feature relating to the entire first frame image obtained by inputting the first frame image to a trained model, to an intermediate layer of the auto-encoder, the trained model being produced using a dataset of an image different from an image obtained by photographing the target and estimating a label for an input image.

### Claim 3 (depends on 1)

The non-transitory computer-readable recording medium according to claim 1, wherein
the target included in the plurality of frame images is detected using a trained object detection model that is generated using a dataset including a plurality of images obtained by photographing the target and that specifies a region of the target included in an input image.

### Claim 4 (depends on 1)

The non-transitory computer-readable recording medium according to claim 1, the process further comprising:
acquiring target data including target moving image data obtained by photographing an estimation target;
detecting the estimation target included in each of a plurality of target frame images included the target moving image data;
inputting, to the auto-encoder trained by the parameter update, a combined image obtained by combining a plurality of partial images including the estimation target and a plurality of peripheral region images of the estimation target detected in a given number of preceding and succeeding fourth frame images of a third frame image in a time series of the target moving image data;
inputting a partial image, in the plurality of partial images, corresponding to the third frame image to the neural network trained by the parameter update; and
outputting an image including a region estimated as the estimation target in the third frame image based on a combination output image obtained by combining an output image from the auto-encoder and an output image from the neural network and position information indicative of a position at which the partial image in the third frame image is cut out.

### Claim 5 (depends on 1)

The non-transitory computer-readable recording medium according to claim 1, wherein
each of the plurality of frame images is an ultrasonic image.

### Claim 6 (depends on 4)

The non-transitory computer-readable recording medium according to claim 4, wherein
each of the plurality of frame images and the plurality of target frame images is an ultrasonic image.

### Claim 7

A training method executed by a computer, the training method comprising:
acquiring training data including moving image data obtained by photographing a target and a plurality of annotation images each indicative of a region of the target in each of a plurality of frame images included in the moving image data; and
executing a training process using the training data, wherein
the training process comprises:
detecting the target included in the plurality of frame images;
inputting a combined image to an auto-encoder, the combined image being obtained by combining a plurality of partial images including the target and a plurality of peripheral region images of the target, the plurality of partial images and plurality of peripheral region images being detected in a given number of preceding and succeeding second frame images in a time series of the moving image data of a first frame image from among the plurality of frame images;
inputting a partial image, in the plurality of partial images, corresponding to the first frame image to a neural network that performs a segmentation process for an image; and
performing parameter update of the auto-encoder and the neural network, based on a difference between a combination output image obtained by combining an output image from the auto-encoder and an output image from the neural network and a partial image of the annotation image indicative of a region of the target in the first frame image.

### Claim 8 (depends on 7)

The training method according to claim 7, wherein
the training process comprises:
inputting a feature relating to the entire first frame image obtained by inputting the first frame image to a trained model, to an intermediate layer of the auto-encoder, the trained model being produced using a dataset of an image different from an image obtained by photographing the target and estimating a label for an input image.

### Claim 9 (depends on 7)

The training method according to claim 7, wherein
the target included in the plurality of frame images is detected using a trained object detection model that is generated using a dataset including a plurality of images obtained by photographing the target and that specifies a region of the target included in an input image.

### Claim 10 (depends on 7)

The training method according to claim 7, the process further comprising:
acquiring target data including target moving image data obtained by photographing an estimation target;
detecting the estimation target included in each of a plurality of target frame images included the target moving image data;
inputting, to the auto-encoder trained by the parameter update, a combined image obtained by combining a plurality of partial images including the estimation target and a plurality of peripheral region images of the estimation target detected in a given number of preceding and succeeding fourth frame images of a third frame image in a time series of the target moving image data;
inputting a partial image, in the plurality of partial images, corresponding to the third frame image to the neural network trained by the parameter update; and
outputting an image including a region estimated as the estimation target in the third frame image based on a combination output image obtained by combining an output image from the auto-encoder and an output image from the neural network and position information indicative of a position at which the partial image in the third frame image is cut out.

### Claim 11 (depends on 7)

The training method according to claim 7, wherein
each of the plurality of frame images is an ultrasonic image.

### Claim 12 (depends on 10)

The training method according to claim 10, wherein
each of the plurality of frame images and the plurality of target frame images is an ultrasonic image.

### Claim 13

An information processing apparatus, comprising:
a memory; and
a processor coupled to the memory, the processor being configured to:
acquire training data including moving image data obtained by photographing a target and a plurality of annotation images indicative of a region of the target in each of a plurality of frame images included in the moving image data; and
execute a training process of an auto-encoder and a neural network, which performs a segmentation process for an image, using the training data, wherein
the processor performs, in the training process:
detecting the target included in the plurality of frame images;
inputting a combined image to an auto-encoder, the combined image being obtained by combining a plurality of partial images including the target and a plurality of peripheral region images of the target, the plurality of partial images and plurality of peripheral region images being detected in a given number of preceding and succeeding second frame images in a time series of the moving image data of a first frame image from among the plurality of frame images;
inputting a partial image, in the plurality of partial images, corresponding to the first frame image to the neural network; and
performing parameter update of the auto-encoder and the neural network, based on a difference between a combination output image obtained by combining an output image from the auto-encoder and an output image from the neural network and a partial image of the annotation image indicative of a region of the target in the first frame image.

### Claim 14 (depends on 13)

The information processing apparatus according to claim 13, wherein,
in the training process, the processor inputs a feature relating to the entire first frame image obtained by inputting the first frame image to a trained model, to an intermediate layer of the auto-encoder, the trained model being produced using a dataset of an image different from an image obtained by photographing the target and estimates a label for an input image.

### Claim 15 (depends on 13)

The information processing apparatus according to claim 13, wherein
the target included in the plurality of frame images is detected using a trained object detection model, which is produced using a dataset including a plurality of images obtained by photographing the target and that specifies a region of the target included in an input image.

### Claim 16 (depends on 13)

The information processing apparatus according to claim 13, wherein the processor:
acquires target data including target moving image data obtained by photographing an estimation target;
detects an estimation target included in each of a plurality of target frame images included in the target moving image data;
inputs, to the auto-encoder trained by the parameter update, a combined image obtained by combining a plurality of partial images including the estimation target and a plurality of peripheral region images of the estimation target detected in a given number of preceding and succeeding fourth frame images of a third frame image in a time series of the target moving image data;
inputs a partial image, in the plurality of partial images, corresponding to the third frame image to the neural network trained by the parameter update; and
outputs an image including a region estimated as the estimation target in the third frame image based on a combination output image obtained by combining an output image from the auto-encoder and an output image from the neural network and position information indicative of a position at which the partial image in the third frame image is cut out.

### Claim 17 (depends on 13)

The information processing apparatus according to claim 13, wherein
each of the plurality of frame images is an ultrasonic image.

### Claim 18 (depends on 16)

The information processing apparatus according to claim 16, wherein
each of the plurality of frame images and the plurality of target frame images is an ultrasonic image.
