# US 2023/0410249 A1 · Image processing method and apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410249 A1
- Document ID: US20230410249A1
- Status: Published application
- Kind: A1
- Application number: 18/457,499
- Filed: 2023-08-29
- Published: 2023-12-21
- Earliest claimed: 2020-11-24
- Assignees: Korea Advanced Institute of Science and Technology
- Applicants: Korea Advanced Institute of Science and Technology
- Inventors: Junyong Noh; Sanghun Park; Kwanggyoon Seo
- Web page: https://patentgrep.com/patent/US20230410249A1

## CPC

- G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403)
- G06T 11/00: Two-dimensional [2D] image generation
- G06N 3/045: Combinations of networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06T 3/4046: using neural networks
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Disclosed is an image processing method and apparatus. The image processing method includes extracting a content latent code and a style latent code for each of a plurality of input images, obtaining a content feature vector by calculating a weighted sum of content latent codes extracted from the input images based on a morphing control parameter, obtaining a style feature vector by calculating a weighted sum of style latent codes extracted from the input images based on the morphing control parameter, and generating a morphing image based on the content feature vector and the style feature vector.

## Claims

### Claim 1

A training method, comprising:
obtaining training data comprising a first image, a second image, and a ground truth morphing image that is based on a morphing control parameter;
obtaining an output morphing image by inputting the first image, the second image, and the morphing control parameter to a morphing generator; and
training the morphing generator based on a loss function that is based on at least one of the images comprised in the training data and the output morphing image,
wherein the loss function is determined based on at least one of:
an adversarial loss associated with a discrimination between the image comprised in the training data and the output morphing image obtained from the morphing generator; or
a pixel-wise reconstruction loss associated with a difference between pixels of the image comprised in the training data and pixels of the output morphing image.

### Claim 2 (depends on 1)

The training method of claim 1, wherein the adversarial loss comprises a conditional adversarial loss in which class label information of one of the images comprised in the training data is set as a condition.

### Claim 3 (depends on 1)

The training method of claim 1, wherein the adversarial loss is determined based on the morphing control parameter, an adversarial loss associated with a discrimination between the first image and the output morphing image, and an adversarial loss associated with a discrimination between the second image and the output morphing image, and
the pixel-wise reconstruction loss is determined based on the ground truth morphing image corresponding to the morphing control parameter among the images comprised in the training data and on the output morphing image.

### Claim 4 (depends on 1)

The training method of claim 1, wherein the morphing control parameter comprises a content morphing control parameter and a style morphing control parameter,
wherein the obtaining of the output morphing image comprises:
obtaining a first output morphing image comprising a content of the second image and a style of the first image; and
obtaining a second output morphing image comprising a content of the first image and a style of the second image,
wherein the training of the morphing generator comprises:
training the morphing generator based on an adversarial loss associated with a discrimination between the first output morphing image and the first image, and an adversarial loss associated with a discrimination between the second output morphing image and the second image.

### Claim 5 (depends on 4)

The training method of claim 4, wherein the training of the morphing generator comprises:
obtaining a third morphing image comprising a content of the second output morphing image and the style of the second image by inputting the second output morphing image and the second image to the morphing generator;
obtaining a fourth morphing image comprising a content of the first output morphing image and the style of the first image by inputting the first output morphing image and the first image to the morphing generator;
training the morphing generator based on an adversarial loss and a pixel-wise reconstruction loss that are based on the third output morphing image and the second image; and
training the morphing generator based on an adversarial loss and a pixel-wise reconstruction loss that are based on the fourth output morphing image and the first image.

### Claim 6 (depends on 1)

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the image processing method of claim 1.

### Claim 7

An image processing apparatus, comprising:
at least one processor;
wherein, the processor is configured to:
obtain training data comprising a first image, a second image, and a ground truth morphing image that is based on a morphing control parameter;
obtain an output morphing image by inputting the first image, the second image, and the morphing control parameter to a morphing generator; and
train the morphing generator based on a loss function that is based on at least one of the images comprised in the training data and the output morphing image,
wherein the loss function is determined based on at least one of:
an adversarial loss associated with a discrimination between the image comprised in the training data and the output morphing image obtained from the morphing generator; or
a pixel-wise reconstruction loss associated with a difference between pixels of the image comprised in the training data and pixels of the output morphing image.

### Claim 8 (depends on 7)

The image processing apparatus of claim 7, wherein the adversarial loss comprises a conditional adversarial loss in which class label information of one of the images comprised in the training data is set as a condition.

### Claim 9 (depends on 7)

The image processing apparatus of claim 7, wherein the adversarial loss is determined based on the morphing control parameter, an adversarial loss associated with a discrimination between the first image and the output morphing image, and an adversarial loss associated with a discrimination between the second image and the output morphing image, and
the pixel-wise reconstruction loss is determined based on the ground truth morphing image corresponding to the morphing control parameter among the images comprised in the training data and on the output morphing image.

### Claim 10 (depends on 7)

The image processing apparatus of claim 7, wherein the morphing control parameter comprises a content morphing control parameter and a style morphing control parameter,
for the obtaining of the output morphing image, wherein the processor is configured to:
obtain a first output morphing image comprising a content of the second image and a style of the first image; and
obtain a second output morphing image comprising a content of the first image and a style of the second image,
wherein the training of the morphing generator comprises:
training the morphing generator based on an adversarial loss associated with a discrimination between the first output morphing image and the first image, and an adversarial loss associated with a discrimination between the second output morphing image and the second image.

### Claim 11 (depends on 10)

The image processing apparatus of claim 10, for the training of the morphing generator, wherein the processor is configured to:
obtain a third morphing image comprising a content of the second output morphing image and the style of the second image by inputting the second output morphing image and the second image to the morphing generator;
obtain a fourth morphing image comprising a content of the first output morphing image and the style of the first image by inputting the first output morphing image and the first image to the morphing generator;
train the morphing generator based on an adversarial loss and a pixel-wise reconstruction loss that are based on the third output morphing image and the second image; and
train the morphing generator based on an adversarial loss and a pixel-wise reconstruction loss that are based on the fourth output morphing image and the first image.
