# US 2022/0092106 A1 · Method, system and computer program for coloring based on deep learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0092106 A1
- Document ID: US20220092106A1
- Status: Published application
- Kind: A1
- Application number: 17/449,710
- Filed: 2021-10-01
- Published: 2022-03-24
- Earliest claimed: 2019-04-02
- Assignees: Naver Webtoon Ltd.; Korea University Research and Business Foundation
- Applicants: Naver Webtoon Ltd.; Korea University Research and Business Foundation
- Inventors: Jae Hyuk CHANG; Jae Gul CHOO; Seung Joo YOO; Sung Hyo CHUNG; Ga Young LEE; Hyo Jin BAHNG
- Web page: https://patentgrep.com/patent/US20220092106A1

## CPC

- G06F 16/535: Filtering based on additional data, e.g. user or group profiles
- G06N 3/08: Learning methods
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/094: Adversarial learning
- G06T 11/40: Filling planar surfaces by adding surface attributes, e.g. adding colours or textures
- G06T 13/00: Animation
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A deep learning-based coloring system includes a memory network configured to provide a color feature in response to a specific query and a coloring network configured to perform coloring, based on the color feature generated by the memory network. The memory network includes: a query generation unit configured to generate a query; a neighbor calculation unit configured to calculate k-nearest neighbors, based on similarities between the query and key memory values; a color feature determination unit configured to generate color features for indicating color information stored in the key memory; a threshold triplet loss calculation unit configured to calculate a threshold triplet loss, based on a comparison between a threshold and a distance between the color features; and a memory update unit configured to update a memory, based on whether a distance between a top value and a value of a newly input query is within the threshold.

## Claims

### Claim 1

A deep learning-based coloring system comprising:
a memory network configured to provide a color feature in response to a specific query and including a key memory related to a spatial feature and a value memory related to a color feature; and
a coloring network configured to perform coloring, based on the color feature generated by the memory network, wherein
the memory network comprises:
a query generation unit configured to generate a query;
a neighbor calculation unit configured to calculate k-nearest neighbors, based on similarities between the query and key memory values;
a color feature determination unit configured to generate a plurality of color features for indicating color information stored in the key memory;
a threshold triplet loss calculation unit configured to calculate a threshold triplet loss, based on a comparison between a threshold and a distance between the plurality of color features generated by the color feature determination unit; and
a memory update unit configured to update a memory, based on whether a distance between a top value and a value of a newly input query is within the threshold.

### Claim 2 (depends on 1)

The deep learning-based coloring system of claim 1, wherein the k-nearest neighbors are determined to be memory slots having similarities between the query and the spatial feature from 1 to k, based on cosine similarities between the query and the key memory values.

### Claim 3 (depends on 2)

The deep learning-based coloring system of claim 2, wherein the pre-assigned value is a value memory value having a same index as a key memory value having a highest cosine similarity with the query, when a 1-nearest neighbor is calculated from among the k-nearest neighbors.

### Claim 4 (depends on 1)

The deep learning-based coloring system of claim 1, wherein the threshold triplet loss is a value that maximizes a similarity between the query and a positive key and minimizes a similarity between the query and a negative key.

### Claim 5 (depends on 4)

The deep learning-based coloring system of claim 4, wherein the positive key is a key memory value having a same index as a nearest value memory value from among value memory values of which distances to a color feature value of the query are less than the threshold, and
the negative key is a key memory value having a same index as a value memory value having a smallest distance from among value memory values of which distances to the color feature value of the query exceed the threshold.

### Claim 6 (depends on 1)

The deep learning-based coloring system of claim 1, wherein the color feature comprises two types of variants, wherein the two types of variants include Cast obtained by calculating a probability value regarding how many of 313 colors pre-set for each image there are, and CRGB obtained by extracting 10 pixel values most dominant for each image by using a pre-set library.

### Claim 7 (depends on 1)

The deep learning-based coloring system of claim 1, wherein the memory network determines that images having spatial features corresponding to the k-nearest neighbors and color features corresponding to color distances to the query, which are less than the threshold, belong to a same class.

### Claim 8 (depends on 1)

The deep learning-based coloring system of claim 1, wherein the coloring network comprises a generator and a determiner,
wherein the determiner receives a gray scale input image and the color feature.

### Claim 9 (depends on 1)

The deep learning-based coloring system of claim 1, wherein the memory network is configured to additionally store an age, wherein the age indicates a life of an item stored in the memory.

### Claim 10

A deep learning-based coloring method performed by a computing device comprising a key memory related to a spatial feature and a value memory related to a color feature, the deep learning-based coloring method comprising:
generating a query;
calculating k-nearest neighbors, based on similarities between the query and key memory values;
generating a plurality of color features for indicating color information stored in the key memory;
calculating a threshold triplet loss, based on a comparison between a threshold and a distance between the plurality of color features; and
updating a memory, based on whether a distance between a pre-assigned value and a value of a newly input query is within the threshold.

### Claim 11 (depends on 10)

The deep learning-based coloring method of claim 10, wherein the k-nearest neighbors are determined to be memory slots having similarities between the query and the spatial feature from 1 to k, based on cosine similarities between the query and the key memory values.

### Claim 12 (depends on 11)

The deep learning-based coloring method of claim 11, wherein the pre-assigned value is a value memory value having a same index as a key memory value having a highest cosine similarity with the query, when a 1-nearest neighbor is calculated from among the k-nearest neighbors.

### Claim 13 (depends on 10)

The deep learning-based coloring method of claim 10, wherein the threshold triplet loss is a value that maximizes a similarity between the query and a positive key and minimizes a similarity between the query and a negative key, wherein the positive key is a key memory value having a same index as a nearest value memory value from among value memory values of which distances to a color feature value of the query are less than the threshold, and
the negative key is a key memory value having a same index as a value memory value having a smallest distance from among value memory values of which distances to the color feature value of the query exceed the threshold.

### Claim 14 (depends on 10)

The deep learning-based coloring method of claim 10, wherein the color feature comprises two types of variants, wherein the two types of variants include Cast obtained by calculating a probability value regarding how many of 313 colors pre-set for each image there are, and CRGB obtained by extracting 10 pixel values most dominant for each image by using a pre-set library.
