# US 2023/0267600 A1 · Vector quantized auto-encoder codebook learning for manufacturing display extreme minor defects detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267600 A1
- Document ID: US20230267600A1
- Status: Published application
- Kind: A1
- Application number: 17/728,882
- Filed: 2022-04-25
- Published: 2023-08-24
- Earliest claimed: 2022-02-24
- Assignees: Samsung Display Co., Ltd.
- Applicants: Samsung Display Co., Ltd.
- Inventors: Shuhui Qu; Qisen Cheng; Janghwan Lee
- Web page: https://patentgrep.com/patent/US20230267600A1

## CPC

- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06T 7/001: {using an image reference approach}
- G06N 3/045: Combinations of networks
- G06N 3/0475: Generative networks
- G06N 3/048: Activation functions
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/094: Adversarial learning
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A system including: a memory, an encoder, a decoder, and a processor, the processor being connected to the memory, the encoder, and the decoder. The system is configured to: receive, at the encoder, an input image, divide, by the encoder, the input image into a plurality of image patches, select, by the encoder, codes corresponding to the plurality of image patches of the input image, from a codebook including the codes. The system is further configured to determine, by the encoder, an assigned code matrix including the codes corresponding to the plurality of image patches of the input image, receive, by the decoder, the assigned code matrix from the encoder. The system is further configured to generate, by the decoder, a reconstructed image based on the assigned code matrix.

## Claims

### Claim 1

A system comprising:
a memory,
an encoder;
a decoder; and
a processor, the processor being connected to the memory, the encoder, and the decoder, wherein the system is configured to:
receive, at the encoder, an input image;
divide, by the encoder, the input image into a plurality of image patches;
select, by the encoder, codes corresponding to the plurality of image patches of the input image, from a codebook comprising the codes;
determine, by the encoder, an assigned code matrix comprising the codes corresponding to the plurality of image patches of the input image;
receive, by the decoder, the assigned code matrix from the encoder;
generate, by the decoder, a reconstructed image based on the assigned code matrix.

### Claim 2 (depends on 1)

The system of claim 1, wherein the codes corresponding to the plurality of image patches of the input image are selected from the codebook using a look-up function, wherein the look-up function, the codebook, and the assigned code matrix are stored in the memory.

### Claim 3 (depends on 2)

The system of claim 2, wherein the system is further configured to:
extract, by the encoder, latent features from the plurality of image patches of the input image; and
determine, by the encoder, a latent features matrix comprising the latent features extracted from the plurality of image patches of the input image.

### Claim 4 (depends on 3)

The system of claim 3, wherein the look-up function is configured to select a nearest code as an assignment for each of the latent features in the latent feature matrix to determine the assigned code matrix.

### Claim 5 (depends on 3)

The system of claim 3, wherein the encoder is configured to select a code corresponding to an image patch of the plurality of image patches of the input image from the codebook by comparing similarity measures between a latent feature representation of the image patch in the latent feature matrix and a corresponding code in the codebook.

### Claim 6 (depends on 5)

The system of claim 5, wherein the similarity measures comprise an euclidean distance or a mahalanobis distance between the latent feature representation of the image patch in the latent feature matrix and the corresponding code in the codebook.

### Claim 7 (depends on 1)

The system of claim 1, wherein the assigned code matrix comprises the codes corresponding to the plurality of image patches of the input image.

### Claim 8 (depends on 7)

The system of claim 7, wherein a code from among the codes in the assigned code matrix is assigned to an image patch of the plurality of image patches of the input image based on a vector quantization of a latent feature corresponding to the image patch.

### Claim 9 (depends on 8)

the system of claim 8, wherein a vector quantization loss of the system comprises a reconstruction loss that occurs during generation of the reconstructed image and a loss that occurs during the vector quantization of latent features corresponding to the plurality of image patches of the input image.

### Claim 10 (depends on 9)

The system of claim 9, further comprising:
a patch-wise discriminator network configured to operate as a generative adversarial network to determine an adversarial training loss between the input image and the reconstructed image.

### Claim 11 (depends on 10)

The system of claim 10, wherein a total loss of the system in generating the reconstructed image from the input image comprises the vector quantization loss and the adversarial training loss.

### Claim 12 (depends on 1)

The system of claim 1, further configured to:
receive, by the encoder, a test input image;
divide, by the encoder, the test input image into a plurality of image patches;
extract, by the encoder, latent features from the plurality of image patches of the test input image;
encode, by the encoder, each of the plurality of image patches into a latent feature vector based on the extracted latent features;
assign, by the encoder, a code to each of the plurality of image patches to determine assigned codes of the plurality of image patches;
determine, by the encoder, a patch-set comprising the assigned codes;
determine, by the encoder, an anomaly score of each of the assigned codes of the patch-set;
compare, by the encoder, the anomaly score of each of the assigned codes of the patch-set with a threshold;
determine, by the encoder, a defect in one or more of the plurality of image patches based on a result of the comparison.

### Claim 13 (depends on 12)

The system of claim 12, wherein a code from the codebook that is of a shortest distance to the latent feature vector of an image patch of the plurality of image patches among the codes in the codebook is assigned to the image patch.

### Claim 14 (depends on 12)

The system of claim 12, wherein the anomaly score of each of the assigned codes of the patch-set is determined based on a probability density function.

### Claim 15

A method comprising:
receiving, at an encoder, an input image;
dividing, at the encoder, the input image into a plurality of image patches;
selecting, by the encoder, codes corresponding to the plurality of image patches of the input image, from a codebook comprising the codes;
determining, by the encoder, an assigned code matrix comprising the codes corresponding to the plurality of image patches of the input image;
receiving, by a decoder, the assigned code matrix from the encoder;
generating, by the decoder, a reconstructed image based on the assigned code matrix.

### Claim 16 (depends on 15)

The method of claim 15, wherein the codes corresponding to the plurality of image patches of the input image are selected from the codebook using a look-up function, wherein the method further comprises:
extracting, by the encoder, latent features from the plurality of image patches of the input image; and
determining, by the encoder, a latent features matrix comprising the latent features extracted from the plurality of image patches of the input image.

### Claim 17 (depends on 16)

The method of claim 16, wherein the look-up function is configured to select a nearest code as an assignment for each of the latent features in the latent feature matrix to determine the assigned code matrix, and
wherein the encoder is configured to select a code corresponding to an image patch of the plurality of image patches of the input image from the codebook by comparing similarity measures between a latent feature representation of the image patch in the latent feature matrix and a corresponding code in the codebook.

### Claim 18 (depends on 17)

The method of claim 17, wherein a code from among the codes in the assigned code matrix is assigned to an image patch of the plurality of image patches of the input image based on a vector quantization of a latent feature corresponding to the image patch,
wherein a vector quantization loss comprises a reconstruction loss that occurs during generation of the reconstructed image and a loss that occurs during the vector quantization of latent features corresponding to the plurality of image patches of the input image, and
wherein a total loss for generating the reconstructed image from the input image comprises the vector quantization loss and an adversarial training loss.

### Claim 19 (depends on 15)

The method of claim 15, further comprising:
extracting, by the encoder, latent features from the plurality of image patches of the input image;
encoding, by the encoder, each of the plurality of image patches into a latent feature vector based on the extracted latent features;
assigning, by the encoder, a code to each of the plurality of image patches to determine assigned codes of the plurality of image patches;
determining, by the encoder, a patch-set comprising the assigned codes;
determining, by the encoder, an anomaly score of each of the assigned codes of the patch-set;
compare, by the encoder, the anomaly score of each of the assigned codes of the patch-set with a threshold;
determine, by the encoder, a defect in one or more of the plurality of image patches based on a result of the comparison.

### Claim 20

A non-transitory computer readable storage medium configured to store instructions that, when executed by a processor included in a computing device, cause the computing device to:
receive, at an encoder of the computer device, an input image;
divide, at the encoder, the input image into a plurality of image patches;
select, by the encoder, codes corresponding to the plurality of image patches of the input image, from a codebook comprising the codes;
determine, by the encoder, an assigned code matrix comprising the codes corresponding to the plurality of image patches of the input image;
receive, by a decoder of the computer device, the assigned code matrix from the encoder;
generate, by the decoder, a reconstructed image based on the assigned code matrix.
