# US 12,118,765 B2 · Method and system for product search based on deep-learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,765 B2
- Document ID: US12118765B2
- Status: Granted
- Kind: B2
- Application number: 17/464,571
- Filed: 2021-09-01
- Published: 2022-03-31
- Granted: 2024-10-15
- Earliest claimed: 2020-09-25
- Assignees: Nhn Corporation
- Applicants: NHN Corporation
- Inventors: Myounghoon Cho
- Web page: https://patentgrep.com/patent/US12118765B2

## CPC

- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06F 18/24133: {Distances to prototypes}
- G06Q 30/0623: {by investigating goods or services}
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06V 10/422: for representing the structure of the pattern or shape of an object therefor
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

A method and system for performing a deep learning based product search obtain an input image including a target product to be searched; transform a model pose included in the input image; obtain a standard input image having the transformed pose of the model; obtain a main product image having an area including the target product by performing deep learning based on the standard input image; extract a feature vector from the main product image; perform a product search for a product similar to the target product based on the feature vector; and output a result of the product search.

## Claims

### Claim 1

A method for performing a deep learning based product search by a product search application executed by a computing device, the method comprising:
preconfiguring a first standard pose used as a reference to unify model poses in images into one predetermined model pose for product search by determining the preconfigured first standard pose, which is used as the reference optimized for performing the product search by an image deep-learning neural network, and setting coordinates of key-points of the preconfigured first standard pose based on the image deep-learning neural network, wherein the image deep-learning neural network is trained to transform the model poses in the images into the preconfigured first standard pose based on a training data set comprising a plurality of product images based on the preconfigured first standard pose;
receiving an input image including a target product to be searched;
transforming a pose of a model included in the received input image into the preconfigured first standard pose by the image deep-learning neural network based on the preconfigured first standard pose;
obtaining a standard input image in which the pose of the model included in the received input image is transformed into the preconfigured first standard pose;
obtaining a main product image having an area including the target product to be searched by performing deep learning based on the standard input image;
extracting a feature vector from the main product image which has the area including the target product and is obtained by performing the deep learning based on the standard input image having the transformed pose of the model;
performing the product search for a product similar to the target product based on the feature vector extracted from the main product image; and
outputting a result of the product search.

### Claim 2 (depends on 1)

The method of claim 1, wherein the transforming of the pose of the model included in the received input image includes transforming the pose of the model included in the input image using deep-learning based on the preconfigured first standard pose.

### Claim 3 (depends on 2)

The method of claim 2, wherein the preconfigured first standard pose includes information on coordinates of key-points of a main body of the model included in the standard input image having the transformed pose of the model.

### Claim 4 (depends on 3)

The method of claim 3, wherein the preconfigured first standard pose is an image generated by graphic imagination of the information on the coordinates of the key-points of the main body of the model or information on coordinates of each key-point of the main body expressed by each image channel.

### Claim 5 (depends on 2)

The method of claim 2, wherein the transforming of the pose of the model included in the input image further includes transforming the pose of the model included in the input image into the preconfigured first standard pose using the image deep-learning neural network trained based on the training data set including the plurality of product images based on the preconfigured first standard pose.

### Claim 6 (depends on 1)

The method of claim 1, wherein the transforming of the pose of the model included in the input image further includes performing semantic segmentation on the input image, detecting one or more object areas included in the input image based on the performed semantic segmentation, and obtaining a semantic label map for the input image based on the input image from which the one or more object areas are detected and the preconfigured first standard pose including information on coordinates of key-points of a main body of the model included in the standard input image having the transformed pose of the model.

### Claim 7 (depends on 6)

The method of claim 6, wherein the semantic label map is a map image that categorizes the one or more object areas included in the input image into a respective key-point of the main body.

### Claim 8 (depends on 7)

The method of claim 7, wherein the obtaining of the standard input image having the transformed pose of the model includes obtaining the standard input image that transforms the pose of the model included in the input image into the preconfigured first standard pose based on the semantic label map.

### Claim 9

A system for performing a deep learning based product search, the system comprising:
a computing device executing a product search application configured to perform a deep learning based product search;
a product search server configured to operate in association with the product search application; and
a shopping mall server configured to provide data required for the deep learning based product search for an online shopping mall service, wherein the computing device is configured to:
preconfigure a first standard pose used as a reference to unify model poses in images into one predetermined model pose for product search by determining the preconfigured first standard pose, which is used as the reference optimized for performing the product search by an image deep-learning neural network, and setting coordinates of key-points of the preconfigured first standard pose based on the image deep-learning neural network, wherein the image deep-learning neural network is trained to transform the model poses in the images into the preconfigured first standard pose based on a training data set comprising a plurality of product images based on the preconfigured first standard pose, receive an input image including a target product to be searched, transform a pose of a model included in the received input image into the preconfigured first standard pose by the image deep-learning neural network based on the preconfigured first standard pose, obtain a standard input image in which the pose of the model included in the received input image is transformed into the preconfigured first standard pose, obtain a main product image having an area including the target product to be searched by performing deep learning based on the standard input image, extracting a feature vector from the main product image which has the area including the target product and is obtained by performing the deep learning based on the standard input image having the transformed pose of the model, perform the product search for a product similar to the target product based on the feature vector extracted from the main product image, and output a result of the product search.

### Claim 10 (depends on 9)

The system of claim 9, wherein the product search application configured to transform the pose of the model included in the input image using deep-learning based on the preconfigured first standard pose.

### Claim 11 (depends on 10)

The system of claim 10, wherein preconfigured the first standard pose includes information on coordinates of key-points of a main body of the model included in the standard input image having the transformed pose of the model.

### Claim 12 (depends on 11)

The system of claim 11, wherein the preconfigured first standard pose is an image generated by graphic imagination of the information on the coordinates of the key-points of the main body of the model or information on coordinates of each key-point of the main body expressed by each image channel.

### Claim 13 (depends on 10)

The system of claim 10, wherein the product search application is configured to transform the pose of the model included in the input image into the preconfigured first standard pose using the image deep-learning neural network trained based on the training data set including the plurality of product images based on the preconfigured first standard pose.

### Claim 14 (depends on 9)

The system of claim 9, wherein the product search application is configured to perform semantic segmentation on the input image, detect one or more object areas included in the input image based on the performed semantic segmentation, and obtain a semantic label map for the input image based on the input image from which the one or more object areas are detected and the preconfigured first standard pose including information on coordinates of key-points of a main body of the model included in the standard input image having the transformed pose of the model.

### Claim 15 (depends on 14)

The system of claim 14, wherein the semantic label map is a map image that categorizes the one or more object areas included in the input image into a respective key-point of the main body.

### Claim 16 (depends on 15)

The system of claim 15, wherein the product search application is configured to obtain the standard input image that transforms the pose of the model included in the input image into the preconfigured first standard pose based on the semantic label map.

### Claim 17

A method for performing deep learning based product search by a product search application executed by a computing device, the method comprising:
preconfiguring a first standard pose used as a reference to unify model poses in images into one predetermined model pose for product search by determining the preconfigured first standard pose, which is used as the reference optimized for performing the product search by an image deep-learning neural network, and setting coordinates of key-points of the preconfigured first standard pose based on the image deep-learning neural network, wherein the image deep-learning neural network is trained to transform the model poses in the images into the preconfigured first standard pose based on a training data set comprising a plurality of product images based on the preconfigured first standard pose;
receiving an input image including a target product to be searched;
detecting a model in the received input image;
transforming a pose of the detected model into the preconfigured first standard pose by the image deep-learning neural network based on the preconfigured first standard pose;
generating a standard input image in which the pose of the detected model included in the received input image is transformed into the preconfigured first standard pose;
determining an area including the target product based on the generated standard input image having the preconfigured first standard pose;
obtaining a main product image by extracting an image of the determined area including the target product;
extracting a feature vector from the main product image obtained by extracting the image of the determined area including the target product;
performing the product search for a product similar to the target product based on the feature vector extracted from the main product image; and
providing a product search result including an image of a product similar to the target product.

### Claim 18 (depends on 17)

The method of claim 17, wherein the transforming of the pose of the detected model into the preconfigured first standard pose includes inputting the input image to the deep learning neural network that transforms the pose of the model into the preconfigured first standard pose when the input image includes the model.