# US 12,008,331 B2 · Utilizing visual and textual aspects of images with recommendation systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,331 B2
- Document ID: US12008331B2
- Status: Granted
- Kind: B2
- Application number: 17/560,436
- Filed: 2021-12-23
- Published: 2023-06-29
- Granted: 2024-06-11
- Earliest claimed: 2021-12-23
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Xun Luan; Aman Gupta; Sirjan Kafle; Ananth Sankar; Di Wen; Saurabh Kataria; Ying Xuan; Sakshi Verma; Bharat Kumar Jain; Xue Xia; Bhargavkumar Kanubhai Patel; Vipin Gupta; Nikita Gupta
- Web page: https://patentgrep.com/patent/US12008331B2

## CPC

- G06F 40/258: Heading extraction; Automatic titling; Numbering
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/09: Supervised learning
- G06V 10/82: using neural networks
- G06V 30/10: Character recognition
- G06V 30/19147: {Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting}
- G06F 40/216: using statistical methods
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 40/56: Natural language generation
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Described herein are systems and methods for generating an embedding—a learned representation—for an image. The embedding for the image is derived to capture visual aspects, as well as textual aspects, of the image. An encoder-decoder is trained to generate the visual representation of the image. An optical character recognition (OCR) algorithm is used to identify text/words in the image. From these words, an embedding is derived by performing an average pooling operation on pre-trained embeddings that map to the identified words. Finally, the embedding representing the visual aspects of the image is combined with the embedding representing the textual aspects of the image to generate a final embedding for the image.

## Claims

### Claim 1

A computer-implemented method comprising:
with a machine learning algorithm, training an encoder-decoder model to generate a caption for an image by generating with the encoder an embedding, and then decoding the embedding with the decoder to generate the caption, wherein a dataset comprising a plurality of images with associated captions is used to train the encoder-decoder model;
generating a first embedding for an image by:
detecting words present in the image with an optical character recognition (OCR) algorithm;
using pre-trained word embeddings to derive a word embedding for each word detected in the image; and
performing an average pooling operation on the word embeddings for each word detected in the image, wherein the result of the average pooling operation is the first embedding for the image;
generating a second embedding for the image by:
using the image as input to the pre-trained encoder-decoder model,
generating from the image with the encoder the second embedding for the image;
combining the first embedding with the second embedding to derive for the image a final embedding; and
storing the final embedding for the image.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein i) the encoder of the encoder-decoder model is a convolutional neural network that outputs a tensor having a size of 64 by 2048, and ii) the encoder-decoder model includes a compression layer between the encoder and the decoder, the method further comprising:
generating the second embedding for the image by:
with the compression layer, compressing the tensor output by the encoder to a size of 64 by D, wherein D is 30, 50 or 100; and
performing an average pooling operation on the individual components of the tensor to generate the second embedding with a dimension of D.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the compression layer is implemented as a 1 by 1 convolutional layer having D filters, each filter having a size of 1 by 1 by 2048.

### Claim 4 (depends on 2)

The computer-implemented method of claim 2, wherein the encoder of the encoder-decoder model is implemented based on the Inception v3 model.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, further comprising:
deriving with the OCR algorithm a confidence score representing a measure of confidence that the OCR algorithm was accurate in detecting the words present in the image; and
combining the confidence score with the first embedding and the second embedding to generate the final embedding for the image.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein combining the first embedding with the second embedding to derive for the image a final embedding comprises:
concatenating the first embedding with the second embedding to derive for the image the final embedding.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising:
using the final embedding for the image as an input feature to a machine learned model that has been trained to generate a score for an item of content that includes the image, wherein the score is used in ranking the content item relative to other content items.

### Claim 8

A system comprising:
a memory storage device for storing computer-executable instructions; and
at least one processor, which, when executing the computer-executable instructions, causes the system to:
with a machine learning algorithm, train an encoder-decoder model to generate a caption for an image by generating with the encoder an embedding, and then decoding the embedding with the decoder to generate the caption, wherein a dataset comprising a plurality of images with associated captions is used to train the encoder-decoder model;
generate a first embedding for the image by:
detecting words present in an image with an optical character recognition (OCR) algorithm;
using pre-trained word embeddings to derive a word embedding for each word detected in the image; and
performing an average pooling operation on the word embeddings for each word detected in the image, wherein the result of the average pooling operation is the first embedding for the image;
generate a second embedding for the image by:
using the image as input to the pre-trained encoder-decoder model, generate from the image with the encoder the second embedding for the image;
concatenate the first embedding with the second embedding to derive for the image a final embedding; and
store the final embedding for the image.

### Claim 9 (depends on 8)

The system of claim 8, wherein i) the encoder of the encoder-decoder model is a convolutional neural network that outputs a tensor having a size of 64 by 2048, and ii) the encoder-decoder model includes a compression layer between the encoder and the decoder, and iii) the computer-executable instructions, when executed by the processor, cause the system to:
generate the second embedding for the image by:
with the compression layer, compressing the tensor output by the encoder to a size of 64 by D, wherein D is 30, 50 or 100; and
performing an average pooling operation on the individual components of the tensor to generate the second embedding with a dimension of D.

### Claim 10 (depends on 9)

The system of claim 9, wherein the compression layer is implemented as a 1 by 1 convolutional layer having D filters, each filter having a size of 1 by 1 by 2048.

### Claim 11 (depends on 9)

The system of claim 9, wherein the encoder of the encoder-decoder model is implemented based on the Inception v3model.

### Claim 12 (depends on 9)

The system of claim 9, wherein the computer-executable instructions, when executed by the at least on processor, cause the system to:
derive with the OCR algorithm a confidence score representing a measure of confidence that the OCR algorithm was accurate in detecting the words present in the image; and
combine the confidence score with the first embedding and the second embedding to generate the final embedding for the image.

### Claim 13 (depends on 8)

The system of claim 8, wherein the computer-executable instructions, when executed by the at least on processor, cause the system to:
use the final embedding for the image as an input feature to a machine learned model that has been trained to generate a score for an item of content that includes the image, wherein the score is used in ranking the content item relative to other content items.

### Claim 14

A system comprising:
means for training an encoder-decoder model to generate a caption for an image by generating with the encoder an embedding, and then decoding the embedding with the decoder to generate the caption, wherein a dataset comprising a plurality of images with associated captions is used to train the encoder-decoder model;
means for generating a first embedding for the image by:
detecting words present in an image with an optical character recognition (OCR) algorithm;
using pre-trained word embeddings to derive a word embedding for each word detected in the image; and
performing an average pooling operation on the word embeddings for each word detected in the image, wherein the result of the average pooling operation is the first embedding for the image; means for generating a second embedding for the image by:
using the image as input to the pre-trained encoder-decoder model, generating from the image with the encoder the second embedding for the image;
means for combining the first embedding with the second embedding to derive for the image a final embedding; and
means for storing the final embedding for the image.

### Claim 15 (depends on 14)

The system of claim 14, wherein i) the encoder of the encoder-decoder model is a convolutional neural network that outputs a tensor having a size of 64 by 2048, and ii) the encoder-decoder model includes a compression layer between the encoder and the decoder, the system further comprising:
means for generating the second embedding for the image by:
with the compression layer, compressing the tensor output by the encoder to a size of 64 by D, wherein D is 30, 50 or 100; and
performing an average pooling operation on the individual components of the tensor to generate the second embedding with a dimension of D.

### Claim 16 (depends on 15)

The system method of claim 15, wherein the compression layer is implemented as a 1 by 1 convolutional layer having D filters, each filter having a size of 1 by 1 by 2048.

### Claim 17 (depends on 15)

The system of claim 15, wherein the encoder of the encoder-decoder model is implemented based on the Inception v3 model.

### Claim 18 (depends on 14)

The system of claim 14, further comprising:
means for deriving with the OCR algorithm a confidence score representing a measure of confidence that the OCR algorithm was accurate in detecting the words present in the image; and
means for combining the confidence score with the first embedding and the second embedding to generate the final embedding for the image.

### Claim 19 (depends on 14)

The system of claim 14, further comprising:
means for concatenating the first embedding with the second embedding to derive for the image the final embedding.

### Claim 20 (depends on 14)

The system of claim 14, further comprising:
means for using the final embedding for the image as an input feature to a machine learned model that has been trained to generate a score for an item of content that includes the image, wherein the score is used in ranking the content item relative to other content items.