# US 2024/0202519 A1 · Generating vector representations of documents

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202519 A1
- Document ID: US20240202519A1
- Status: Published application
- Kind: A1
- Application number: 18/395,172
- Filed: 2023-12-22
- Published: 2024-06-20
- Earliest claimed: 2014-01-31
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Quoc V. Le
- Web page: https://patentgrep.com/patent/US20240202519A1

## CPC

- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 16/583: using metadata automatically derived from the content
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating document vector representations. One of the methods includes obtaining a new document; and determining a vector representation for the new document using a trained neural network system, wherein the trained neural network system has been trained to receive an input document and a sequence of words from the input document and to generate a respective word score for each word in a set of words, wherein each of the respective word scores represents a predicted likelihood that the corresponding word follows a last word in the sequence in the input document, and wherein determining the vector representation for the new document using the trained neural network system comprises iteratively providing each of the plurality of sequences of words to the trained neural network system to determine the vector representation for the new document using gradient descent.

## Claims

### Claim 1

(canceled)

### Claim 2

A method comprising:
obtaining a new document, wherein the new document includes a plurality of sequences of words, and, for each sequence of words, a word that follows a last word in the sequence of words in the new document;
generating a vector representation of the new document using a trained neural network system, wherein generating the vector representation of the new document using the trained neural network system comprises, for each iteration step of multiple iteration steps:
obtaining a current sequence of words from the plurality of sequences of words;
processing (i) data identifying the new document and (ii) the current sequence of words by the neural network system and in accordance with (i) trained values of a set of word parameters and (ii) current values of a set of document parameters to generate a respective word score for each word in a pre-determined set of words;
computing a gradient with respect to the vector representation of an error function that measures an error between the respective word scores and a target set of word scores that identifies the word that follows the last word in the current sequence of words in the new document; and
adjusting the current values of the set of document parameters based on the gradient using gradient descent while holding the trained values of the set of word parameters fixed; and
processing, by a text classification system, an input comprising the vector representation of the new document to generate a classification output for the new document.

### Claim 3 (depends on 2)

The method of claim 2, wherein the classification output comprises a respective score for each of a set of topics, with each score representing an estimated likelihood that the new document is about or relates to the corresponding topic.

### Claim 4 (depends on 2)

The method of claim 2, wherein the text classification system comprises a machine learning system.

### Claim 5 (depends on 4)

The method of claim 4, wherein the machine learning system comprises one of: a logistic regression system, a Support Vector Machines (SVM) system, or a k-means system.

### Claim 6 (depends on 2)

The method of claim 2, further comprising associating the vector representation with the document in a repository.

### Claim 7 (depends on 2)

The method of claim 2, wherein the trained values of the set of word parameters are determined from a training of the trained neural network system on different training documents.

### Claim 8 (depends on 7)

The method of claim 7, wherein the different training documents are unlabeled.

### Claim 9 (depends on 2)

The method of claim 2, wherein obtaining the current sequences of words from the plurality of sequences of words comprises:
applying a sliding window to the new document to extract each possible sequence of a predetermined fixed length from the new document.

### Claim 10 (depends on 2)

The method of claim 2, wherein the document comprises a web page.

### Claim 11

A system comprising:
one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
obtaining a new document, wherein the new document includes a plurality of sequences of words, and, for each sequence of words, a word that follows a last word in the sequence of words in the new document;
generating a vector representation of the new document using a trained neural network system, wherein generating the vector representation of the new document using the trained neural network system comprises, for each iteration step of multiple iteration steps:
obtaining a current sequence of words from the plurality of sequences of words;
processing (i) data identifying the new document and (ii) the current sequence of words by the neural network system and in accordance with (i) trained values of a set of word parameters and (ii) current values of a set of document parameters to generate a respective word score for each word in a pre-determined set of words;
computing a gradient with respect to the vector representation of an error function that measures an error between the respective word scores and a target set of word scores that identifies the word that follows the last word in the current sequence of words in the new document; and
adjusting the current values of the set of document parameters based on the gradient using gradient descent while holding the trained values of the set of word parameters fixed; and
processing, by a text classification system, an input comprising the vector representation of the new document to generate a classification output for the new document.

### Claim 12 (depends on 11)

The system of claim 11, wherein the classification output comprises a respective score for each of a set of topics, with each score representing an estimated likelihood that the new document is about or relates to the corresponding topic.

### Claim 13 (depends on 11)

The system of claim 11, wherein the text classification system comprises a machine learning system.

### Claim 14 (depends on 13)

The system of claim 13, wherein the machine learning system comprises one of: a logistic regression system, a Support Vector Machines (SVM) system, or a k-means system.

### Claim 15 (depends on 11)

The system of claim 11, wherein the operations further comprise associating the vector representation with the document in a repository.

### Claim 16 (depends on 11)

The system of claim 11, wherein the trained values of the set of word parameters are determined from a training of the trained neural network system on different training documents.

### Claim 17 (depends on 16)

The system of claim 16, wherein the different training documents are unlabeled.

### Claim 18 (depends on 11)

The system of claim 11, wherein obtaining the current sequences of words from the plurality of sequences of words comprises:
applying a sliding window to the new document to extract each possible sequence of a predetermined fixed length from the new document.

### Claim 19 (depends on 11)

The system of claim 11, wherein the document comprises a web page.

### Claim 20

One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
obtaining a new document, wherein the new document includes a plurality of sequences of words, and, for each sequence of words, a word that follows a last word in the sequence of words in the new document;
generating a vector representation of the new document using a trained neural network system, wherein generating the vector representation of the new document using the trained neural network system comprises, for each iteration step of multiple iteration steps:
obtaining a current sequence of words from the plurality of sequences of words;
processing (i) data identifying the new document and (ii) the current sequence of words by the neural network system and in accordance with (i) trained values of a set of word parameters and (ii) current values of a set of document parameters to generate a respective word score for each word in a pre-determined set of words;
computing a gradient with respect to the vector representation of an error function that measures an error between the respective word scores and a target set of word scores that identifies the word that follows the last word in the current sequence of words in the new document; and
adjusting the current values of the set of document parameters based on the gradient using gradient descent while holding the trained values of the set of word parameters fixed; and
processing, by a text classification system, an input comprising the vector representation of the new document to generate a classification output for the new document.

### Claim 21 (depends on 20)

The storage media of claim 20, wherein the classification output comprises a respective score for each of a set of topics, with each score representing an estimated likelihood that the new document is about or relates to the corresponding topic.
