# US 2020/0401658 A1 · Probabilistic word embeddings for text classification

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0401658 A1
- Document ID: US20200401658A1
- Status: Published application
- Kind: A1
- Application number: 16/444,794
- Filed: 2019-06-18
- Published: 2020-12-24
- Earliest claimed: 2019-06-18
- Assignees: Sap SE
- Applicants: Sap SE
- Inventors: Gil Katz; Mathis Lamarre
- Web page: https://patentgrep.com/patent/US20200401658A1

## CPC

- G06F 40/295: Named entity recognition
- H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
- G06F 16/35: Clustering; Classification
- G06F 40/216: using statistical methods
- H04L 51/046: Interoperability with other network applications or services
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Disclosed are systems, methods, and non-transitory computer-readable media for probabilistic word embeddings for text classification. A text classification system receives a message including a keyword and determines an embedding probability distribution representing the keyword. The text classification system then determines an embedding value for the keyword based on the embedding probability distribution. The text classification system uses the embedding value as input into a set of mathematical functions, yielding a first set of coefficient values for the keyword. Each respective mathematical function from the set corresponds to a respective classification label from a set of classification labels and defines a continuous surface. Each respective mathematical function is determined from embedding values for a set of known keywords, distribution variance values for the set of known keywords, and a subset of coefficient values for the set of known keywords that corresponds to the respective classification label.

## Claims

### Claim 1

A method comprising:
receiving, from a client device, a first message transmitted as part of a communication session, the first message including at least a first keyword;
determining, by one or more computer processors, a first embedding probability distribution representing the first keyword;
determining, by the one or more computer processors, a first embedding value for the first keyword based on the first embedding probability distribution;
using the first embedding value as input into each mathematical function in a set mathematical functions, yielding a first set of coefficient values for the first keyword, each respective mathematical function from the set of mathematical functions corresponding to a respective classification label from a set of classification labels and defining a continuous surface, each respective mathematical function determined from embedding values for a set of known keywords, distribution variance values for the set of known keywords, and a subset of coefficient values for the set of known keywords, the respective subset of coefficient values corresponding to the respective classification label;
for each classification label from the set of classification labels, calculating, by the one or more computer processors, a probability score based on the first set of coefficient values, yielding a set of probability scores for the first message; and
assigning a first classification label from the set of classification labels to the first message based on the set of probability scores for the first message.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining the first embedding value for the first keyword comprises:
determining a mean vector of the first embedding probability distribution representing the first keyword.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
comparing the probability scores from the set of probability scores to identify a highest probability score, yielding a comparison; and
determining, based on the comparison, that a first probability score from the set of probability scores is the highest probability score, the first probability score corresponding to the first classification label.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
for each known keyword from the set of known keywords, determining, based on an embedding probability distribution representing the respective known keyword, an embedding value for the respective known keyword, yielding the embedding values for the set of known keywords; and
for each known keyword, determining, based on the embedding probability distribution representing the respective known keyword, a distribution variance value for the respective known keyword, yielding the distribution variance values for the set of known keywords; and
determining the set of mathematical functions based on the embedding values for the set of known keywords, the distribution variance values for the set of known keywords, and the respective subset of coefficient values corresponding to the respective classification label, each mathematical function defining a continuous surface that includes each coefficient value from the respective subset of coefficient values across an embedding space, a shape of the surface being determined based on the distribution variance values for the set of known keywords.

### Claim 5 (depends on 4)

The method of claim 4, wherein determining the set of mathematical functions comprises:
comparing a first distribution variance value corresponding to a first known keyword from the set of known keywords to a threshold distribution variance value, yielding a comparison;
determining a first adjusted distribution variance value based on the comparison; and
determining the space of the surface spanning from a first coefficient value corresponding to the first known keyword to a second coefficient value based on the first adjusted distribution variance value.

### Claim 6 (depends on 1)

The method of claim 1, wherein each classification label from the set of classification labels corresponds to a possible intent of the text.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
determining a response message to the first message based on a first intent corresponding to the first classification label; and
transmitting the response message to the client device.

### Claim 8

A computing system comprising:
one or more computer processors; and
one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the computing system to perform operations comprising:
receiving, from a client device, a first message transmitted as part of a communication session, the first message including at least a first keyword;
determining a first embedding probability distribution representing the first keyword;
determining a first embedding value for the first keyword based on the first embedding probability distribution;
using the first embedding value as input into each mathematical function in a set mathematical functions, yielding a first set of coefficient values for the first keyword, each respective mathematical function from the set of mathematical functions corresponding to a respective classification label from a set of classification labels and defining a continuous surface, each respective mathematical function determined from embedding values for a set of known keywords, distribution variance values for the set of known keywords, and a subset of coefficient values for the set of known keywords, the respective subset of coefficient values corresponding to the respective classification label;
for each classification label from the set of classification labels, calculating a probability score based on the first set of coefficient values, yielding a set of probability scores for the first message; and
assigning a first classification label from the set of classification labels to the first message based on the set of probability scores for the first message.

### Claim 9 (depends on 8)

The computing system of claim 8, wherein determining the first embedding value for the first keyword comprises:
determining a mean vector of the first embedding probability distribution representing the first keyword.

### Claim 10 (depends on 8)

The computing system of claim 8, the operations further comprising:
comparing the probability scores from the set of probability scores to identify a highest probability score, yielding a comparison; and
determining, based on the comparison, that a first probability score from the set of probability scores is the highest probability score, the first probability score corresponding to the first classification label.

### Claim 11 (depends on 8)

The computing system of claim 8, the operations further comprising:
for each known keyword from the set of known keywords, determining, based on an embedding probability distribution representing the respective known keyword, an embedding value for the respective known keyword, yielding the embedding values for the set of known keywords; and
for each known keyword, determining, based on the embedding probability distribution representing the respective known keyword, a distribution variance value for the respective known keyword, yielding the distribution variance values for the set of known keywords; and
determining the set of mathematical functions based on the embedding values for the set of known keywords, the distribution variance values for the set of known keywords, and the respective subset of coefficient values corresponding to the respective classification label, each mathematical function defining a continuous surface that includes each coefficient value from the respective subset of coefficient values across an embedding space, a shape of the surface being determined based on the distribution variance values for the set of known keywords.

### Claim 12 (depends on 11)

The computing system of claim 11, wherein determining the set of mathematical functions comprises:
comparing a first distribution variance value corresponding to a first known keyword from the set of known keywords to a threshold distribution variance value, yielding a comparison;
determining a first adjusted distribution variance value based on the comparison; and
determining the space of the surface spanning from a first coefficient value corresponding to the first known keyword to a second coefficient value based on the first adjusted distribution variance value.

### Claim 13 (depends on 8)

The computing system of claim 8, wherein each classification label from the set of classification labels corresponds to a possible intent of the text.

### Claim 14 (depends on 13)

The method of claim 13, the operations further comprising:
determining a response message to the first message based on a first intent corresponding to the first classification label; and
transmitting the response message to the client device.

### Claim 15

A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors of a computing system, cause the computing system to perform operations comprising:
receiving, from a client device, a first message transmitted as part of a communication session, the first message including at least a first keyword;
determining a first embedding probability distribution representing the first keyword;
determining a first embedding value for the first keyword based on the first embedding probability distribution;
using the first embedding value as input into each mathematical function in a set mathematical functions, yielding a first set of coefficient values for the first keyword, each respective mathematical function from the set of mathematical functions corresponding to a respective classification label from a set of classification labels and defining a continuous surface, each respective mathematical function determined from embedding values for a set of known keywords, distribution variance values for the set of known keywords, and a subset of coefficient values for the set of known keywords, the respective subset of coefficient values corresponding to the respective classification label;
for each classification label from the set of classification labels, calculating a probability score based on the first set of coefficient values, yielding a set of probability scores for the first message; and
assigning a first classification label from the set of classification labels to the first message based on the set of probability scores for the first message.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein determining the first embedding value for the first keyword comprises:
determining a mean vector of the first embedding probability distribution representing the first keyword.

### Claim 17 (depends on 15)

The non-transitory computer-readable medium of claim 15, the operations further comprising:
comparing the probability scores from the set of probability scores to identify a highest probability score, yielding a comparison; and
determining, based on the comparison, that a first probability score from the set of probability scores is the highest probability score, the first probability score corresponding to the first classification label.

### Claim 18 (depends on 15)

The non-transitory computer-readable medium of claim 15, the operations further comprising:
for each known keyword from the set of known keywords, determining, based on an embedding probability distribution representing the respective known keyword, an embedding value for the respective known keyword, yielding the embedding values for the set of known keywords; and
for each known keyword, determining, based on the embedding probability distribution representing the respective known keyword, a distribution variance value for the respective known keyword, yielding the distribution variance values for the set of known keywords; and
determining the set of mathematical functions based on the embedding values for the set of known keywords, the distribution variance values for the set of known keywords, and the respective subset of coefficient values corresponding to the respective classification label, each mathematical function defining a continuous surface that includes each coefficient value from the respective subset of coefficient values across an embedding space, a shape of the surface being determined based on the distribution variance values for the set of known keywords.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein determining the set of mathematical functions comprises:
comparing a first distribution variance value corresponding to a first known keyword from the set of known keywords to a threshold distribution variance value, yielding a comparison;
determining a first adjusted distribution variance value based on the comparison; and
determining the space of the surface spanning from a first coefficient value corresponding to the first known keyword to a second coefficient value based on the first adjusted distribution variance value.

### Claim 20 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein each classification label from the set of classification labels corresponds to a possible intent of the text, the operations further comprising:
determining a response message to the first message based on a first intent corresponding to the first classification label; and
transmitting the response message to the client device.