# US 2020/0004819 A1 · Predicting probablity of occurrence of a string using sequence of vectors

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0004819 A1
- Document ID: US20200004819A1
- Status: Published application
- Kind: A1
- Application number: 16/021,689
- Filed: 2018-06-28
- Published: 2020-01-02
- Earliest claimed: 2018-06-27
- Assignees: Abbyy Production LLC
- Applicants: ABBYY Production LLC
- Inventors: Evgenii Mikhaylovich Indenbom; Daniil Garryevich Anastasiev
- Web page: https://patentgrep.com/patent/US20200004819A1

## CPC

- G06F 40/30: Semantic analysis
- G06F 40/242: Dictionaries
- G06F 40/253: Grammatical analysis; Style critique
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/048: Activation functions
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods are disclosed to receive a plurality of strings where each string of the plurality of strings comprises a plurality of symbols. For each string of the plurality of strings, a first sequence of vectors is generated based at least on a maximum length of word for each symbol in the string. The first sequence of vectors is provided to a machine learning unit for each string of the plurality of strings. A probability of occurrence of each string of the plurality of strings is obtained from the machine learning unit.

## Claims

### Claim 1

A method comprising:
receiving a plurality of strings, each string of the plurality of strings comprising a plurality of symbols;
for each string of the plurality of strings, generating, by a processing device, a first sequence of vectors based at least on a maximum length of word for each symbol in the string, wherein the maximum length of word for each symbol in the string corresponds to length of a longest possible word within the string that starts with the symbol, the longest possible word comprising one or more of a word: 1) with a commonplace meaning, or 2) found in a dictionary;
providing to a machine learning unit the first sequence of vectors for each string of the plurality of strings; and
obtaining from the machine learning unit a probability of occurrence of each string of the plurality of strings.

### Claim 2 (depends on 1)

The method of claim 1, wherein each vector of the first sequence of vectors corresponds to each symbol in the string.

### Claim 3 (depends on 2)

The method of claim 2, wherein each vector of the first sequence of vectors is derived by joining together a first vector comprising the maximum length of word for a given symbol in the string and a second vector comprising a symbol vector for the given symbol in the string.

### Claim 4

(canceled)

### Claim 5

(canceled)

### Claim 6 (depends on 3)

The method of claim 3, wherein the symbol vector for each symbol in the string is based on one or more of:
a symbol embedding;
a unified vector for symbols of an alphabet that is different from the alphabet used in the string;
a vector for frequently occurring punctuation marks;
a vector for rare punctuation marks; or
a vector for numerals.

### Claim 7 (depends on 1)

The method of claim 1, wherein each string of the plurality of strings differs from each other string of the plurality of strings by one symbol, the one symbol being located on a same position of each string.

### Claim 8 (depends on 1)

The method of claim 1, wherein the machine learning unit comprises:
a first fully connected layer and a second fully connected layer to apply matrix transformation on the first sequence of vectors for each string; and
a third fully connected layer to be used as an output layer.

### Claim 9 (depends on 8)

The method of claim 8, wherein a batch normalization function and a rectifier linear unit activation function are applied on a first output of the first fully connected layer and on a second output of the second fully connected layer, and wherein a sigmoid activation function is applied on a third output of the third fully connected layer.

### Claim 10 (depends on 1)

The method of claim 1, wherein the probability of occurrence for a given string having a value nearest to ‘one’ relative to the probability of occurrence values of the remaining strings indicates that the given string has the highest probability of occurrence.

### Claim 11

A system comprising:
a memory device storing instructions;
a processing device coupled to the memory device, the processing device to execute the instructions to:
receive a plurality of strings, each string of the plurality of strings comprising a plurality of symbols;
for each string of the plurality of strings, generate a first sequence of vectors based at least on a maximum length of word for each symbol in the string wherein the maximum length of word for each symbol in the string corresponds to length of a longest possible word within the string that starts with the symbol, the longest possible word comprising a word with a commonplace meaning;
provide to a machine learning unit the first sequence of vectors for each string of the plurality of strings; and
obtain from the machine learning unit a probability of occurrence of each string of the plurality of strings.

### Claim 12 (depends on 11)

The system of claim 11, wherein each vector of the first sequence of vectors corresponds to each symbol in the string.

### Claim 13 (depends on 12)

The system of claim 12, wherein each vector of the first sequence of vectors is derived by joining together a first vector comprising the maximum length of word for a given symbol in the string and a second vector comprising a symbol vector for a given symbol in the string.

### Claim 14

(canceled)

### Claim 15 (depends on 14)

The system of claim 14, wherein the longest possible word is found in a dictionary.

### Claim 16 (depends on 13)

The system of claim 13, wherein the symbol vector for each symbol in the string is based on one or more of:
a symbol embedding;
a unified vector for symbols of an alphabet that is different from the alphabet used in the string;
a vector for frequently occurring punctuation marks;
a vector for rare punctuation marks; or
a vector for numerals.

### Claim 17 (depends on 11)

The system of claim 11, wherein each string of the plurality of strings differs from each other string of the plurality of strings by one symbol, the one symbol being located on a same position of each string.

### Claim 18

A non-transitory computer-readable storage medium storing instructions that, when executed by a processing device, cause the processing device to:
receive a plurality of strings, each string of the plurality of strings comprising a plurality of symbols;
for each string of the plurality of strings, generate a first sequence of vectors based at least on a maximum length of word for each symbol in the string, wherein the maximum length of word for each symbol in the string corresponds to length of a longest possible word within the string that starts with the symbol, the longest possible word comprising a word found in a dictionary;
provide to a machine learning unit the first sequence of vectors for each string of the plurality of strings; and
obtain from the machine learning unit a probability of occurrence of each string of the plurality of strings.

### Claim 19 (depends on 18)

The non-transitory computer-readable storage medium of claim 18, wherein the machine learning unit comprises:
a first fully connected layer and a second fully connected layer to apply matrix transformation on the first sequence of vectors for each string; and
a third fully connected layer to be used as an output layer.

### Claim 20 (depends on 19)

The non-transitory computer-readable storage medium of claim 19, wherein a batch normalization function and a rectifier linear unit activation function are applied on a first output of the first fully connected layer and on a second output of the second fully connected layer, and wherein a sigmoid activation function is applied on a third output of the third fully connected layer.
