# US 12,112,139 B2 · Vocabulary generation for neural machine translation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,139 B2
- Document ID: US12112139B2
- Status: Granted
- Kind: B2
- Application number: 17/535,365
- Filed: 2021-11-24
- Published: 2023-05-25
- Granted: 2024-10-08
- Earliest claimed: 2021-11-24
- Assignees: Beijing Youzhuju Network Technology Co., Ltd.
- Applicants: Beijing Youzhuju Network Technology Co. Ltd.
- Inventors: Jingjing Xu; Chun Gan; Hao Zhou; Lei Li; Zaixiang Zheng
- Web page: https://patentgrep.com/patent/US12112139B2

## CPC

- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G06F 40/237: Lexical tools
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Implementations of the present disclosure relate to methods, devices, and computer program products for generating a destination vocabulary from a source vocabulary. In a method, a group of candidate vocabularies are determined from the source vocabulary based on a corpus, a size of a candidate vocabulary in the group of candidate vocabularies being different from a size of the source vocabulary. A group of marginal scores are obtained for the group of candidate vocabularies, respectively, a marginal score in the group of marginal scores being obtained for the candidate vocabulary based on a corpus entropy of the candidate vocabulary and a size of the candidate vocabulary. The destination vocabulary is selected from the group of candidate vocabularies based on the group of marginal scores. With these implementations, both of the corpus entropy and the vocabulary size are considered in the vocabulary generation, and thus a balance may be achieved therebetween, which may increase the performance of the generated vocabulary.

## Claims

### Claim 1

A method for generating a destination vocabulary by a machine learning model, comprising:
inputting a sequence of token candidates and data indicative of a training corpus into the machine learning model, the sequence of token candidates generated based on a source vocabulary, and the training corpus comprising texts in at least one language;
generating a group of candidate vocabularies at a plurality of timesteps by the machine learning model, the machine learning model is configured to generate optimal vocabularies with a computational efficiency by balancing a corpus entropy and a vocabulary size, wherein a size of a candidate vocabulary in the group of candidate vocabularies is different from a size of the source vocabulary;
computing a group of marginal scores corresponding to the group of candidate vocabularies, respectively, wherein a marginal score in the group of marginal scores corresponding to a candidate vocabulary in the group of candidate vocabularies is computed based on a corpus entropy of the candidate vocabulary and a size of the candidate vocabulary, wherein computing the marginal score comprises computing a negative derivation of the corpus entropy to the size of the candidate vocabulary, and wherein the computing a negative derivation of the corpus entropy to the size of the candidate vocabulary further comprises:
computing an entropy difference between the corpus entropy and a previous corpus entropy of a previous vocabulary, and
computing the negative derivation based on the entropy difference and a predefined step length; and
selecting the destination vocabulary from the group of candidate vocabularies based on the group of marginal scores.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating the group of candidate vocabularies comprises:
obtaining a plurality of timesteps for determining the group of candidate vocabularies based on a predefined step length and the size of the source vocabulary;
determining at least one candidate vocabulary in the group of candidate vocabularies in a timestep of the plurality of timesteps, a size of the at least one candidate vocabularies being different from the size of the source vocabulary by the predefined step length.

### Claim 3 (depends on 2)

The method of claim 2, wherein determining the at least one candidate vocabulary in the timestep comprises:
selecting a subset of tokens from the source vocabulary based on frequency for tokens included in the source vocabulary occurring in the corpus; and
obtaining the at least one candidate vocabulary by any of:
adding, into a token in the subset of tokens, at least one character included in the corpus; or
removing, from the subset of tokens, at least one character included in the corpus.

### Claim 4 (depends on 3)

The method of claim 3, wherein obtaining the at least one candidate vocabulary comprises:
obtaining a character set that includes all characters in the corpus;
generating a transport matrix for determining the least one candidate vocabulary, a first dimension in the transport matrix corresponding to the subset of tokens and a second dimension in the transport matrix corresponding to characters in the character set, and an element at a position in the transport matrix indicating a probability for a character corresponding to the position being transported to a token corresponding to the position; and
obtaining the at least one candidate vocabulary based on the transport matrix and a previous vocabulary that is obtained in a previous timestep before the timestep.

### Claim 5 (depends on 4)

The method of claim 4, wherein constrains for the transport matrix comprise at least one of:
a sum of elements in the second dimension corresponds to a probability of a character in the character set; and
a sum of a difference between the element and a probability for a token in the subset of tokens being below a predefined threshold.

### Claim 6 (depends on 4)

The method of claim 4, wherein determining the group of candidate vocabularies further comprises: removing a token from the candidate vocabulary in response to an element in the transport matrix corresponding to the token being below a predefined threshold.

### Claim 7 (depends on 1)

The method of claim 1, wherein selecting the destination vocabulary comprises:
selecting a plurality of extreme candidate vocabularies for the plurality of timesteps respectively, an extreme candidate vocabulary in the plurality of extreme candidate vocabularies having an extreme corpus entropy among the at least one candidate vocabulary that is obtained in the timestep;
selecting an extreme candidate vocabulary from the plurality of extreme candidate vocabularies, a difference between a corpus entropy of the selected extreme candidate vocabulary and a corpus entropy of a previous extreme candidate vocabulary that is obtained in a previous timestep having an extreme value; and
identifying the selected extreme candidate vocabulary as the destination vocabulary.

### Claim 8 (depends on 7)

The method of claim 7, wherein selecting the plurality of extreme candidate vocabularies comprises: selecting an extreme candidate vocabulary for the timestep by,
generating a distance matrix corresponding to the transport matrix, a first dimension in the distance matrix corresponding to the subset of tokens and a second dimension in the distance matrix corresponding to characters in the character set, and an element at a position in the distance matrix indicating a distance for transporting a character corresponding to the position to a token corresponding to the position; and
determining an extreme corpus entropy for the timestep based on the transport matrix and the distance matrix; and
selecting a candidate vocabulary that has the extreme corpus entropy as the extreme candidate vocabulary for the timestep.

### Claim 9 (depends on 1)

The method of claim 1, wherein the corpus entropy of the candidate vocabulary is determined by:
obtaining frequency for tokens in the candidate vocabulary occurring in the corpus;
determining an average length of tokens in the candidate vocabulary; and
obtaining the corpus entropy based on the frequency and the average length.

### Claim 10

An electronic device, comprising a computer processor coupled to a computer-readable memory unit, the memory unit comprising instructions that when executed by the computer processor implements operations comprising:
inputting a sequence of token candidates and data indicative of a training corpus into the machine learning model, the sequence of token candidates generated based on a source vocabulary, and the training corpus comprising texts in at least one language;
generating a group of candidate vocabularies at a plurality of timesteps by the machine learning model, the machine learning model is configured to generate optimal vocabularies with a computational efficiency by balancing a corpus entropy and a vocabulary size, wherein a size of a candidate vocabulary in the group of candidate vocabularies is different from a size of the source vocabulary;
computing a group of marginal scores corresponding to the group of candidate vocabularies, respectively, wherein a marginal score in the group of marginal scores corresponding to a candidate vocabulary in the group of candidate vocabularies is computed based on a corpus entropy of the candidate vocabulary and a size of the candidate vocabulary, wherein computing the marginal score comprises computing a negative derivation of the corpus entropy to the size of the candidate vocabulary, and wherein the computing a negative derivation of the corpus entropy to the size of the candidate vocabulary further comprises:
computing an entropy difference between the corpus entropy and a previous corpus entropy of a previous vocabulary, and
computing the negative derivation based on the entropy difference and a predefined step length; and
selecting the destination vocabulary from the group of candidate vocabularies based on the group of marginal scores.

### Claim 11 (depends on 10)

The device of claim 10, wherein generating the group of candidate vocabularies comprises:
obtaining a plurality of timesteps for determining the group of candidate vocabularies based on a predefined step length and the size of the source vocabulary;
determining at least one candidate vocabulary in the group of candidate vocabularies in a timestep of the plurality of timesteps, a size of the at least one candidate vocabularies being different from the size of the source vocabulary by the predefined step length.

### Claim 12 (depends on 11)

The device of claim 11, wherein determining the at least one candidate vocabulary in the timestep comprises:
selecting a subset of tokens from the source vocabulary based on frequency for tokens included in the source vocabulary occurring in the corpus; and
obtaining the at least one candidate vocabulary by any of:
adding, into a token in the subset of tokens, at least one character included in the corpus; or
removing, from the subset of tokens, at least one character included in the corpus.

### Claim 13 (depends on 12)

The device of claim 12, wherein obtaining the at least one candidate vocabulary comprises:
obtaining a character set that includes all characters in the corpus;
generating a transport matrix for determining the least one candidate vocabulary, a first dimension in the transport matrix corresponding to the subset of tokens and a second dimension in the transport matrix corresponding to characters in the character set, and an element at a position in the transport matrix indicating a probability for a character corresponding to the position being transported to a token corresponding to the position; and
obtaining the at least one candidate vocabulary based on the transport matrix and a previous vocabulary that is obtained in a previous timestep before the timestep.

### Claim 14 (depends on 13)

The device of claim 13, wherein constrains for the transport matrix comprise at least one of:
a sum of elements in the second dimension corresponds to a probability of a character in the character set; and
a sum of a difference between the element and a probability for a token in the subset of tokens being below a predefined threshold.

### Claim 15 (depends on 10)

The device of claim 10, wherein selecting the destination vocabulary comprises:
selecting a plurality of extreme candidate vocabularies for the plurality of timesteps respectively, an extreme candidate vocabulary in the plurality of extreme candidate vocabularies having an extreme corpus entropy among the at least one candidate vocabulary that is obtained in the timestep;
selecting an extreme candidate vocabulary from the plurality of extreme candidate vocabularies, a difference between a corpus entropy of the selected extreme candidate vocabulary and a corpus entropy of a previous extreme candidate vocabulary that is obtained in a previous timestep having an extreme value; and
identifying the selected extreme candidate vocabulary as the destination vocabulary.

### Claim 16 (depends on 15)

The device of claim 15, wherein selecting the plurality of extreme candidate vocabularies comprises: selecting an extreme candidate vocabulary for the timestep by,
generating a distance matrix corresponding to the transport matrix, a first dimension in the distance matrix corresponding to the subset of tokens and a second dimension in the distance matrix corresponding to characters in the character set, and an element at a position in the distance matrix indicating a distance for transporting a character corresponding to the position to a token corresponding to the position; and
determining an extreme corpus entropy for the timestep based on the transport matrix and the distance matrix; and
selecting a candidate vocabulary that has the extreme corpus entropy as the extreme candidate vocabulary for the timestep.

### Claim 17 (depends on 10)

The device of claim 10, wherein generating the group of candidate vocabularies further comprises: removing a token from the candidate vocabulary in response to an element in a transport matrix corresponding to the token being below a predefined threshold.

### Claim 18

A computer program product, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by an electronic device to cause the electronic device to perform operations comprises:
inputting a sequence of token candidates and data indicative of a training corpus into the machine learning model, the sequence of token candidates generated based on a source vocabulary, and the training corpus comprising texts in at least one language;
generating a group of candidate vocabularies at a plurality of timesteps by the machine learning model, the machine learning model is configured to generate optimal vocabularies with a computational efficiency by balancing a corpus entropy and a vocabulary size, wherein a size of a candidate vocabulary in the group of candidate vocabularies is different from a size of the source vocabulary;
computing a group of marginal scores corresponding to the group of candidate vocabularies, respectively, wherein a marginal score in the group of marginal scores corresponding to a candidate vocabulary in the group of candidate vocabularies is computed based on a corpus entropy of the candidate vocabulary and a size of the candidate vocabulary, wherein computing the marginal score comprises computing a negative derivation of the corpus entropy to the size of the candidate vocabulary, and wherein the computing a negative derivation of the corpus entropy to the size of the candidate vocabulary further comprises:
computing an entropy difference between the corpus entropy and a previous corpus entropy of a previous vocabulary, and
computing the negative derivation based on the entropy difference and a predefined step length; and
selecting the destination vocabulary from the group of candidate vocabularies based on the group of marginal scores.