# US 2023/0274100 A1 · Techniques and Models for Multilingual Text Rewriting

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274100 A1
- Document ID: US20230274100A1
- Status: Published application
- Kind: A1
- Application number: 17/682,282
- Filed: 2022-02-28
- Published: 2023-08-31
- Earliest claimed: 2022-02-28
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Xavier Eduardo Garcia; Orhan Firat; Noah Constant; Xiaoyue Guo
- Web page: https://patentgrep.com/patent/US20230274100A1

## CPC

- G06F 40/30: Semantic analysis
- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G06F 40/166: Editing, e.g. inserting or deleting
- G06F 40/197: Version control (for software G06F8/71)
- G06F 40/253: Grammatical analysis; Style critique
- G06F 40/44: Statistical methods, e.g. probability models
- G06F 40/51: Translation evaluation
- G06F 40/56: Natural language generation
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/096: Transfer learning
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The technology provides a model-based approach for multilingual text rewriting that is applicable across many languages and across different styles including formality levels or other textual attributes. The model is configured to manipulate both language and textual attributes jointly. This approach supports zero-shot formality-sensitive translation, with no labeled data in the target language. An encoder-decoder architectural approach with attribute extraction is used to train rewriter models that can thus be used in “universal” textual rewriting across many different languages. A cross-lingual learning signal can be incorporated into the training approach. Certain training processes do not employ any exemplars. This approach enables not just straight translation, but also the ability to create new sentences with different attributes.

## Claims

### Claim 1

A system configured for a multilingual text rewriting model, the system comprising:
memory configured to store a set of text exemplars in a source language and a set of rewritten texts in a plurality of languages different from the source language; and
one or more processing elements operatively coupled to the memory, the one or more processing elements implementing a multilingual text rewriter as a neural network having:
a corruption module configured to generate a corrupted version of an input text sequence based on the set of text exemplars in the source language;
an encoder module comprising an encoder neural network configured to receive the corrupted version of the input text sequence and to generate a set of encoded representations of the corrupted version of the input text sequence;
a style extractor module configured to extract a set of style vector representations associated with the input text sequence; and
a decoder module comprising a decoder neural network configured to receive the set of encoded representations of the corrupted version of the input text sequence and the set of style vector representations and to output the set of rewritten texts in the plurality of languages, each style vector representation of the set being added element-wise to one of the set of encoded representations of the corrupted version of the input text sequence;
wherein a set of model weights is shared by the encoder module and the style extractor module, and a unique token is appended to the input text sequence for style extraction instead of mean-pooling all of the style vector representations in the set; and
wherein the system is configured to provide rewritten text in selected ones of the plurality of languages according to a change in a least a sentiment or a formality of the input text sequence.

### Claim 2 (depends on 1)

The system of claim 1, wherein the encoder module, the style extractor module and the decoder module are configured as transformer stacks initialized from a common pretrained language model.

### Claim 3 (depends on 1)

The system of claim 1, wherein both the encoder module and the decoder module are attention-based neural network modules.

### Claim 4 (depends on 1)

The system of claim 1, wherein the corruption module employs a corruption function C for a given pair of non-overlapping spans (s1, s2) of the input text sequence, so that the model is capable of reconstructing span s2 from C(s2) and Vs1, where Vs1 is an attribute vector of span s1.

### Claim 5 (depends on 4)

The system of claim 4, wherein the corruption function C is as follows:
C:=ƒ(⋅, source language, −Vs1).

### Claim 6 (depends on 1)

The system of claim 1, wherein the style extractor module includes a set of style extractor elements including a first subset configured to operate on different style exemplars and a second subset configured to operate on the input text sequence prior to corruption by the corruption module.

### Claim 7 (depends on 1)

The system of claim 1, wherein during training the corruption module is configured to employ at least one of token-level corruption or style-aware back-translation corruption.

### Claim 8 (depends on 1)

The system of claim 1, wherein the set of model weights is initialized with the weights of a pretrained text-to-text model.

### Claim 9 (depends on 1)

The system of claim 1, wherein the model weights for the style extractor module are not tied to the weights of the encoder module during training.

### Claim 10 (depends on 1)

The system of claim 1, wherein the system is configured to extract pairs of random non-overlapping spans of tokens from each line of text in a given text exemplar, and configured to use a first-occurring span in the text as an exemplar of attributes of a second span in the text.

### Claim 11 (depends on 1)

The system of claim 1, wherein a cross-lingual learning signal is added to a training objective for training the model.

### Claim 12 (depends on 1)

The system of claim 1, wherein the encoder module is configured to use negation of a true exemplar vector associated with the input text sequence as an attribute vector in a forward pass operation.

### Claim 13 (depends on 1)

The system of claim 1, wherein the decoder module is configured to receive a set of stochastic tuning ranges that provide conditioning for the decoder module.

### Claim 14 (depends on 1)

The system of claim 1, wherein:
the set of style vector representations corresponds to a set of attributes associated with the input text sequence; and
the system is configured to input a set of exemplars illustrating defined attributes and to extract corresponding attribute vectors for use in rewriting the input text sequence into the plurality of languages according to the defined attributes.

### Claim 15 (depends on 14)

The system of claim 14, wherein:
the system is configured to form an attribute delta vector including a scale factor; and
the attribute delta vector is added to the set of encoded representations of the corrupted version of the input text sequence before processing by the decoder module.

### Claim 16

A computer-implemented method for providing multilingual text rewriting according to a machine learning model, the method comprising:
obtaining an input text sequence based on a set of text exemplars in a source language;
generating, by a corruption module, a corrupted version of the input text sequence;
receiving, by an encoder neural network of an encoder module, the corrupted version of the input text sequence;
generating, by the encoder module, a set of encoded representations of the corrupted version of the input text sequence;
extracting, by a style extractor module, a set of style vector representations associated with the input text sequence;
receiving, by a decoder neural network of a decoder module, the set of encoded representations of the corrupted version of the input text sequence and the set of style vector representations, in which each style vector representation of the set is added element-wise to one of the set of encoded representations of the corrupted version of the input text sequence;
outputting, by the decoder module, a set of rewritten texts in a plurality of languages different from the source language; and
storing the rewritten text in selected ones of the plurality of languages according to a change in a least a sentiment or a formality of the input text sequence;
wherein a set of model weights is shared by the encoder module and the style extractor module, and a unique token is appended to the input text sequence for style extraction instead of mean-pooling all of the style vector representations in the set.

### Claim 17 (depends on 16)

The method of claim 16, wherein generating the corrupted version of the input text sequence is performed according to a corruption function C for a given pair of non-overlapping spans (s1, s2) of the input text sequence, so that the model is capable of reconstructing span s2 from C(s2) and Vs1, where Vs1 is an attribute vector of span s1.

### Claim 18 (depends on 16)

The method of claim 16, further comprising:
extracting pairs of random non-overlapping spans of tokens from each line of text in a given text exemplar; and
using a first-occurring span in the text as an exemplar of attributes of a second span in the text.

### Claim 19 (depends on 16)

The method of claim 16, further comprising adding a cross-lingual learning signal to a training objective for training the model.

### Claim 20 (depends on 16)

The method of claim 16, further comprising applying a set of stochastic tuning ranges to selectively condition the decoder module.

### Claim 21 (depends on 16)

The method of claim 16, wherein:
the set of style vector representations corresponds to a set of attributes associated with the input text sequence; and
outputting the set of rewritten texts includes generating one or more versions of the input text sequence in selected ones of the plurality of languages according to the set of attributes.

### Claim 22

A system comprising:
a memory configured to store a set of text exemplars in a source language; and
one or more processors operatively coupled to the memory, the one or more processors configured to:
generate a corrupted version of an input text sequence based on the set of text exemplars in the source language;
generate a set of encoded representations of a corrupted version of the input text sequence;
extract a set of style vector representations associated with the input text sequence;
generate a set of rewritten texts in a plurality of languages different from the source language based on the set of encoded representations and the set of style vector representations; and
store, in the memory, the set of rewritten texts in selected ones of the plurality of languages based on one or more changes associated with the input text sequence.

### Claim 23

A system comprising:
a memory configured to store a set of text exemplars in a source language;
an encoder configured to receive a corrupted version of an input text sequence based on the set of text exemplars in the source language and generate a set of encoded representations of the corrupted version of the input text sequence; and
a decoder operably coupled to the encoder, the decoder configured to:
receive the set of encoded representations of the corrupted version of the input text sequence and a set of style vector representations associated with the input text sequence, after each style vector representation is added element-wise to a respective encoded representation of the set of encoded representations, and
output a set of rewritten texts in a plurality of languages different from the source language based on the set of encoded representations and the set of style vector representations,
wherein the system is configured to provide rewritten texts in selected ones of the plurality of languages based on one or more changes associated with the input text sequence.

### Claim 24 (depends on 23)

The system of claim 23, wherein the memory is configured to store the rewritten texts in selected ones of the plurality of languages.

### Claim 25 (depends on 23)

The system of claim 23, wherein the one or more changes include a change in at least a sentiment or a formality of the input text sequence.
