# US 2025/0095631 A1 · Position-based text-to-speech model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0095631 A1
- Document ID: US20250095631A1
- Status: Published application
- Kind: A1
- Application number: 18/528,116
- Filed: 2023-12-04
- Published: 2025-03-20
- Earliest claimed: 2023-09-18
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Puneet Mathur; Franck Dernoncourt; Quan Hung Tran; Jiuxiang Gu; Ani Nenkova; Vlad Ion Morariu; Rajiv Bhawanji Jain; Dinesh Manocha
- Web page: https://patentgrep.com/patent/US20250095631A1

## CPC

- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G10L 13/04: Details of speech synthesis systems, e.g. synthesiser structure or memory management
- G10L 13/047: Architecture of speech synthesisers
- G10L 13/06: Elementary speech units used in speech synthesisers; Concatenation rules
- G10L 25/18: the extracted parameters being spectral information of each sub-band
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Position-based text-to-speech model and training techniques are described. A digital document, for instance, is received by an audio synthesis service. A text-to-speech model is utilized by the audio synthesis service to generate digital audio from text included in the digital document. The text-to-speech model, for instance, is configured to generate a text encoding and a document positional encoding from an initial text sequence of the digital document. The document positional encoding is based on a location of the text encoding within the digital document. Digital audio is then generated by the text-to-speech model that includes a spectrogram having a reordered text sequence, which is different from the initial text sequence, by decoding the text encoding and the document positional encoding.

## Claims

### Claim 1

A method comprising:
receiving, by a processing device, a digital document having text arranged in an initial text sequence;
generating, by the processing device, a text encoding and a document positional encoding from the digital document, the document positional encoding is based on a location of the text encoding within the digital document; and
generating, by the processing device, digital audio including a spectrogram having a reordered text sequence, which is different from the initial text sequence, by decoding the text encoding and the document positional encoding.

### Claim 2 (depends on 1)

The method as described in claim 1, wherein the document positional encoding is based on coordinates defined in relation to a page of the digital document.

### Claim 3 (depends on 1)

The method as described in claim 1, wherein the document positional encoding is based on a bounding box defined for the text.

### Claim 4 (depends on 3)

The method as described in claim 3, wherein the document positional encoding includes four two-dimensional positional encoding defining a relative spatial position of the text within the digital document.

### Claim 5 (depends on 1)

The method as described in claim 1, wherein the generating includes embedding the document positional encoding as part of the text encoding.

### Claim 6 (depends on 1)

The method as described in claim 1, wherein:
the generating the text encoding and the document position encoding is performed by a text layout encoder of a text-to-speech model using machine learning; and
the generating the digital audio including the spectrogram having the reordered text sequence is performed using a reading sequence decoder of the text-to-speech model using machine learning.

### Claim 7 (depends on 6)

The method as described in claim 6, wherein the text-to-speech model is trained using curriculum learning.

### Claim 8 (depends on 6)

The method as described in claim 6, wherein the generating the text encoding and the document position encoding is performed jointly by the text layout encoder.

### Claim 9 (depends on 6)

The method as described in claim 6, wherein the generating the digital audio including the spectrogram having the reordered text sequence is performed jointly using the reading sequence decoder.

### Claim 10 (depends on 1)

The method as described in claim 1, wherein the generating the text encoding further comprises generating a text sequence positional encoding as part of the text encoding, the text sequence positional encoding defining a position of the text encoding within a text sequence of the digital document.

### Claim 11 (depends on 1)

The method as described in claim 1, wherein the generating includes converting the text from the digital document into a phoneme and wherein the text encoding is generated based on the phoneme.

### Claim 12 (depends on 1)

The method as described in claim 1, wherein the generating the digital audio includes classifying whether the document position encoding indicates a break in the digital document.

### Claim 13

A system comprising:
a text-to-phoneme converter module implemented by a processing device to convert text in a digital document into a plurality of phonemes; and
a text-to-speech model implemented by the processing device to convert the plurality of phonemes into digital audio using machine learning, the text-to-speech model including:
a text layout encoder to generate a plurality of text encodings based on the plurality of phonemes using machine learning, the plurality of text encodings having embedded, respectively, a document positional encoding based on a location of a respective said text encoding within the digital document; and
a reading sequence decoder to decode the plurality of text encodings into the digital audio.

### Claim 14 (depends on 13)

The system as described in claim 13, wherein the reading sequence decoder is configured to generate reordered text sequence in the digital audio which is different from an initial text sequence of the plurality of phonemes.

### Claim 15 (depends on 14)

The system as described in claim 14, wherein the reading sequence decoder is configured to generate the digital audio as including a spectrogram having the reordered text sequence.

### Claim 16 (depends on 13)

The system as described in claim 13, wherein the document positional encoding is based on coordinates defined in relation to a page of the digital document.

### Claim 17 (depends on 13)

The system as described in claim 13, wherein the text layout encoder is further configured to generate a text sequence positional encoding as part of the text encoding, the text sequence positional encoding defining a position of the text encoding within a text sequence of the digital document.

### Claim 18 (depends on 13)

The system as described in claim 13, wherein the reading sequence decoder is further configured as a classifier to determine whether a respective said document positional encoding associated with a respective said text encoding indicates a break in the digital document.

### Claim 19

One or more computer readable storage media storing instructions that, responsive to execution by a processing device, causes the processing device to perform operations including:
receiving a digital document having text; and
generating digital audio based on the digital document, the digital audio including a spectrogram having a reading order generated jointly by a text layout encoder and a reading order sequence decoder of a text-to-speech model.

### Claim 20 (depends on 19)

The one or more computer readable storage media as described in claim 19, wherein the text-to-speech model is trained using curriculum learning.