# US 2023/0145198 A1 · Method for outputting text in artificial intelligence virtual assistant service and electronic device for supporting the same

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0145198 A1
- Document ID: US20230145198A1
- Status: Published application
- Kind: A1
- Application number: 17/296,463
- Filed: 2021-05-21
- Published: 2023-05-11
- Earliest claimed: 2020-05-22
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Kichul KIM; Yoonjae PARK; Jooyong BYEON; Youngkyu KIM; Byungkeon PARK; Soowon JANG; Changyong JEONG; Sungbin JIN; Jaeyung YEO
- Web page: https://patentgrep.com/patent/US20230145198A1

## CPC

- G06F 40/35: Discourse or dialogue representation
- G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units}
- G06F 40/117: Tagging; Marking up (details of markup languages G06F40/143); Designating a block; Setting of attributes (style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154)
- G06F 40/143: Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
- G06F 40/166: Editing, e.g. inserting or deleting
- G06F 40/232: Orthographic correction, e.g. spell checking or vowelisation
- G06F 40/30: Semantic analysis
- G06N 20/00: Machine learning
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 25/63: for estimating an emotional state
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

According to an embodiment, an electronic device comprises: a memory, a communication module comprising communication circuitry, and a processor operatively connected with the memory and the communication module. The processor is configured to control the electronic device to: obtain a utterance text corresponding to utterance speech, obtain an intent of the utterance text and emotion information based on the utterance speech and the utterance text, obtain a response text for the utterance text based on the intent of the utterance text and the emotion information, obtain a markup language including information about an output unit of text of the response text based on at least one of the intent of the utterance text, the emotion information, or the response text, and add the markup language to the response text and provide the response text. The text output unit is at least one selected from among a phoneme unit, a consonant and vowel unit, a syllable unit, or a word unit. A text output method in an artificial intelligence virtual assistant service of an electronic device may be performed using an artificial intelligence model.

## Claims

### Claim 1

An electronic device, comprising:
a memory;
a communication module comprising communication circuitry; and
a processor operatively connected with the memory and the communication module, wherein
the processor is configured to control the electronic device to:
obtain an utterance text corresponding to utterance speech;
obtain an intent of the utterance text and emotion information based on the utterance speech and the utterance text;
obtain a response text for the utterance text based on the intent of the utterance text and the emotion information;
obtain a markup language including information about an output unit of text of the response text based on at least one of the intent of the utterance text, the emotion information, or the response text; and
add the markup language to the response text and provide the response text, wherein
the output unit of text includes at least one selected from among a phoneme unit, a consonant and vowel unit, a syllable unit, or a word unit.

### Claim 2 (depends on 1)

The electronic device of claim 1, wherein the markup language further includes information about at least one of a text reproduction speed, a font, a font color, or a text reproduction pause.

### Claim 3 (depends on 1)

The electronic device of claim 1, wherein the processor is configured to control the electronic device to obtain the markup language including information about a speed of a text output corresponding to the response text based on a speed of a sound output corresponding to the response text.

### Claim 4 (depends on 1)

The electronic device of claim 1, wherein the markup language further includes information about an operation for displaying a typographical error in at least a portion of the response text and, based on deleting the typographical error, displaying at least the portion of the response text.

### Claim 5 (depends on 1)

The electronic device of claim 1, wherein the markup language is selected from among a plurality of markup languages stored in the memory, based on at least one of the response text, the intent of the utterance text, or the emotion information, wherein each of the plurality of markup languages match at least one of a plurality of response texts, intents of a plurality of utterance texts, or a plurality of pieces of emotion information, and wherein the processor is configured to control the electronic device to select a markup language corresponding to at least one of a response text, an intent of utterance text, or emotion information having a highest similarity to the response text, the intent of the utterance text, or the emotion information, based on none of the plurality of markup languages matching the response text, the intent of the utterance text, or the emotion information.

### Claim 6 (depends on 1)

The electronic device of claim 1, wherein the markup language matches the response text, and wherein the response text and the matched markup language are received from a developer terminal device and stored in the memory.

### Claim 7 (depends on 6)

The electronic device of claim 6, wherein the processor is configured to control the electronic device to:
in response to the response text being received from the developer terminal device, identify at least one response text having similarity to the response text of a predetermined value or more among a plurality of response texts stored in the memory;
provide at least one markup language used for the at least one response text among the plurality of markup languages stored in the memory, as the markup language for the response text, to the developer terminal device; and
in response to one of the at least one markup language being selected by the developer terminal device or a markup language corresponding to the response text is input, match the selected markup language or the input markup language with the response text and store the selected markup language of the input markup language in the memory.

### Claim 8 (depends on 1)

The electronic device of claim 1, wherein the memory includes a trained artificial intelligence (AI) model configured to: output the intent of the input text and emotion information using text and speech as input data, and wherein the processor is configured to control the electronic device to obtain the emotion information and the intent of the utterance text for the utterance speech and the utterance text using the trained AI model.

### Claim 9 (depends on 1)

The electronic device of claim 1, further comprising a display, wherein the processor is configured to control the electronic device to display the response text in the selected text output unit.

### Claim 10

A method of controlling an electronic device, the method comprising:
obtaining an utterance text corresponding to utterance speech;
obtaining an intent of the utterance text and emotion information based on the utterance speech and the utterance text;
obtaining a response text for the utterance text based on the intent of the utterance text and the emotion information;
obtaining a markup language including information about an output unit of text of the response text based on at least one of the intent of the utterance text, the emotion information, or the response text; and
adding the markup language to the response text and providing the response text, wherein the output unit of text is at least one selected from among a phoneme unit, a consonant and vowel unit, a syllable unit, or a word unit.

### Claim 11 (depends on 10)

The method of claim 10, wherein the markup language further includes information about at least one of a text reproduction speed, a font, a font color, a font size, or a text reproduction pause.

### Claim 12 (depends on 10)

The method of claim 10, wherein obtaining the markup language includes obtaining the markup language including information about a speed of a text output corresponding to the response text based on a speed of a sound output corresponding to the response text.

### Claim 13 (depends on 10)

The method of claim 10, wherein the markup language further includes information about an operation for displaying a typographical error in at least a portion of the response text and, based on deleting the typographical error, displaying at least the portion of the response text.

### Claim 14 (depends on 10)

The method of claim 10, wherein the markup language is selected from among a plurality of markup languages stored in a memory of the electronic device, based on at least one of the response text, the intent of the utterance text, or the emotion information, wherein each of the plurality of markup languages match at least one of a plurality of response texts, intents of a plurality of utterance texts, or a plurality of pieces of emotion information, and wherein obtaining the markup language includes selecting a markup language corresponding to at least one of a response text, an intent of utterance text, or emotion information having a highest similarity to the response text, the intent of the utterance text, or the emotion information, based on none of the plurality of markup languages matching the response text, the intent of the utterance text, or the emotion information.

### Claim 15 (depends on 14)

The method of claim 14, wherein the markup language matches the response text, and wherein the response text and the matched markup language are received from a developer terminal device and stored in the memory.

### Claim 16 (depends on 15)

The method of claim 15, further comprising:
in response to the response text being received from the developer terminal device, identifying at least one response text having a similarity to the response text being a predetermined value or more among a plurality of response texts stored in the memory;
providing at least one markup language used for the at least one response text among the plurality of markup languages stored in the memory, as the markup language for the response text, to the developer terminal device; and
in response to one of the at least one markup language being selected by the developer terminal device or a markup language corresponding to the response text being input, matching the selected markup language or the input markup language with the response text and storing them in the memory.

### Claim 17 (depends on 10)

The method of claim 10, wherein a memory of the electronic device includes a trained artificial intelligence (AI) model configured to output the intent of the input text and emotion information using text and speech as input data, and wherein the emotion information and the intent of the utterance text for the utterance speech and the utterance text are obtained using the trained AI model.

### Claim 18 (depends on 10)

The method of claim 10, further comprising displaying the response text in the selected text output unit, on a display of the electronic device.

### Claim 19

A terminal device, comprising:
a microphone;
a display;
a communication module comprising communication circuitry; and
a processor, wherein the processor is configured to control the terminal device to:
transmit utterance speech received through the microphone to an external server;
receive a response text corresponding to the utterance speech from the external server; and
display the response text on the display in at least one text output unit selected from among a phoneme unit, a consonant and vowel unit, a syllable unit, or a word unit, based on a markup language including information about the text output unit added to the response text.

### Claim 20 (depends on 19)

The terminal device of claim 19, wherein the markup language further includes at least one of a text reproduction speed, a font, a font color, a font size, or a text reproduction pause.
