# US 2025/0094740 A1 · Augmented reality speech balloon system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094740 A1
- Document ID: US20250094740A1
- Status: Published application
- Kind: A1
- Application number: 18/969,990
- Filed: 2024-12-05
- Published: 2025-03-20
- Earliest claimed: 2017-02-20
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Piers Cowburn; Qi Pan; Eitan Pilipski
- Web page: https://patentgrep.com/patent/US20250094740A1

## CPC

- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06F 40/205: Parsing
- G06F 40/30: Semantic analysis
- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 20/20: in augmented reality scenes
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G10L 15/25: using position of the lips, movement of the lips or face analysis
- G10L 21/10: Transforming into visible information
- G10L 25/63: for estimating an emotional state
- G06V 40/175: {Static expression}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Disclosed is an augmented reality system to generate and cause display of an augmented reality interface at a client device. Various embodiments may detect speech, identify a source of the speech, transcribe the speech to a text string, generate a speech bubble based on properties of the speech and that includes a presentation of the text string, and cause display of the speech bubble at a location in the augmented reality interface based on the source of the speech.

## Claims

### Claim 1

A system comprising:
a memory; and
at least one hardware processor coupled to the memory and comprising instructions that causes the system to perform operations comprising:
causing display at a client device of a presentation of image data, the presentation of the image data comprising a depiction of a set of facial features;
detecting ambient sound containing speech;
extracting speech properties from the ambient sound containing speech;
detecting movement of one or more facial features from the set of facial features;
determining an emotional effect of the speech based on the movement of the one or more facial features and the speech properties;
selecting a speech bubble from a speech bubble library based on the emotional effect of the speech; and
causing display of the speech bubble within the presentation of the image data.

### Claim 2 (depends on 1)

The system of claim 1, wherein the presentation of the image data corresponds with the ambient sound containing speech, and determining the emotional effect is based on a volume of the speech.

### Claim 3 (depends on 1)

The system of claim 1, wherein the ambient sound includes a non-verbal sound, and the operations further comprise:
comparing the non-verbal sound to an onomatopoeia library in response to the detecting ambient sound containing speech;
identifying an onomatopoeia from the onomatopoeia library based on the non-verbal sound; and
selecting the speech bubble based on at least the onomatopoeia.

### Claim 4 (depends on 2)

The system of claim 2, wherein the movement of the one or more facial features corresponds with a source within the presentation of the image data, and the operations further comprise:
identifying the source of the movement of the one or more facial features at a position within the presentation of the image data; and
causing display of the speech bubble based on the position of the source of the movement of the one or more facial features.

### Claim 5 (depends on 2)

The system of claim 2, wherein the determining the emotional effect further comprises:
generating a transcript based on the movement of the one or more facial features and the speech properties; and
determining the emotional effect based on the transcript.

### Claim 6 (depends on 2)

The system of claim 2, wherein the determining the emotional effect further comprises:
generating a transcript based on the movement of one or more facial features and the speech properties; and
wherein the speech bubble includes the transcript.

### Claim 7 (depends on 6)

The system of claim 6, wherein the speech bubble comprises a set of graphical properties, the graphical properties based on the emotional effect and the length of the transcript.

### Claim 8

A method comprising:
causing display at a client device of a presentation of image data, the presentation of the image data comprising a depiction of a set of facial features;
detecting ambient sound containing speech;
extracting speech properties from the ambient sound containing speech;
detecting movement of one or more facial features from the set of facial features;
determining an emotional effect of the speech based on the movement of the one or more facial features and the speech properties;
selecting a speech bubble from a speech bubble library based on the emotional effect of the speech; and
causing display of the speech bubble within the presentation of the image data.

### Claim 9 (depends on 8)

The method of claim 8, wherein the presentation of the image data corresponds with the ambient sound containing speech, and the determining the emotional effect is based on and a volume of the speech.

### Claim 10 (depends on 9)

The method of claim 9, wherein the ambient sound includes a non-verbal sound, and the method further comprises:
comparing the non-verbal sound to an onomatopoeia library in response to the detecting the ambient sound containing speech;
identifying an onomatopoeia from the onomatopoeia library based on the non-verbal sound; and
selecting the speech bubble based on at least the onomatopoeia.

### Claim 11 (depends on 9)

The method of claim 9, wherein the movement of the one or more facial features corresponds with a source within the presentation of the image data, and the method further comprises:
identifying the source of the movement of the one or more facial features at a position within the presentation of the image data; and
causing display of the speech bubble based on the position of the source of the movement of the one or more facial features.

### Claim 12 (depends on 9)

The method of claim 9, wherein the determining the emotional effect further comprises:
generating a transcript based on the movement of the one or more facial features and the speech properties; and
determining the emotional effect based on the transcript.

### Claim 13 (depends on 9)

The method of claim 9, wherein the determining the emotional effect further comprises:
generating a transcript based on the movement of the one or more facial features; and
wherein the speech bubble includes the transcript.

### Claim 14 (depends on 13)

The method of claim 13, wherein the speech bubble comprises a set of graphical properties, the graphical properties based on the emotional effect and the length of the transcript.

### Claim 15

A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations including:
causing display at a client device of a presentation of image data, the presentation of the image data comprising a depiction of a set of facial features;
detecting ambient sound containing speech;
extracting speech properties from the ambient sound containing speech;
detecting movement of one or more facial features from the set of facial features;
determining an emotional effect of the speech based on the movement of the one or more facial features and the speech properties;
selecting a speech bubble from a speech bubble library based on the emotional effect of the speech; and
causing display of the speech bubble within the presentation of the image data.

### Claim 16 (depends on 15)

The non-transitory machine-readable storage medium of claim 15, wherein the presentation of the image data corresponds with the ambient sound containing speech, and wherein the determining the emotional effect is based on a volume of the speech.

### Claim 17 (depends on 16)

The non-transitory machine-readable storage medium of claim 16, wherein the ambient sound includes a non-verbal sound, and the operations further comprise:
comparing the non-verbal sound to an onomatopoeia library in response to the detecting the ambient sound containing speech;
identifying an onomatopoeia from the onomatopoeia library based on the non-verbal sound; and
selecting the speech bubble based on at least the onomatopoeia.

### Claim 18 (depends on 16)

The non-transitory machine-readable storage medium of claim 16, wherein the movement of the one or more facial features corresponds with a source within the presentation of the image data, and the operations further comprise:
identifying the source of the movement of the one or more facial features at a position within the presentation of the image data; and
causing display of the speech bubble based on the position of the source of the movement of the one or more facial features.

### Claim 19 (depends on 16)

The non-transitory machine-readable storage medium of claim 16, wherein the determining the emotional effect further comprises:
generating a transcript based on the movement of the one or more facial features and the speech properties; and
determining the emotional effect based on the transcript.

### Claim 20 (depends on 16)

The non-transitory machine-readable storage medium of claim 16, wherein the determining the emotional effect further comprises:
generating a transcript based on the movement of the one or more facial features and the speech properties; and
wherein the speech bubble includes the transcript.