# US 2023/0055477 A1 · Speech-enabled augmented reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0055477 A1
- Document ID: US20230055477A1
- Status: Published application
- Kind: A1
- Application number: 17/445,653
- Filed: 2021-08-23
- Published: 2023-02-23
- Earliest claimed: 2021-08-23
- Assignees: SoundHound, Inc.
- Applicants: SoundHound, Inc.
- Inventors: Keyvan MOHAJER; Morris MICHAEL; Bernard MONT-REYNAUD
- Web page: https://patentgrep.com/patent/US20230055477A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 18/24: Classification techniques
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 20/20: in augmented reality scenes
- G10L 15/08: Speech classification or search
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06T 2207/20092: Interactive image processing based on input by user
- G06V 20/64: Three-dimensional [3D] objects
- G06V 40/18: Eye characteristics, e.g. of the iris
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/16: using artificial neural networks
- G10L 2015/088: {Word spotting}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L 2015/226: {using non-speech characteristics}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Methods and systems for implementing an intuitive interaction between the user and the virtual content of augmented reality applications are disclosed. By implementing an augmented reality inquiry mode of a device, the system can enable a user to interact with relevant virtual objects via a speech-enabled interface. The speech-enabled augmented reality system can identify visual objects in images and recognize virtual objects corresponding to the visual objects, determine one or more relevant objects from the virtual objects based on relevance factors. Once the interaction session is established, a user can further interact with the relevant virtual objects, notably through voice commands addressed to the object. Accordingly, the present subject matter can enable a natural and hands-free interaction between the user and any virtual object that the user is interested in.

## Claims

### Claim 1

A computer-implemented method for implementing an AR inquiry mode, comprising:
receiving, by a camera of a device, an image;
recognizing one or more virtual objects in the image;
determining, based on a relevance score, a respective probability that a user will interact with the one or more virtual objects;
determining a relevant object based on the respective probability exceeding a predetermined threshold from the one or more virtual objects in the image;
overlying, in the image, text indicating a corresponding key phrase associated with the relevant object on a display of the device;
receiving speech audio from the user;
inferring a key phrase associated with the relevant object based on the speech audio; and
enabling an interaction session with the user, wherein the user can obtain information related to the relevant obj ect via a voice interface of the device.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
prior to receiving an image, receiving an explicit user input to activate the AR inquiry mode; and
activating the AR inquiry mode.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, further comprising:
initializing the AR inquiry mode by capturing the visual surroundings of the device with a camera of the device.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, further comprising:
prior to receiving an image, inferring, based on user input data, an implied user intention to activate the AR inquiry mode; and
activating the AR inquiry mode.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, further comprising:
initializing the AR inquiry mode by capturing the visual surroundings of the device with a camera of the device.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprising:
determining location data of the one or more virtual objects in the image.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising:
determining a respective type of the one or more virtual objects in the image; and
requesting data entries for the one or more virtual objects based on the respective type.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, further comprising:
requesting data entries for the one or more virtual objects; and
receiving a plurality of available data entries related to the relevant object.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, further comprising:
determining, based on the plurality of available data entries, one or more suggested queries; and
rendering, in the image, text indicating the one or more suggested queries on the display.

### Claim 10

(canceled)

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein the relevance score comprises one or more of the user's input, user's gesture data, location and/or position data of the relevant object and a predetermined relevancy designation.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, further comprises:
receiving, from an information provider, customized information related to the relevant object; and
providing the customized information to the user in the interaction session.

### Claim 13 (depends on 1)

The computer-implemented method of claim 1, wherein the corresponding key phrase is a predetermined wake-up phrase.

### Claim 14 (depends on 1)

The computer-implemented method of claim 1, wherein a rendering of the text indicating the corresponding key phrase is anchored to the relevant object in the image.

### Claim 15 (depends on 14)

The computer-implemented method of claim 14, wherein the image is dynamically updated by the camera, and wherein the rendering of the text indicating the corresponding key phrase is adjusted in real-time.

### Claim 16 (depends on 1)

The computer-implemented method of claim 1, further comprising:
tracking and reconstructing the one or more virtual objects via image processing by the device over a period of time.

### Claim 17 (depends on 1)

The computer-implemented method of claim 1, wherein enabling an interaction session with the user comprises:
receiving additional speech audio of a user;
inferring, by the speech recognition system, a query associated with the relevant object based on the additional speech audio;
determining, by the device, a response to the query, and
providing a response to the query via the voice interface of the device.

### Claim 18 (depends on 17)

The computer-implemented method of claim 17, wherein enabling an interaction session with the user comprises:
determining, by the speech recognition system, the query is ambiguous;
generating one or more disambiguating questions; and
providing the one or more disambiguating questions to the user.

### Claim 19

A computer-implemented method, comprising:
receiving, by a camera of a device, an image;
showing the image on a display of the device;
recognizing one or more virtual objects in the image;
determining, based on a relevance score, a respective probability that a user will interact with the one or more virtual objects;
determining a relevant object based on the respective probability exceeding a predetermined threshold from the one or more virtual objects in the image; and
overlaying, in the image, text indicating a corresponding key phrase associated with the relevant object on the display.

### Claim 20 (depends on 19)

The computer-implemented method of claim 19, further comprising:
determining location data of the one or more virtual objects in the image.

### Claim 21 (depends on 19)

The computer-implemented method of claim 19, further comprising:
determining a respective type of the one or more virtual objects in the image; and
requesting data entries for the one or more virtual objects based on the respective type.

### Claim 22 (depends on 19)

The computer-implemented method of claim 19, further comprising:
requesting data entries for the one or more virtual objects; and
receiving a plurality of available data entries related to the relevant object.

### Claim 23

(canceled)

### Claim 24 (depends on 19)

The computer-implemented method of claim 19, wherein the device comprises one of a smartphone, a smart car, smart glasses, and an AR headset.

### Claim 25 (depends on 19)

The computer-implemented method of claim 19, wherein the device is a smart car, and wherein the display is at least one of a head-up display of the smart car or a dashboard display.

### Claim 26

A computer system, comprising:
at least one processor;
a display;
at least one camera; and
memory including instructions that, when executed by the at least one processor, cause the computer system to:
receive, by the at least one camera, an image;
recognize one or more virtual objects in the image;
determining, based on a relevance score, a respective probability that a user will interact with the one or more virtual objects;
determine at least one relevant object based on the respective probability exceeding a predetermined threshold from the one or more virtual objects in the image;
overlay, in the image, text indicating a corresponding key phrase associated with the at least one relevant object on the display;
receive speech audio from the user;
infer a key phrase associated with a relevant object based on the speech audio; and
enable an interaction session with the user, wherein the user can obtain information related to the relevant object.

### Claim 27 (depends on 26)

The computer system of claim 26, further comprising instructions that, when executed by the at least one processor, cause the computer system to:
determine location data of the one or more virtual objects in the image.

### Claim 28 (depends on 26)

The computer system of claim 26, further comprising instructions that, when executed by the at least one processor, cause the computer system to:
request data entries for the one or more virtual objects; and
receive a plurality of available data entries related to the at least one relevant object.

### Claim 29

(canceled)

### Claim 30 (depends on 26)

The computer system of claim 26, wherein the relevance score comprises one or more of the user's input, user's gesture data, location and/or position data of the relevant object and a predetermined relevancy designation.

### Claim 31 (depends on 26)

The computer system of claim 26, further comprising instructions that, when executed by the at least one processor, cause the computer system to:
receive, from an information provider, customized information related to the at least one relevant object; and
provide the customized information to the user in the interaction session.
