# US 2022/0044677 A1 · Efficient and low latency automated assistant control of smart devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044677 A1
- Document ID: US20220044677A1
- Status: Published application
- Kind: A1
- Application number: 17/506,809
- Filed: 2021-10-21
- Published: 2022-02-10
- Earliest claimed: 2019-10-15
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: David Roy Schairer; Di Lin; Lucas Palmer
- Web page: https://patentgrep.com/patent/US20220044677A1

## CPC

- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G05B 19/042: using digital processors (G05B19/05 takes precedence)
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L 2015/226: {using non-speech characteristics}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

Various implementations relate to techniques, for controlling smart devices, that are low latency and/or that provide computational efficiencies (client and/or server) and/or network efficiencies. Those implementations relate to generating and/or utilizing cache entries, of a cache that is stored locally at an assistant client device, in control of various smart devices (e.g., smart lights, smart thermostats, smart plugs, smart appliances, smart routers, etc.). Each of the cache entries includes a mapping of text to one or more corresponding semantic representations.

## Claims

### Claim 1

A method implemented by one or more processors of a client device, the method comprising:
storing, in a cache on the client device: a cache entry that includes a mapping of text to a semantic representation, wherein the semantic representation includes a locally interpretable semantic representation that is locally interpretable by the client device;
capturing, at the client device, audio data that captures a spoken utterance;
processing, using a voice-to-text model stored locally on the client device, the audio data to generate current text that corresponds to the spoken utterance;
determining that the current text matches the text of the cache entry; and
in response to determining that the current text matches the text of the cache entry, and in response to the cache entry including the mapping of the text to the semantic representation:
processing, at the client device, the locally interpretable semantic representation to generate a given control command, the given control command differing from the locally interpretable semantic representation; and
transmitting, via a local channel, the given control command to cause the altering of at least the given state of the given smart device.

### Claim 2 (depends on 1)

The method of claim 1, wherein the locally interpretable semantic representation comprises:
an identifier of the given smart device; and
a current-state dependent state change representation that defines the altering, of at least the given state of the given smart device, relative to a corresponding current value for the given state of the given smart device.

### Claim 3 (depends on 2)

The method of claim 2, wherein processing the locally interpretable semantic representation to generate the given control command comprises:
determining, at the client device, a current value for the given state of the given smart device;
generating the given control command based on applying the current value to the current-state dependent state change representation.

### Claim 4 (depends on 3)

The method of claim 3, wherein the given smart device is a smart light, wherein the given state is a brightness intensity level, and the current value reflects a current brightness intensity level.

### Claim 5 (depends on 3)

The method of claim 3, wherein the given smart device is a smart thermostat, wherein the given state is a temperature set point, and the current value reflects a current temperature set point.

### Claim 6 (depends on 1)

The method of claim 1, wherein the locally interpretable semantic representation comprises an indication of a manufacturer of the smart device.

### Claim 7 (depends on 6)

The method of claim 6, wherein generating the given control command comprises utilizing an adapter, stored locally at the assistant client device, responsive to the adapter being assigned to the indication of the manufacturer of the smart device.

### Claim 8 (depends on 6)

The method of claim 6, further comprising:
selecting the local channel, for transmitting the given control command, based on the local channel being assigned to the indication of the manufacturer of the smart device.

### Claim 9 (depends on 1)

The method of claim 1, wherein processing the audio data to generate the current text is performed without detection of an explicit automated assistant invocation at the client device.

### Claim 10 (depends on 9)

The method of claim 9, wherein transmitting the given control command is further in response to determining that the current text satisfies one or more criteria for enabling matching text to be acted upon without explicit automated assistant invocation at the assistant client device.

### Claim 11 (depends on 1)

The method of claim 1, wherein the locally interpretable semantic representation comprises content that reflects the altering of at least the given state of the given smart device, and further comprising:
in response to determining that the current text matches the text of the cache entry:
rendering the content at the client device.

### Claim 12 (depends on 11)

The method of claim 11, wherein the content comprises graphics and wherein rendering the content comprises rending the graphics at a display of the client device.

### Claim 13 (depends on 11)

The method of claim 11, wherein the content comprises text, and further comprising:
processing the text, using a local text-to-speech engine of the client device, to generate synthesized speech that corresponds to the text;
wherein rendering the content comprises rendering the synthesized speech via one or more speakers of the client device.

### Claim 14 (depends on 11)

The method of claim 11, wherein the content comprises audio data and wherein rendering the content comprises rending the audio data via one or more speakers of the client device.

### Claim 15

A client device comprising:
one or more speakers;
one or more microphones;
one or more processors;
memory storing computer-executable instructions which, when executed by the one or more processors, causes the one or more processors to perform a method comprising:
storing, in a cache on the client device: a cache entry that includes a mapping of text to a semantic representation, wherein the semantic representation includes a locally interpretable semantic representation that is locally interpretable by the client device;
capturing, via the one or more microphones, audio data that captures a spoken utterance;
processing, using a voice-to-text model stored locally on the client device, the audio data to generate current text that corresponds to the spoken utterance;
determining that the current text matches the text of the cache entry; and
in response to determining that the current text matches the text of the cache entry, and in response to the cache entry including the mapping of the text to the semantic representation:
processing, at the client device, the locally interpretable semantic representation to generate a given control command, the given control command differing from the locally interpretable semantic representation; and
transmitting, via a local channel, the given control command to cause the altering of at least the given state of the given smart device.

### Claim 16 (depends on 15)

The client device of claim 15, wherein the locally interpretable semantic representation comprises:
an identifier of the given smart device; and
a current-state dependent state change representation that defines the altering, of at least the given state of the given smart device, relative to a corresponding current value for the given state of the given smart device.

### Claim 17 (depends on 15)

The client device of claim 15, wherein the locally interpretable semantic representation comprises an indication of a manufacturer of the smart device.

### Claim 18 (depends on 17)

The client device of claim 17, wherein generating the given control command comprises utilizing an adapter, stored locally at the assistant client device, responsive to the adapter being assigned to the indication of the manufacturer of the smart device.

### Claim 19 (depends on 17)

The client device of claim 17, wherein the method further comprises:
selecting the local channel, for transmitting the given control command, based on the local channel being assigned to the indication of the manufacturer of the smart device.

### Claim 20 (depends on 15)

The client device of claim 15, wherein the locally interpretable semantic representation comprises content that reflects the altering of at least the given state of the given smart device, and wherein the method further comprises:
in response to determining that the current text matches the text of the cache entry:
rendering the content at the client device.
