# US 2021/0342047 A1 · Identifying and Controlling Smart Devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342047 A1
- Document ID: US20210342047A1
- Status: Published application
- Kind: A1
- Application number: 17/378,856
- Filed: 2021-07-19
- Published: 2021-11-04
- Earliest claimed: 2018-06-25
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Ibrahim Badr; Gokhan H. Bakir; Roland Peter Kehl; Nils Grimsmo
- Web page: https://patentgrep.com/patent/US20210342047A1

## CPC

- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- H04N 21/43615: {Interfacing a Home Network, e.g. for connecting the client to a plurality of peripherals (home Audio Video Interoperability [HAVI] data switching networks H04L12/2805)}
- G06F 1/1686: {the I/O peripheral being an integrated camera}
- G06F 1/1698: {the I/O peripheral being a sending/receiving arrangement to establish a cordless communication link, e.g. radio or infrared link, integrated cellular phone (details of antennas disposed inside a computer H01Q1/2266)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 9/453: {Help systems}
- G06V 20/20: in augmented reality scenes
- G08C 17/02: using a radio link
- H04M 1/72415: for remote control of appliances
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G08C 2201/30: User interface
- H04M 2250/52: including functional features of a camera
- H04W 88/02: Terminal devices
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08C: TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS (fluid pressure transmitting systems F15B; mechanical means for transferring the output of a sensing member into a different variable G01D5/00; mechanical control systems G05G)
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

Methods, systems, and apparatus for controlling smart devices are described. In one aspect a method includes receiving image data for an image captured by a camera of a mobile device of a user and determining that the image depicts at least one of a smart device or a physical control for the smart device. In response to determining that that the image depicts a smart device or a physical control for the smart device, identifying one or more user interface controls for controlling the smart device, and generating and presenting, at a display of the mobile device, the one or more user interface controls for controlling the smart device. The method can further include detecting, at the display of the mobile device, user interaction with at least one of the one or more user interface controls, and controlling the smart device based on the detected user interaction.

## Claims

### Claim at position 1

-40. (canceled)

### Claim 41

A computer-implemented method, the method comprising:
receiving, by a computing system comprising one or more processors, image data for an image captured by a camera of a computing device of a user;
determining, by the computing system based on the image data, that the image depicts a smart device;
in response to determining that the image depicts a smart device, registering, by the computing system, the computing device with the smart device;
receiving, by the computing system, a user interaction to control the smart device; and
controlling, by the computing system, the smart device based on the user interaction to control the smart device.

### Claim 42 (depends on 41)

The computer-implemented method of claim 41, further comprising:
determining, by the computing system, a class of the smart device based on the image data;
determining, by the computing system, one or more user interface controls associated with the class; and
providing, by the computing system, the one or more user interface controls to the computing device of the user.

### Claim 43 (depends on 42)

The computer-implemented method of claim 42, wherein determining the class comprises:
processing, by the computing system, the image data with one or more coarse classifiers.

### Claim 44 (depends on 42)

The computer-implemented method of claim 42, wherein determining the class comprises:
processing, by the computing system, the image data with one or more object recognizers for detecting object classes.

### Claim 45 (depends on 42)

The computer-implemented method of claim 42, wherein the class is at least one of a smart speaker, an appliance, a television, or a physical control.

### Claim 46 (depends on 42)

The computer-implemented method of claim 42, wherein the class is a smart speaker class, and wherein the one or more user interface controls comprise one or more volume controls.

### Claim 47 (depends on 42)

The computer-implemented method of claim 42, wherein the class is a smart light class, and wherein the one or more user interface controls comprise one or more light switch controls.

### Claim 48 (depends on 41)

The computer-implemented method of claim 41, further comprising:
determining, by the computing system, one or more controls associated with the smart device based on at least one of the image data or a registration file; and
providing, by the computing system, an augmented-reality user interface to the computing device, wherein the augmented-reality user interface comprises the one or more controls.

### Claim 49 (depends on 41)

The computer-implemented method of claim 41, wherein the smart device comprises a virtual assistant control device, wherein the virtual assistant control device controls one or more additional smart devices.

### Claim 50 (depends on 41)

The computer-implemented method of claim 41, wherein registering, by the computing system, the computing device with the smart device comprises:
obtaining, by the computing system via an established wireless connection, a registration file of the smart device, the registration file comprising smart device identification data;
sending, by the computing system based on the smart device identification data, user device identification data to the smart device via the established wireless connection; and
storing, by the computing system, the registration file to register the computing device of the user to the smart device.

### Claim 51 (depends on 41)

The computer-implemented method of claim 41, wherein determining that the image depicts the smart device further comprises establishing a communicative network connection between the smart device and the computing device.

### Claim 52

A computing system, comprising:
one or more processors; and
one or more tangible, non-transitory computer readable media storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations, the operations comprising:
receiving image data for an image captured by a camera of a computing device of a user;
determining, based on the image data, that the image depicts a smart device;
in response to determining that the image depicts a smart device, registering the computing device with the smart device;
receiving a user interaction to control the smart device; and
controlling the smart device based on the user interaction to control the smart device.

### Claim 53 (depends on 52)

The computing system of claim 52, wherein the user interaction comprises an interaction with a user interface provided by the computing device.

### Claim 54 (depends on 52)

The computing system of claim 52, further comprising:
processing the image data with one or more machine learning models to determine a class of the smart device;
determining one or more user interface controls associated with the class; and
providing the one or more user interface controls with the computing device.

### Claim 55 (depends on 54)

The computing system of claim 54, wherein the one or more machine learning models are trained using labeled image data, wherein the labeled image data comprises one or more labeled images labeled with their respective classes.

### Claim 56 (depends on 54)

The computing system of claim 54, wherein the one or more machine learning models comprise a convolutional network to classify the smart device based on visual features.

### Claim 57 (depends on 54)

The computing system of claim 54, further comprising:
generating a confidence value that indicates a confidence level of the class; and
providing the confidence value that indicates the confidence level of the class to the computing device.

### Claim 58

One or more non-transitory computer readable media storing computer-readable instructions that when executed by one or more processors cause the one or more processors to perform operations, the operations comprising:
receiving image data for an image captured by a camera of a computing device of a user;
determining, based on the image data, that the image depicts a smart device;
in response to determining that the image depicts a smart device, registering the computing device with the smart device;
receiving a user interaction to control the smart device; and
controlling the smart device based on the user interaction to control the smart device.

### Claim 59 (depends on 58)

The one or more non-transitory computer readable media of claim 58, wherein determining that the image depicts a smart device comprises:
processing the image data with a machine learning model, wherein the machine learning model is trained to receive data related to visual features of the image and output a classification into one or more classes of objects.

### Claim 60 (depends on 58)

The one or more non-transitory computer readable media of claim 58, further comprising:
providing a virtual assistant application for display on the computing device, wherein the virtual assistant application comprises one or more user interface controls for controlling the smart device, wherein the user interaction comprises an interaction with the one or more user interface controls.
