# US 12,112,032 B2 · Controlling a device based on processing of image data that captures the device and/or an installation environment of the device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,032 B2 - Document ID: US12112032B2 - Status: Granted - Kind: B2 - Application number: 18/142,902 - Filed: 2023-05-03 - Published: 2023-08-31 - Granted: 2024-10-08 - Earliest claimed: 2021-12-16 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Yuzhao Ni - Web page: https://patentgrep.com/patent/US12112032B2 ## CPC - G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials - H04L 12/2814: {Exchanging control software or macros for controlling appliance services in a home automation network (arrangements for maintenance or administration involving configuration of the network and network elements H04L41/08)} - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - H04N 23/631: {Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Implementations discussed herein relate to systems, methods, and apparatuses for controlling one or more internet of things (IoT) devices via a graphical camera interface of a portable electronic device. The graphical camera interface can be presented at the portable electronic device in response to the user accessing a camera application via the portable electronic device and directing a camera of the portable electronic device at the IoT device. One or more images generated from an output of the camera can be processed to identify the IoT device that the user is intending to control. Information related to a context in which the user aimed their camera at the IoT device can be used to assist in narrowing down a selection of the IoT device from multiple different IoT devices that may be associated with the user and/or an image captured by the camera. ## Claims ### Claim 1 A method implemented by one or more processors, the method comprising: receiving image data corresponding to an output of a camera that is integral to a portable client device and is directed at an environment, wherein the portable client device includes an automated assistant interface that is configured to provide an interface between a user and an automated assistant; receiving a spoken utterance from the user, wherein the spoken utterance is directed to the automated assistant; processing the spoken utterance to identify a request of the user, wherein the request includes a command to adjust a state of a controllable electronic device; identifying, based on the image data and contextual data that is in addition to the image data, a particular controllable electronic device that is included in the environment and that is captured in the image data; determining, based on both (i) the command included in the request identified based on the spoken utterance and (ii) the particular controllable electronic device being identified based on the image data and the contextual data, that the request is directed to the particular controllable electronic device, wherein determining that the request is directed to the particular controllable electronic device further comprises: identifying, based on the image data, an additional controllable electronic device that is included in the environment and that is captured in the image data; determining a first metric associated the particular controllable device, wherein the first metric indicates a first probability that the request is directed to the particular controllable device; determining a second metric associated with the additional controllable device, wherein the second metric indicates a second probability that the request is directed to the additional controllable device; and determining, based on the first metric and the second metric, that the particular controllable electronic device, in lieu of the additional controllable electronic device, is the controllable electronic device to which the request is directed; and causing, in response to receiving the request and in response to determining that the request is directed to the controllable electronic device, the state of the particular controllable electronic device to be modified in accordance with the command of the request. ### Claim 2 (depends on 1) The method of claim 1, wherein the contextual data includes room adjacency data that characterizes a layout of the environment. ### Claim 3 (depends on 1) The method of claim 1, wherein the contextual data characterizes a context of the user when the user provided the spoken utterance. ### Claim 4 (depends on 1) The method of claim 1, wherein the contextual data characterizes a context of the portable client device when the image data was captured. ### Claim 5 (depends on 1) The method of claim 1, wherein identifying the particular controllable electronic device is further based on the command. ### Claim 6 (depends on 5) The method of claim 5, wherein identifying the particular controllable electronic device based on the command includes: determining that the command is related to an adjustable setting of the particular controllable electronic device. ### Claim 7 A system, comprising: a portable client device, wherein the portable client device includes one or more processors and memory operably coupled with the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations: receiving image data corresponding to an output of a camera that is integral to the portable client device and is directed at an environment, wherein the portable client device includes an automated assistant interface that is configured to provide an interface between a user and an automated assistant; receiving a spoken utterance from the user, wherein the spoken utterance is directed to the automated assistant; processing the spoken utterance to identify a request of the user, wherein the request includes a command to adjust a state of a controllable electronic device; identifying, based on the image data and contextual data that is in addition to the image data, a particular controllable electronic device that is included in the environment and that is captured in the image data; determining, based on both (i) the command included in the request identified based on the spoken utterance and (ii) the particular controllable electronic device being identified based on the image data and the contextual data, that the request is directed to the particular controllable electronic device, wherein determining that the request is directed to the particular controllable electronic device further comprises: identifying, based on the image data, an additional controllable electronic device that is included in the environment and that is captured in the image data; determining a first metric associated the particular controllable device, wherein the first metric indicates a first probability that the request is directed to the particular controllable device; determining a second metric associated with the additional controllable device, wherein the second metric indicates a second probability that the request is directed to the additional controllable device; and determining, based on the first metric and the second metric, that the particular controllable electronic device, in lieu of the additional controllable electronic device, is the controllable electronic device to which the request is directed; and causing, in response to receiving the request and in response to determining that the request is directed to the controllable electronic device, the state of the particular controllable electronic device to be modified in accordance with the command of the request. ### Claim 8 (depends on 7) The system of claim 7, wherein the contextual data includes room adjacency data that characterizes a layout of the environment. ### Claim 9 (depends on 7) The system of claim 7, wherein the contextual data characterizes a context of the user when the user provided the spoken utterance. ### Claim 10 (depends on 7) The system of claim 7, wherein the contextual data characterizes a context of the portable client device when the image data was captured. ### Claim 11 (depends on 7) The system of claim 7, wherein identifying the particular controllable electronic device is further based on the command. ### Claim 12 (depends on 11) The system of claim 11, wherein identifying the particular controllable electronic device based on the command includes: determining that the command is related to an adjustable setting of the particular controllable electronic device. ### Claim 13 A non-transitory computer readable storage medium configured to store instructions that, when executed by one or more processors, cause the one or more processors to perform operations that include: receiving image data corresponding to an output of a camera that is integral to a portable client device and is directed at an environment, wherein the portable client device includes an automated assistant interface that is configured to provide an interface between a user and an automated assistant; receiving a spoken utterance from the user, wherein the spoken utterance is directed to the automated assistant; processing the spoken utterance to identify a request of the user, wherein the request includes a command to adjust a state of a controllable electronic device; identifying, based on the image data and contextual data that is in addition to the image data, a particular controllable electronic device that is included in the environment and that is captured in the image data; determining, based on both (i) the command included in the request identified based on the spoken utterance and (ii) the particular controllable electronic device being identified based on the image data and the contextual data, that the request is directed to the particular controllable electronic device, wherein determining that the request is directed to the particular controllable electronic device further comprises: identifying, based on the image data, an additional controllable electronic device that is included in the environment and that is captured in the image data; determining a first metric associated the particular controllable device, wherein the first metric indicates a first probability that the request is directed to the particular controllable device; determining a second metric associated with the additional controllable device, wherein the second metric indicates a second probability that the request is directed to the additional controllable device; and determining, based on the first metric and the second metric, that the particular controllable electronic device, in lieu of the additional controllable electronic device, is the controllable electronic device to which the request is directed; and causing, in response to receiving the request and in response to determining that the request is directed to the controllable electronic device, the state of the particular controllable electronic device to be modified in accordance with the command of the request. ### Claim 14 (depends on 13) The non-transitory computer readable storage medium of claim 13, wherein the contextual data includes room adjacency data that characterizes a layout of the environment. ### Claim 15 (depends on 13) The non-transitory computer readable storage medium of claim 13, wherein the contextual data characterizes a context of the user when the user provided the spoken utterance. ### Claim 16 (depends on 13) The non-transitory computer readable storage medium of claim 13, wherein the contextual data characterizes a context of the portable client device when the image data was captured. ### Claim 17 (depends on 13) The non-transitory computer readable storage medium of claim 13, wherein identifying the particular controllable electronic device is further based on the command.