# US 11,609,541 B2 · System and method of IOT device control using augmented reality > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,609,541 B2 - Document ID: US11609541B2 - Status: Granted - Kind: B2 - Application number: 17/462,409 - Filed: 2021-08-31 - Published: 2021-12-23 - Granted: 2023-03-21 - Earliest claimed: 2017-12-06 - Assignees: ARRIS Enterprises LLC - Applicants: ARRIS Enterprises LLC - Inventors: Adrian Becea; Ganesh Nagarajan; Stephen E. Strand; Charles Ross Denneler - Web page: https://patentgrep.com/patent/US11609541B2 ## CPC - G05B 19/042: using digital processors (G05B19/05 takes precedence) - G06F 16/951: Indexing; Web crawling techniques - H04L 67/025: for remote control or remote monitoring of applications - G05B 2219/23298: Remote load of program, through internet - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - H04L 67/141: Setup of application sessions (admission control or resource allocation in data switching networks H04L47/70) - H04L 67/303: Terminal profiles - H04W 4/70: Services for machine-to-machine communication [M2M] or machine type communication [MTC] - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) ## Abstract A method and system for remotely controlling an external device includes storing, in a computing device, a plurality of device profiles, each device profile including connection data, one or more control commands, and at least one associated image; receiving, by an imaging device interfaced with the computing device, a live rendering of a physical environment; detecting, by the computing device, a registered image in the live rendering of the physical environment; identifying, in the computing device, a specific device profile where the at least one associated image includes the detected registered image; establishing, by the computing device, a communication channel with an external device using the connection data included in the identified specific device profile; and electronically transmitting, by the computing device, at least one of the one or more control commands included in the identified specific device profile to the external device using the established communication channel. ## Claims ### Claim 1 A method for remotely controlling an external device, comprising: storing a plurality of device profiles, wherein each device profile includes connection data, one or more control commands, and at least one associated image; receiving a live rendering of a physical environment, the live rendering including at least one registered image and at least one unregistered image; detecting the registered image in the live rendering of the physical environment; identifying a specific device profile of the plurality of device profiles where the at least one associated image includes the detected registered image; establishing a communication channel with the external device using the connection data included in the identified specific device profile; receiving, a user instruction associated with at least one of the one or more control commands included in the identified specific device profile, wherein the user instruction is one of a voice instruction and a gesture instruction; and after receiving the user instruction, transmitting to the external device, using the established communication channel, the at least one of the one or more control commands for controlling the external device, wherein the method further comprises selecting the at least one unregistered image in the live rendering for initiating a registration process, the registration process associating the at least one unregistered image with the external device by associating the at least one unregistered image with the specific device profile and the at least one associated image for the external device. ### Claim 2 (depends on 1) The method of claim 1, further comprising: displaying the received live rendering of the physical environment. ### Claim 3 (depends on 2) The method of claim 2, further comprising: indicating, on the displayed live rendering of the physical environment, the detected registered image. ### Claim 4 (depends on 3) The method of claim 3, further comprising: displaying the one or more control commands included in the identified specific device profile superimposed on the indicated detected registered image. ### Claim 5 (depends on 1) The method of claim 1, wherein the external device is an Internet-enabled computing device. ### Claim 6 (depends on 1) The method of claim 1, wherein the live rendering of the physical environment is a virtual representation of the physical environment based on stored three dimensional spatial data. ### Claim 7 (depends on 1) The method of claim 1, wherein the at least one unregistered image in the live rendering is an image of an object that is unrelated to the external device. ### Claim 8 (depends on 1) The method of claim 1, wherein registering process of the at least one unregistered image includes registering two or more images for controlling the external device. ### Claim 9 A system for remotely controlling an external device, comprising: a device database configured to store a plurality of device profiles, wherein each device profile includes connection data, one or more control commands, and at least one associated image; an imaging device configured to receive a live rendering of a physical environment, the live rendering including at least one registered image and at least one unregistered image; a hardware controller programmed to operate as a detection module configured to detect the registered image in the live rendering of the physical environment; the hardware controller programmed to operate as a querying module configured to execute a query on the device database to identify a specific device profile of the plurality of device profiles where the at least one associated image includes the detected registered image; a communications interface configured to establish a communication channel with the external device using the connection data included in the identified specific device profile; an input interface configured to receive a user instruction associated with at least one of the one or more control commands included in the identified specific device profile, wherein the user instruction is one of a voice instruction and a gesture instruction; and a transmitting device configured to transmit, after receipt of the user instruction, the at least one of the one or more control commands included in the identified specific device profile to the external device using the established communication channel for controlling the external device, wherein the input interface is further configured to receive an instructions for selecting the at least one unregistered image in the live rendering for initiating a registration process, the registration process associating the at least one unregistered image with the external device by associating the at least one unregistered image with the specific device profile and the at least one associated image for the external device. ### Claim 10 (depends on 9) The system of claim 9, further comprising: a display device configured to display the received live rendering of the physical environment. ### Claim 11 (depends on 10) The system of claim 10, wherein the display device is further configured to indicate, on the displayed live rendering of the physical environment, the detected registered image. ### Claim 12 (depends on 11) The system of claim 11, wherein the display device is further configured to display the one or more control commands included in the identified specific device profile superimposed on the indicated detected registered image. ### Claim 13 (depends on 10) The system of claim 10, wherein the external device is an Internet-enabled computing device. ### Claim 14 (depends on 10) The system of claim 10, wherein the live rendering of the physical environment is a virtual representation of the physical environment based on stored three dimensional spatial data. ### Claim 15 A non-transitory computer readable media having instructions operable to cause one or more processors to perform the operations comprising: storing a plurality of device profiles, wherein each device profile includes connection data, one or more control commands, and at least one associated image; receiving a live rendering of a physical environment, the live rendering including at least one registered image and at least one unregistered image; detecting the registered image in the live rendering of the physical environment; identifying a specific device profile of the plurality of device profiles where the at least one associated image includes the detected registered image; establishing a communication channel with an external device using the connection data included in the identified specific device profile; receiving a user instruction associated with at least one of the one or more control commands included in the identified specific device profile, wherein the user instruction is one of a voice instruction input and a gesture instruction; and after receiving the user instruction, electronically transmitting to the external device, using the established communication channel, the at least one of the one or more control commands for controlling the external device, wherein the non-transitory computer readable media causes the one more processors to be further execute the operation of selecting the at least one unregistered image in the live rendering for initiating a registration process, the registration process associating the at least one unregistered image with the external device by associating the at least one unregistered image with the specific device profile and the at least one associated image for the external device. ### Claim 16 (depends on 15) The non-transitory computer readable media of claim 15, being further configured to perform the operations comprising: displaying the received live rendering of the physical environment. ### Claim 17 (depends on 15) The non-transitory computer readable media of claim 15, being further configured to perform the operations comprising: indicating, on the displayed live rendering of the physical environment, the detected registered image. ### Claim 18 (depends on 15) The non-transitory computer readable media of claim 15, being further configured to perform the operations comprising: displaying the one or more control commands included in the identified specific device profile superimposed on the indicated detected registered image. ### Claim 19 (depends on 15) The non-transitory computer readable media of claim 15, wherein the external device is an Internet-enabled computing device.