# US 12,223,601 B2 · Three dimensional virtual room-based user interface for a home automation system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,601 B2 - Document ID: US12223601B2 - Status: Granted - Kind: B2 - Application number: 18/201,046 - Filed: 2023-05-23 - Published: 2023-09-14 - Granted: 2025-02-11 - Earliest claimed: 2019-09-11 - Assignees: Savant Systems, LLC - Applicants: Savant Systems, Inc. - Inventors: Robert P. Madonna; Maxwell R. Madonna; David W. Tatzel; Michael A. Molta; Timothy Kallman - Web page: https://patentgrep.com/patent/US12223601B2 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06T 19/003: {Navigation within 3D models or images} - G05B 15/02: electric - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06T 15/20: Perspective computation - G06T 15/503: {Blending, e.g. for anti-aliasing} - G06T 15/506: {Illumination models} - H04L 12/2814: {Exchanging control software or macros for controlling appliance services in a home automation network (arrangements for maintenance or administration involving configuration of the network and network elements H04L41/08)} - H04L 12/282: {based on user interaction within the home (receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204)} - H04L 12/2829: {involving user profiles according to which the execution of a home appliance functionality is automatically triggered} - G05B 2219/2614: HVAC, heating, ventillation, climate control - G05B 2219/2642: Domotique, domestic, home control, automation, smart house - H04L 2012/285: {Generic home appliances, e.g. refrigerators} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract In one embodiment, a user-navigable 3-D virtual room-based user interface for a home automation system is provided. A control app renders and displays the user-navigable 3-D virtual room from a perspective defined by a virtual camera, wherein the control app renders the user-navigable 3-D virtual room based on data from at least one of a plurality of 2-D images of the physical room captured from different respective positions in the physical room. In response to an explicit navigation command or implicit action, the control app alters a position or orientation of the virtual camera. The control app re-renders and displays the user-navigable 3-D virtual room from a new perspective defined by the altered position or orientation, wherein the new perspective does not coincide with the position in the physical room from which any of the 2-D images were captured, by blending data from multiple 2-D images captured from different positions. ## Claims ### Claim 1 A method for controlling a home automation system using a user-navigable three-dimensional (3-D) virtual room that corresponds to a physical room, comprising: rendering and displaying, by a control application (app) executing on an electronic device, the user-navigable 3-D virtual room from a perspective defined by a virtual camera, wherein the control app renders the user-navigable 3-D virtual room based on data from at least one of a plurality of two-dimensional (2-D) images of the physical room captured from different respective positions in the physical room, and the user-navigable 3-D virtual room includes depictions of one or more devices present in the physical room that are under the control of the home automation system, depictions of one or more boundaries of the physical room and depictions of one or more furnishings present in the physical room; receiving an explicit navigation command or implicit action from a user; in response to the explicit navigation command or implicit action, translating or rotating the virtual camera, by the control app, to alter a position or an orientation of the virtual camera; re-rendering and displaying, by the control app, the user-navigable 3-D virtual room from a new perspective defined by the altered position or orientation, wherein the new perspective does not coincide with the position in the physical room from which any of the 2-D images were captured, and the control app re-renders the user-navigable 3-D virtual room by blending data from multiple 2-D images captured from different positions to show the user-navigable 3-D virtual room from the new perspective; receiving a user interaction associated with the user-navigable 3-D virtual room; and in response to the user interaction, causing the home automation system to change a state of a device in the physical room. ### Claim 2 (depends on 1) The method of claim 1, further comprising: dynamically updating, by the control app, an appearance of the depiction of the device, the depictions of one or more boundaries, or the depictions of one or more furnishings in the user-navigable 3-D virtual room to reflect the changed state. ### Claim 3 (depends on 1) The method of claim 1, wherein the receiving an explicit navigation command or implicit action is receiving an explicit navigation command, and the explicit navigation command includes a touch gesture on a touch sensitive display screen or a movement of a cursor by the user. ### Claim 4 (depends on 1) The method of claim 1, wherein the receiving an explicit navigation command or implicit action is receiving an implicit action, and the implicit action includes a change in position or orientation of the electronic device by the user. ### Claim 5 (depends on 1) The method of claim 1, wherein the user interaction is an interaction with a depiction of the device within the user-navigable 3-D virtual room by the user. ### Claim 6 (depends on 1) The method of claim 1, further comprising: displaying, by the control app, a menu, and wherein the user interaction is an interaction with a menu interface element of the menu by the user. ### Claim 7 (depends on 1) The method of claim 1, wherein the device is a lighting device and the state of the device includes at least one of an illumination level, color, or color temperature. ### Claim 8 (depends on 7) The method of claim 7, further comprising: dynamically updating, by the control app, an appearance of the depiction of the lighting device to show the lighting device with changed illumination level, color, or color temperature, the depictions of one or more boundaries to show the one or more boundaries with changed shadows or reflections resulting from the changed illumination level, color, or color temperature, or the depictions of one or more furnishings to show the one or more furnishings with changed shadows or reflections resulting from the changed illumination level, color, or color temperature. ### Claim 9 (depends on 1) The method of claim 1, wherein the device is an electronic window blind and the state of the device includes a blind position. ### Claim 10 (depends on 9) The method of claim 9, further comprising: dynamically updating, by the control app, an appearance of the depiction of the electronic window blind to show the electronic window blind with changed blind position, the depictions of one or more boundaries to show the one or more boundaries with changed shadows or reflections resulting from the changed blind position, or the depictions of one or more furnishings to show the one or more furnishings with shadows or reflections resulting from the changed blind position. ### Claim 11 (depends on 1) The method of claim 1, wherein the depictions of one or more devices, depictions of one or more boundaries and depictions of one or more furnishings are each substantially photo-realistic depictions. ### Claim 12 A non-transitory electronic device readable medium having software encoded thereon, the software, when executed on one or more processors of one or more electronic devices, operable to: render and display a user-navigable three-dimensional (3-D) virtual room from a perspective defined by a virtual camera based on data from at least one of a plurality of two-dimensional (2-D) images of a physical room captured from different respective positions and orientations in the physical room, wherein the user-navigable 3-D virtual room includes depictions of one or more devices present in the physical room, receive an explicit navigation command or implicit action from a user; in response to the explicit navigation command or implicit action, translate or rotate the virtual camera to alter a position or an orientation of the virtual camera; re-render and display the user-navigable 3-D virtual room from a new perspective defined by the altered position or orientation of the virtual camera, wherein the new perspective does not coincide with the position or orientation in the physical room from which any of the 2-D images were captured, and the user-navigable 3-D virtual room is re-rendered by blending data from multiple 2-D images captured from different positions or orientations to show the user-navigable 3-D virtual room from the new perspective; receive a user interaction associated with the user-navigable 3-D virtual room; in response to the user interaction, change a state of a device in the physical room. ### Claim 13 (depends on 12) The non-transitory electronic device readable medium of claim 12, wherein the software when executed is further operable to: dynamically update an appearance of the depiction of the device in the user-navigable 3-D virtual room to reflect the changed state. ### Claim 14 (depends on 12) The non-transitory electronic device readable medium of claim 12, wherein the user-navigable 3-D virtual room further includes depictions of one or more boundaries of the physical room or depictions of one or more furnishings present in the physical room. ### Claim 15 (depends on 12) The non-transitory electronic device readable medium of claim 12, wherein the device is a lighting device and the state of the device includes at least one of an illumination level, color or color temperature. ### Claim 16 (depends on 12) The non-transitory electronic device readable medium of claim 12, wherein the device is an electronic window blind and the state of the device includes a blind position. ### Claim 17 (depends on 12) The non-transitory electronic device readable medium of claim 12, wherein the device is a display device and the state of the device includes a media content state. ### Claim 18 A method for controlling a home automation system using a user-navigable three-dimensional (3-D) virtual room that corresponds to a physical room, comprising: rendering and displaying, by a control application (app) executing on an electronic device, the user-navigable 3-D virtual room from a perspective defined by a virtual camera, wherein the control app renders the user-navigable 3-D virtual room based on data from a plurality of images of the physical room captured from different respective positions in the physical room; receiving an explicit navigation command or implicit action from a user; in response to the explicit navigation command or implicit action, translating or rotating the virtual camera, by the control app, to alter a position or an orientation of the virtual camera; re-rendering and displaying, by the control app, the user-navigable 3-D virtual room from a new perspective defined by the altered position or orientation, wherein the new perspective does not coincide with the position in the physical room from which any of the images were captured; receiving a user interaction associated with the user-navigable 3-D virtual room; and in response to the user interaction, causing the home automation system to change a state of a device in the physical room. ### Claim 19 (depends on 18) The method of claim 18, further comprising: dynamically updating, by the control app, an appearance of the user-navigable 3-D virtual room to reflect the changed state. ### Claim 20 (depends on 18) The method of claim 18, wherein the receiving an explicit navigation command or implicit action is receiving an explicit navigation command, and the explicit navigation command includes a touch gesture on a touch sensitive display screen or a movement of a cursor by the user. ### Claim 21 (depends on 18) The method of claim 18, wherein the receiving an explicit navigation command or implicit action is receiving an implicit action, and the implicit action includes a change in position or orientation of the electronic device by the user. ### Claim 22 (depends on 18) The method of claim 18, wherein the user interaction is an interaction with a depiction of the device within the user-navigable 3-D virtual room by the user. ### Claim 23 (depends on 18) The method of claim 18, further comprising: displaying, by the control app, a menu, and wherein the user interaction is an interaction with a menu interface element of the menu by the user. ### Claim 24 (depends on 18) The method of claim 18, wherein the device is a lighting device and the state of the device includes at least one of an illumination level, color, or color temperature. ### Claim 25 (depends on 18) The method of claim 18, wherein the user-navigable 3-D virtual room includes depictions of one or more devices present in the physical room that are under the control of the home automation system, depictions of one or more boundaries of the physical room and depictions of one or more furnishings present in the physical room, and the depictions of one or more devices, depictions of one or more boundaries and depictions of one or more furnishings are each substantially photo-realistic depictions. ### Claim 26 (depends on 18) The method of claim 18, further comprising: generating the user-navigable 3-D virtual room by stitching the images of the physical room and a 3-D space model of the physical room together to link image data to corresponding locations in the 3-D space model, tagging depictions of one or more devices visible in the stitched images and 3-D space model with hit regions, and assigning appearance changes to the one or more devices in the tagged, stitched images and 3-D space model. ### Claim 27 A non-transitory electronic device readable medium having software encoded thereon, the software, when executed on one or more processors of one or more electronic devices, operable to: generate a user-navigable 3-D virtual room by stitching a plurality of images of the physical room captured from different respective positions in the physical room and a 3-D space model of the physical room together to link image data to corresponding locations in the 3-D space model, tagging depictions of one or more devices visible in the stitched images and 3-D space model with hit regions, and assigning appearance changes to the one or more devices in the tagged, stitched images and 3-D space model; render and display the user-navigable 3-D virtual room from a perspective defined by a virtual camera; receive an explicit navigation command or implicit action from a user; in response to the explicit navigation command or implicit action, translate or rotate the virtual camera to alter a position or an orientation of the virtual camera; re-render and display the user-navigable 3-D virtual room from a new perspective defined by the altered position or orientation, wherein the new perspective does not coincide with the position in the physical room from which any of the images were captured; receive a user interaction associated with the user-navigable 3-D virtual room; and in response to the user interaction, cause the home automation system to change a state of a device in the physical room. ### Claim 28 (depends on 27) The non-transitory electronic device readable medium of claim 27, wherein the software when executed is further operable to: dynamically update an appearance of the depiction of the device in the user-navigable 3-D virtual room to reflect the changed state.