# US 11,444,799 B2 · Method and system of controlling device using real-time indoor image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,444,799 B2
- Document ID: US11444799B2
- Status: Granted
- Kind: B2
- Application number: 16/930,440
- Filed: 2020-07-16
- Published: 2020-11-05
- Granted: 2022-09-13
- Earliest claimed: 2017-08-31
- Assignees: Hanwha Techwin Co., Ltd.
- Applicants: Hanwha Techwin Co., Ltd.
- Inventors: Ho Jung Lee; Mi Ran Cho; Yeon Woo Kim
- Web page: https://patentgrep.com/patent/US11444799B2

## CPC

- H04L 12/282: {based on user interaction within the home (receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204)}
- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 18/40: Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- H04L 67/12: specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
- H04L 67/75: Indicating network or usage conditions on the user display
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06T 17/10: Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
- H04L 67/131: Protocols for games, networked simulations or virtual reality
- H04L 2012/2841: {Wireless}
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A device and a method for controlling a device using a real-time image are provided. The method includes: receiving an image captured by an image capturing device connected to a network to display the image in real-time; searching for the device that is connected to the network and is controllable; designating, within the image, a setting zone corresponding to the device; receiving a user input; and controlling the device selected according to the user input. A location of the setting zone within the image may be updated according to a change in the image. The user may receive immediate visual feedback on how the devices are being controlled. The user may control a device displayed on the screen on which the real-time indoor image is displayed without having to navigate through different sub-menus for different devices.

## Claims

### Claim 1

A method of controlling a device, the method comprising:
receiving an image captured by an image capturing device connected to a network and displaying the image in real time on a touch screen;
searching for a first device and a second device that are connected to the network and is controllable;
displaying first guide information for guiding a user to designate a first setting zone corresponding to the first device that is displayed in the image;
receiving a first user input within the image at a position of the first device in the image, and designating the first setting zone at the position of the first user input;
displaying second guide information for guiding the user to designate a second setting zone corresponding to the second device that is not displayed in the image;
receiving a second user input at a position at which the first device is not located, within the image; and
based on a determination that the second device is not recognized in the image, designating the second setting zone at the position of the second user input that does not overlap the position of the first device and positions of setting zones of other devices, and displaying the second setting zone in the image after designating the second setting zone;
controlling the first device and the second device according to control inputs interacting with the first setting zone and the second setting zone, respectively; and
updating the location of the first setting zone of the first device by changing the position of the first setting zone that is designated by the user to a different position within the image according to a change in the image in real time.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
displaying, on the image, at least a part of the second setting zone selected according to the second user input.

### Claim 3 (depends on 1)

The method of claim 1, wherein the first setting zone is designated based on a result of object recognition within the image.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
determining whether the first device is recognized in the image.

### Claim 5 (depends on 4)

The method of claim 4, wherein the designating the first setting zone further comprises, based on a determination that the first device is recognized in the image, designating the position of the first device recognized in the image as the position of the first setting zone of the first device.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
based on the second device selected according to the second user input not being recognized in the image, controlling the image capturing device to capture the second device selected according to the second user input while the second device is controlled or after the second device has been controlled.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
after designating the second setting zone in the image, displaying the second setting zone of the second device in the image while the second device is controlled or after the second device has been controlled.

### Claim 8 (depends on 1)

The method of claim 1, wherein information on the first device is displayed on the first setting zone.

### Claim 9 (depends on 1)

The method of claim 1, wherein the second user input is one of a touch input and a voice input.

### Claim 10

A device control system for controlling a device, the device control system comprising:
a camera connected to a network;
a display;
an input interface;
a processor; and
a computer-readable storage medium storing instructions which, when executed by the processor, cause the processor to perform operations comprising:
receiving an image captured by the camera connected to the network and displaying the image in real time on a touch screen;
searching for a first device and a second device that are connected to the network and is controllable;
displaying first guide information for guiding a user to designate a first setting zone corresponding to the first device that is displayed in the image;
receiving a first user input within the image at a position of the first device in the image, and designating the first setting zone at the position of the first user input;
displaying second guide information for guiding the user to designate a second setting zone corresponding to the second device that is not displayed in the image;
receiving a second user input at a position at which the first device is not located, within the image; and
based on a determination that the second device is not recognized in the image, designing the second setting zone at the position of the second user input that does not overlap with the position of the first device and positions of setting zones of other devices, and displaying the second setting zone in the image after designating the second setting zone;
controlling the first device and the second device according to control inputs interacting with the first setting zone and the second setting zone, respectively; and
updating the position of the first setting zone of the first device by changing the position of the first setting zone of the first device that is designated by the user to a different position within the image according to a change in the image in real time.

### Claim 11 (depends on 10)

The device control system of claim 10, wherein the change in the image comprises at least one of a change in an angle of view of the image and a change in a position of the second device within the angle of view.

### Claim 12 (depends on 10)

The device control system of claim 10, wherein the display is configured to display, on the image, at least a part of the second setting zone of the second device selected according to the second user input.

### Claim 13 (depends on 10)

The device control system of claim 10, wherein the second setting zone is designated based on at least one of the second user input and a result of object recognition within the image.

### Claim 14 (depends on 10)

The device control system of claim 10, wherein the operations further comprise:
based on the second device selected according to the second user input not being recognized in the image, controlling the camera to capture the second device selected according to the second user input while the second device is controlled or after the second device has been controlled.

### Claim 15 (depends on 10)

The device control system of claim 10, wherein the operations further comprise:
after designating the second setting zone, displaying the second setting zone of the second device on the image while the second device is controlled or after the second device has been controlled.

### Claim 16 (depends on 10)

The device control system of claim 10, wherein the display is configured to display information on the first device is displayed on the first setting zone.

### Claim 17 (depends on 10)

The device control system of claim 10, wherein the second user input is one of a touch input and a voice input.