# US 2020/0401231 A1 · Mobile terminal and control method therefor > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401231 A1 - Document ID: US20200401231A1 - Status: Published application - Kind: A1 - Application number: 16/977,449 - Filed: 2018-06-08 - Published: 2020-12-24 - Earliest claimed: 2018-03-02 - Assignees: LG Electronics Inc. - Applicants: LG Electronics Inc. - Inventors: Yehan AHN; Minho KIM; Hoyoung KIM; Mansoo SIN - Web page: https://patentgrep.com/patent/US20200401231A1 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06T 7/11: Region-based segmentation - G06T 7/194: involving foreground-background segmentation - G06T 7/50: Depth or shape recovery - G06T 7/73: using feature-based methods - G06V 40/107: {Static hand or arm} - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G06V 40/67: {by interactive indications to the user} - G06T 2200/16: involving adaptation to the client's capabilities - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20021: Dividing image into blocks, subimages or windows - H04M 1/72403: with means for local support of applications that increase the functionality - H04M 1/725: Cordless telephones (user interfaces specially adapted therefor H04M1/724) - H04M 2201/38: Displays - H04M 2250/52: including functional features of a camera - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) ## Abstract The present invention relates to a device and a control method therefor and, more specifically, the device comprises: a memory for storing at least one command; a depth camera for capturing at least one hand of a user; a display module; and a controller for controlling the memory, the depth camera, and the display module. The controller controls the depth camera so as to capture the at least one hand of a user and controls the display module so as to output a visual feedback that changes on the basis of the captured hand of a user. ## Claims ### Claim 1 A method of controlling a device including a depth camera, the method comprising: acquiring a depth image using the depth camera; changing the device from a first state into a second state by analyzing the acquired depth image with reference to a memory; determining whether a user's finger points to the device with reference to the memory; changing the device from the second state into a third state based on determining that the user's finger points to the device; tracking an end point of the user's finger; and displaying a target point having a position changeable in response to the tracked end point of the user's finger. ### Claim 2 (depends on 1) The method of claim 1, wherein changing the device from the first state into the second state comprises: removing values of pixels failing to exist between a minimum reference distance and a maximum reference distance by comparing pixel values of the depth image; among unremoved pixels, estimating pixels having top, bottom, right and left sides attached as a single segment; when a plurality of segments exist, regarding a particular segment having a greatest pixel number as being valid only; and when the pixel number of the particular segment is equal to or greater than a preset threshold, changing the device into the second state corresponding to a ready state. ### Claim 3 (depends on 2) The method of claim 2, wherein determining whether the user's finger points to the device comprises estimating a smallest one of pixel values in the particular segment as a pixel value closest to the device. ### Claim 4 (depends on 3) The method of claim 3, wherein determining whether the user's finger points to the device further comprises: determining a first image including pixels located from a point estimated as the pixel value closest to the device to a first distance; determining a second image including pixels located from the point estimated as the pixel value closest to the device to a second distance; and determining a third image including pixels located from the point estimated as the pixel value closest to the device to a third distance, wherein the second distance is greater than the first distance, and wherein the third distance is greater than the second distance. ### Claim 5 (depends on 4) The method of claim 4, further comprising determining that the user's finger points to the device when a segment number of the second image is 1, a size of the second image is smaller than a predetermined ratio of a size of the third image, and a distance between a center point of the first image and a center point of the second image is smaller than the pixel value closest to the device. ### Claim 6 (depends on 1) The method of claim 1, wherein displaying the target point comprises changing the position of the target point based on a center point of a specific region in the acquired depth image and a fixed correction ratio. ### Claim 7 A device comprising: a memory configured to store one or more commands; a depth camera configured to capture at least a user's hand; a display; and a controller configured to control the memory, the depth camera and the display, wherein the controller is further configured to: control the depth camera to capture at least the user's hand; and control the display to output a visual feedback changeable based on the captured user's hand. ### Claim 8 (depends on 7) The device of claim 7, wherein the visual feedback includes a shape mirroring the captured user's hand and distinctively displays an end point of a finger located at a distance closest to the device. ### Claim 9 (depends on 7) The device of claim 7, wherein based on determining that the captured user's hand is located between a minimum reference distance and a maximum reference distance from the device, the controller is further configured to control the display to output the visual feedback. ### Claim 10 (depends on 7) The device of claim 7, wherein based on an end point of a finger being located at a distance closest to the device and a reference point of the captured user's hand, the controller is further configured to change a position of a target point output to the display.