# US 2021/0081104 A1 · Electronic apparatus and controlling method thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0081104 A1
- Document ID: US20210081104A1
- Status: Published application
- Kind: A1
- Application number: 16/925,489
- Filed: 2020-07-10
- Published: 2021-03-18
- Earliest claimed: 2019-09-18
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Wonsuk CHUNG; Sunghyun PARK; Soonyong CHO; Jaehwan Kim; Jisup Lee
- Web page: https://patentgrep.com/patent/US20210081104A1

## CPC

- G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
- G06F 1/1686: {the I/O peripheral being an integrated camera}
- G06F 3/0236: {using selection techniques to select from displayed items}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0425: {using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386)}
- G06F 3/0426: {tracking fingers with respect to a virtual keyboard projected or printed on the surface (virtual keyboards on touch screens G06F3/04886)}
- G06V 10/82: using neural networks
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Disclosed are an electronic apparatus and a method of controlling the same. The electronic apparatus includes a camera, a display, a memory, and a processor configured to execute at least one instruction to: detect a plurality of fingers in a plurality of first image frames obtained through the camera and, in response to an identification that a pose of the plurality of detected fingers corresponds to a trigger pose, enter a character input mode, detect a first motion of a finger among the plurality of fingers in a plurality of second image frames obtained through in the character input mode, identify a key corresponding to the first motion, from among a plurality of keys mapped to the finger, based on a position of the finger by the first motion and a reference point set to the finger, and control the display to display information corresponding to the identified key.

## Claims

### Claim 1

An electronic apparatus comprising:
a camera;
a display;
a memory configured to store at least one instruction; and
a processor configured to execute the at least one instruction to:
detect a plurality of fingers in a plurality of first image frames obtained through the camera and, in response to an identification that a pose of the plurality of detected fingers corresponds to a trigger pose, enter a character input mode,
detect a first motion of a finger among the plurality of fingers in a plurality of second image frames obtained through the camera in the character input mode,
identify a key corresponding to the first motion, from among a plurality of keys mapped to the finger, based on a position of the finger by the first motion and a reference point set to the finger,
control the display to display information corresponding to the identified key, and
detect the plurality of fingers in a plurality of third image frames obtained through the camera in the character input mode and, in response to an identification that the pose of the plurality of detected fingers in the plurality of third image frames corresponds to a predetermined pose, re-set a reference point for respective fingers based on a position of the plurality of detected fingers in the plurality of third image frames.

### Claim 2 (depends on 1)

The electronic apparatus of claim 1, wherein the processor is further configured to execute the at least one instruction to, in response to an identification that the plurality of fingers detected in the plurality of third image frames maintain the predetermined pose for a predetermined time, re-set the reference point for the respective fingers based on the position of the plurality of detected fingers in the plurality of third image frames.

### Claim 3 (depends on 1)

The electronic apparatus of claim 1, wherein the processor is further configured to execute the at least one instruction to identify whether the pose of the plurality of detected fingers in the plurality of third image frames corresponds to the predetermined pose based on a position of a fingertip of a remaining finger other than a thumb among the plurality of fingers detected in the plurality of third image frames.

### Claim 4 (depends on 1)

The electronic apparatus of claim 1, wherein the processor is further configured to execute the at least one instruction to, based on a second motion of the finger among the plurality of fingers being detected in a plurality of fourth image frames obtained through the camera after the reference point is re-set in the character input mode, identify a key corresponding to the second motion, from among the plurality keys mapped to the finger, based on a position of the finger by the second motion and a position of a reference point reset to the finger.

### Claim 5 (depends on 1)

The electronic apparatus of claim 1, wherein the plurality of keys mapped to the finger comprises a reference key, determined based on a type of a hand including the finger and a type of the finger, among keys included in a virtual keyboard and at least two keys adjacent to the reference key among the keys included in the virtual keyboard.

### Claim 6 (depends on 5)

The electronic apparatus of claim 5, wherein the processor is further configured to execute the at least one instruction to:
based on a distance between the position of the finger by the first motion and a position of the reference point set to the finger being less than a predetermined value, identify the reference key, among the plurality of keys mapped to the finger, as the key corresponding to the first motion, and
based on the distance between the position of the finger by the first motion and the position of the reference key set to the finger being greater than or equal to the predetermined value, identify a key positioned in a direction in which the finger moves from the reference key, from among the plurality of keys mapped to the finger, as the key corresponding to the first motion based on a moving direction of the finger by the first motion.

### Claim 7 (depends on 1)

The electronic apparatus of claim 1, wherein the processor is further configured to execute the at least one instruction to, in response to the identification that the pose of the plurality of fingers detected in the plurality of first image frames corresponds to the trigger pose, set the reference point for the finger based on a position of the plurality of detected fingers in the plurality of first image frames.

### Claim 8 (depends on 7)

The electronic apparatus of claim 7, wherein the processor is further configured to execute the at least one instruction to, in response to the identification that the pose of the plurality of fingers detected in the plurality of first image frames corresponds to the trigger pose:
control to display a virtual keyboard on the display; and
based on the key corresponding to the first motion being identified in the character input mode, control to display the identified key among keys included in the virtual keyboard to be distinguished from another key.

### Claim 9 (depends on 8)

The electronic apparatus of claim 8, wherein the processor is further configured to execute the at least one instruction to:
based on the key corresponding to the first motion being a deletion key to delete information displayed on the display, delete the information displayed on the display and adjust a transparency of the virtual keyboard based on a frequency of deleting the information.

### Claim 10 (depends on 7)

The electronic apparatus of claim 7, wherein:
the processor is further configured to execute the at least one instruction to, based on a preset user command being input, activate the camera and obtain the plurality of first image frames through the activated camera; and
the preset user command comprises a user command to display a document capable of text input to the display or a user command to select a text input window displayed on the display.

### Claim 11

A method of controlling an electronic apparatus, the method comprising:
detecting a plurality of fingers in a plurality of first image frames obtained through a camera and, in response to an identification that a pose of the plurality of detected fingers corresponds to a trigger pose, entering a character input mode;
detecting a first motion of a finger among the plurality of fingers in a plurality of second image frames obtained through the camera in the character input mode;
identifying a key corresponding to the first motion, from among a plurality of keys mapped to the finger based on a position of the finger by the first motion and a reference point set to the finger;
displaying information corresponding to the identified key on a display; and
detecting the plurality of fingers in a plurality of third image frames obtained through the camera in the character input mode and, in response to an identification that the pose of the plurality of detected fingers corresponds to a predetermined pose, re-setting a reference point for respective fingers based on a position of the plurality of detected fingers in the plurality of third image frames.

### Claim 12 (depends on 11)

The method of claim 11, wherein the re-setting comprises, in response to an identification that the plurality of fingers detected in the plurality of third image frames maintain the predetermined pose for a predetermined time, re-setting the reference point for the respective fingers based on the position of the plurality of detected fingers in the plurality of third image frames.

### Claim 13 (depends on 11)

The method of claim 11, wherein the re-setting comprises identifying whether the pose of the plurality of detected fingers in the plurality of third image frames corresponds to the predetermined pose based on a position of a fingertip of a remaining finger other than a thumb among the plurality of fingers detected in the plurality of third image frames.

### Claim 14 (depends on 11)

The method of claim 11, wherein the re-setting comprises based on a second motion of the finger among the plurality of fingers being detected in a plurality of fourth image frames obtained through the camera after the reference point is re-set in the character input mode, identifying a key corresponding to the second motion, from among the plurality keys mapped to the finger, based on a position of the finger by the second motion and a position of a reference point reset to the finger.

### Claim 15 (depends on 11)

The method of claim 11, wherein the plurality of keys mapped to the finger comprises a reference key, determined based on a type of a hand including the finger and a type of the finger, among keys included in a virtual keyboard and at least two keys adjacent to the reference key among the keys included in the virtual keyboard.

### Claim 16 (depends on 15)

The method of claim 15, wherein the identifying comprises:
based on a distance between the position of the finger by the first motion and a position of the reference point set to the finger being less than a predetermined value, identifying the reference key, among the plurality of keys mapped to the finger, as the key corresponding to the first motion; and
based on the distance between the position of the finger by the first motion and the position of the reference key set to the finger being greater than or equal to the predetermined value, identifying a key positioned in a direction in which the finger moves from the reference key, from among the plurality of keys mapped to the finger, as the key corresponding to the first motion based on a moving direction of the finger by the first motion.

### Claim 17 (depends on 11)

The method of claim 11, further comprising:
in response to the identification that the pose of the plurality of fingers detected in the plurality of first image frames corresponds to the trigger pose, setting the reference point for the finger based on a position of the plurality of detected fingers in the plurality of first image frames.

### Claim 18 (depends on 17)

The method of claim 17, wherein:
the setting comprises, in response to the identification that the pose of the plurality of fingers detected in the plurality of first image frames corresponds to the trigger pose, displaying a virtual keyboard on the display; and
the displaying the virtual keyboard comprises, based on the key corresponding to the first motion being identified in the character input mode, displaying the identified key among keys included in the virtual keyboard to be distinguished from another key.

### Claim 19 (depends on 18)

The method of claim 18, wherein the displaying the virtual keyboard further comprises:
based on the key corresponding to the first motion being a deletion key to delete information displayed on the display, deleting the information displayed on the display and adjusting a transparency of the virtual keyboard based on a frequency of deleting the information.

### Claim 20 (depends on 17)

The method of claim 17, further comprising:
based on a preset user command being input, activating the camera and obtaining the plurality of first image frames through the activated camera,
wherein the preset user command comprises a user command to display a document capable of text input to the display or a user command to select a text input window displayed on the display.
