US 9,753,545 B2 · Input device, input method, and storage medium

Abstract

An input device is a device determining an input by a user based on a motion of a hand of the user in picked-up images associated with an operation screen displaying a selectable area. The input device includes an input determiner (155) determining whether or not the input by the user is a select operation of a selectable area or is a gesture operation based on, in the picked-up images, a change in shape of the hand of the user caused by the moving hand. The determined input is transmitted to an application subjected to an operation through a transmitter (156).

Drawings

Claims6

Dependent claims

An input device that determines an input by a user based on a motion of a hand of the user in picked-up images associated with an operation screen having one or more defined selectable areas, the input device comprising:

a movement detector that detects a movement start time point of the hand and a movement halt time point thereof;
a shape comparator that compares the shape of the hand at the movement start time point with the shape of the hand at the movement halt time point, and calculates a shape difference degree representing a difference degree in shape or a shape matching degree representing a matching degree in shape;
a direction comparator that specifies the movement direction of the hand based on the position of the hand at the movement start time point and the position of the hand at the movement halt time point, specifies a direction of the selectable area with reference to the position of the hand at the movement start time point, compares the specified movement direction of the hand with the specified direction of the selectable area, and calculates a direction difference degree representing a difference degree in direction or a direction matching degree representing a matching degree in direction;
and
an input determiner that adds the shape difference degree or the shape matching degree and the direction difference degree or the direction matching degree with a predetermined weight, and determines whether or not the input by the user is a select operation of a selectable area or is a gesture operation different from the select operation based on the addition value.

2 more elements

a movement detector that detects a movement start time point of the hand and a movement halt time point thereof;
a shape comparator that compares, for each time interval in a set of predetermined time intervals, the shape of the hand at a current time point with the shape of the hand at a previous time point and calculates a shape difference degree representing a difference degree in shape or a shape matching degree representing a matching degree in shape;
a direction comparator that compares, for each time interval in the set of predetermined time intervals, the movement direction at the current time point with the movement direction at the previous time point, and calculates a direction difference degree representing a difference degree in direction or a direction matching degree representing a matching degree in direction;
and
an input determiner that:
for each time interval in the set of predetermined time intervals, adds the shape difference degree or the shape matching degree and the direction difference degree or the direction matching degree with a predetermined weight;
accumulatively adds the addition values for each time interval in the set of predetermined time intervals with the movement start time point being an accumulation start time point;
and
determines that the input by the user is the gesture operation when the accumulated addition value exceeds a predetermined threshold, and determines that the input by the user is the select operation of the selectable area when the accumulated addition value does not exceed the predetermined threshold at the movement halt time point or within a predetermined time period after the accumulation start time point.

6 more elements

A computer-readable non-transitory recording medium having stored therein a program, the program causing a computer that controls an input device determining an input by a user based on a motion of a hand of the user in picked-up images associated with an operation screen displaying one or more selectable areas to execute:

a movement detection function of detecting a movement start time point of the hand and a movement halt time point thereof;
a shape comparison function of comparing, for each time interval in a set of predetermined time intervals, the shape of the hand at a current time point with the shape of the hand at a previous time point and calculating a shape difference degree representing a difference degree in shape or a shape matching degree representing a matching degree in shape;
a direction comparison function of comparing, for each time interval in the set of predetermined time intervals, the movement direction at the current time point with the movement direction at the previous time point, and calculating a direction difference degree representing a difference degree in direction or a direction matching degree representing a matching degree in direction;
and
an input determination function of:
for each time interval in the set of predetermined time intervals, adding the shape difference degree or the shape matching degree and the direction difference degree or the direction matching degree with a predetermined weight;
accumulatively adding the addition values for each time interval in the set of predetermined time intervals with the movement start time point being an accumulation start time point;
and
determining that the input by the user is the gesture operation when the accumulated addition value exceeds a predetermined threshold, and determining that the input by the user is the select operation of the selectable area when the accumulated addition value does not exceed the predetermined threshold at the movement halt time point or within a predetermined time period after the accumulation start time point.

6 more elements

Dependencies are inferred from the claim text.

Description12,133 words

Summary1,023 words

#The present application is a national stage application of International Application No. PCT/JP2013/071893 entitled “INPUT DEVICE, INPUT METHOD, AND STORAGE MEDIUM,” filed on Aug. 13, 2013, which claims the benefit of priority of Japanese Patent Application No. 2012-181223, filed on Aug. 17, 2012, the disclosures of which are incorporated herein in their entirety by reference thereto.

#The present disclosure relates to an input device, an input method, and a non-transitory recording medium.

#An input device that determines an operation instruction from a user based on the motion of a hand of the user is known. For example, Patent Literature 1 discloses an image recognition device that determines an operation by a user based on the hand motion of the user operating a virtual operation screen depicted using a three-dimensional display. In addition, Patent Literature 2 discloses a display control device that sets a way of arrangement of images to be displayed on a display based on a picked-up gesture of a user. Still further, Patent Literature 3 discloses an information processing device that determines whether or not a touch operation by a user is a depress operation (tap operation) or a gesture operation based on the movement amount of the touch operation to a touch panel.

Brief description of the drawings36 figures

#FIG. 1 is a diagram illustrating an input condition through an input device according to an embodiment of the present disclosure;

#FIG. 2A is a diagram illustrating an operation screen displayed on a display of the input device;

#FIG. 2B is a diagram illustrating a picked-up image picked up by an imager of the input device;

Detailed description10,331 words

#An input device according to an embodiment of the present disclosure will be explained below with reference to the accompanying drawings.

#An input device 100 of this embodiment includes a function for determining an input (for example, a user's operation instruction to a device, a selection within selectable areas displaying commands, icons, and the like) from a user to a device based on a hand motion of the user picked up by a camera. The input device 100 is, for example, a personal computer having an external appearance as illustrated in FIG. 1. First, an outline of the process executed by the input device 100 will be explained with reference to FIGS. 2 and 3.

#The input device 100 includes a display 120 that displays, for example, an operation screen illustrated in FIG. 2A and displaying images (icons and buttons) in the selectable area subjected to a selection, and an imager 130 that picks up, for example, an image illustrated in FIG. 2B. As illustrated in FIG. 2C, the image picked up (hereinafter, referred to as a “picked-up image”) is associated with the operation screen. For example, the input device 100 associates each coordinate point on the picked-up image with a coordinate point on the operation screen through a coordinate conversion like affine transformation. Next, when a fingertip of a user is reflected on a portion of the picked-up image, this is accepted as an operation pointing out the associated portion on the operation screen. Hence, an operation of specifying a position on the operation screen can be input without a pointing device, such as a mouse or a touch panel. The selectable area is set on the operation screen in accordance with the condition of an application that is a target of an input operation. The selectable area may simultaneously display, for example, an image of a depressable button, an image of an icon subjected to selection, and the like. Alternatively, in the cases of portions recognizable for the user, such as an upper end of the screen and a lower end thereof, a setting of not displaying a special image in the selectable area is possible. The input device 100 communicates, for example, an application to obtain the coordinates of the selectable area, and stores the obtained coordinates in a RAM. In the following explanation, it is presumed that the coordinates of the selectable area are obtainable in any time with reference to the predetermined area in the RAM. As a specific kind of an operation input, the input device 100 determines, based on the hand motion of the user in the picked-up images, whether or not an input by the user is a select operation or a gesture operation different therefrom. In this case, a “gesture operation” is a predetermined action like a body language signal and a hand gesture defined in advance to operate the device, for example, a circular motion of a hand as illustrated in FIG. 3A, a motion of pivoting a hand around a wrist and an elbow with a finger pointing as illustrated in FIG. 3B (hereinafter, this motion will be referred to as a “flick motion”), and a motion of rotating a hand (hereinafter, referred to as a “rolling motion”), and the like. In addition, a “select operation” (also referred to as a “pointing operation”) is a user's motion of selecting within the selectable area displayed on the screen. The “select operation” includes a hand motion of the user while moving to a point area.

Facts

Application

14/422,124Filed 13 Aug 2013

Priority

JP 2012-18122317 Aug 2012

Published

US 2015/0234469 A120 Aug 2015

Granted

5 Sep 2017

Term ends

13 Aug 2033

Estimated · how

20 years from earliest known nonprovisional domestic filing plus recorded patent term adjustment. Maintenance-fee lapses, terminal disclaimers and statutory extensions have not been checked. The continuity chain may be incomplete; this is not an enforceability determination.

Classification8 codes

G06F 3/017
Input arrangements or combined input and output arrangements for interaction between user and computer: gesture based interaction, e.g. based on a…
G06F 3/011
Input arrangements or combined input and output arrangements for interaction between user and computer: arrangements for interaction with the…
G06F 3/0304
Arrangements for converting the position or the displacement of a member into a coded form: detection arrangements using opto-electronic means
G06F 3/0482
Based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like…:…

Timeline

17 Aug 2012Priority claimedJP 2012-181223
13 Aug 2013Application filedUS 14/422,124
18 Feb 2015Assigned to NEC Solutions Innovators, Ltd.assignment of assignor's interest
20 Aug 2015PublishedUS 2015/0234469 A1
5 Sep 2017GrantedUS 9,753,545 B2
12 Oct 2026TodayWithin its term
13 Aug 2033Term endsEstimated

Input device, input method, and storage medium

Life of this patent

Input device, input method, and storage medium

Life of this patent

Life of this patent

Abstract

Drawings

Drawings

Claims6

Description12,133 words

Drawings

Classification8 codes

Life of this patent

Description12,133 words