Back to results
US 11,635,823 B1Granted
Detecting user input from multi-modal hand bio-metrics
AssigneePort 6 Oy
InventorsJamin Hu, Ville Klar, Eemil Visakorpi, Lauri Tuominen
Life of this patent
Estimating the term…
Abstract
According to an example aspect of the present invention, there is provided an input device and corresponding method which digitizes and transforms minute hand movements and gestures into user interface commands without interfering with the normal use of one's hands. The device and method may, for example, determine user actions based on detected user action characteristics from a plurality of sensors, where the sensors are preferably of different types.
Claims14
A multimodal biometric measurement apparatus, the apparatus comprising:
- a mounting component configured to be worn by a user,
- at least one wrist contour sensor,
- at least one bioacoustic sensor comprising a vibration sensor,
- at least one inertial measurement unit, IMU, comprising an accelerometer and gyroscope, and
- a controller comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to with the at least one processor, to cause the controller at least to:
- receive a first sensor data stream from the at least one wrist contour sensor,
- receive a second sensor data stream from the at least one bioacoustic sensor,
- receive a third sensor data stream from the at least one inertial measurement unit,
- wherein the first, the second and the third sensor data stream are received concurrently, and
- perform preprocessing, in at least one preprocessing sequence, on at least one of the first, the second or the third sensor data stream,
- wherein at least one first preprocessing sequence communicates with at least one second preprocessing sequence, wherein the communication comprises adjusting, based on data from the at least one first preprocessing sequence, the preprocessing parameters of the at least one second preprocessing sequence,
- determine, based on at least one of the first, the second or the third sensor data stream, at least one characteristic of an user action, wherein determining the at least one characteristic of an user action comprises providing at least one of the first, the second or the third sensor data stream as an input to at least one neural network trained to determine at least one characteristic of an user action as an output, wherein the output of the at least one neural network comprises a confidence value of the at least one characteristic of an user action, wherein at least one of the data streams is a preprocessed data stream,
- determine, based on the determined at least one characteristic of an user action, at least one user action, and
- generate at least one user interface, UI, command, based at least in part on the determined at least one user action.
11 more elements
A method to generate UI commands, the method comprising:
- receiving a first sensor data stream from at least one wrist contour sensor,
- receiving a second sensor data stream from at least one bioacoustic sensor,
- receiving a third sensor data stream from at least one inertial measurement unit,
- wherein the first, the second and the third sensor data stream are received concurrently, and
- performing preprocessing, in at least one preprocessing sequence, on at least one of the first, the second or the third sensor data stream,
- wherein at least one first preprocessing sequence communicates with at least one second preprocessing sequence, wherein the communication comprises adjusting, based on data from the at least one first preprocessing sequence, the preprocessing parameters of the at least one second preprocessing sequence,
- determining, based on at least one of the first, the second or the third sensor data stream, at least one characteristic of an user action, wherein determining the at least one characteristic of an user action comprises providing at least one of the first, the second or the third sensor data stream as an input to at least one neural network trained to determine at least one characteristic of an user action as an output, wherein the output of the at least one neural network comprises a confidence value of the at least one characteristic of an user action, wherein at least one of the data streams is a preprocessed data stream,
- determining, based on the determined at least one characteristic of a user action, at least one user action, and
- generating at least one user interface, UI, command, based at least in part on the determined at least one user action.
6 more elements
A non-transitory computer readable medium having stored thereon a set of computer readable instructions that, when executed by at least one processor, cause an apparatus to at least:
- receive a first sensor data stream from at least one wrist contour sensor,
- receive a second sensor data stream from at least one bioacoustic sensor,
- receive a third sensor data stream from at least one inertial measurement unit,
- wherein the first, the second and the third sensor data stream are received concurrently, and
- perform preprocessing, in at least one preprocessing sequence, on at least one of the first, the second or the third sensor data stream,
- wherein at least one first preprocessing sequence communicates with at least one second preprocessing sequence, wherein the communication comprises adjusting, based on data from the at least one first preprocessing sequence, the preprocessing parameters of the at least one second preprocessing sequence,
- determine, based on at least one of the first, the second or the third sensor data stream, at least one characteristic of a user action, wherein determining the at least one characteristic of an user action comprises providing at least one of the first, the second or the third sensor data stream as an input to at least one neural network trained to determine at least one characteristic of an user action as an output, wherein the output of the at least one neural network comprises a confidence value of the at least one characteristic of an user action, wherein at least one of the data streams is a preprocessed data stream,
- determine, based on the determined at least one characteristic of a user action, at least one user action, and
- generate at least one user interface, UI, command, based at least in part on the determined at least one user action.
6 more elements
Dependencies are inferred from the claim text.