# US 2025/0155975 A1 · Detecting user input from multi-modal hand bio-metrics

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0155975 A1
- Document ID: US20250155975A1
- Status: Published application
- Kind: A1
- Application number: 19/023,365
- Filed: 2025-01-16
- Published: 2025-05-15
- Earliest claimed: 2022-03-15
- Assignees: Doublepoint Technologies Oy
- Applicants: Doublepoint Technologies Oy
- Inventors: Jamin Hu; Ville Klar; Eemil Visakorpi; Lauri Tuominen
- Web page: https://patentgrep.com/patent/US20250155975A1

## CPC

- G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06N 3/045: Combinations of networks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

According to an example aspect of the present invention, there is provided an input device and corresponding method which digitizes and transforms minute hand movements and gestures into user interface commands without interfering with the normal use of one's hands. The device and method may, for example, determine user actions based on detected user action characteristics from a plurality of sensors, where the sensors are preferably of different types.

## Claims

### Claim 1

A multimodal biometric measurement apparatus, the apparatus comprising:
a mounting component configured to be worn by a user,
at least one wrist contour sensor,
at least one bioacoustic sensor comprising a vibration sensor,
at least one inertial measurement unit, IMU, comprising an accelerometer and gyroscope, and
a controller comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to with the at least one processor, to cause the controller at least to:
receive a first sensor data stream from the at least one wrist contour sensor,
receive a second sensor data stream from the at least one bioacoustic sensor,
receive a third sensor data stream from the at least one inertial measurement unit,
wherein the at least one wrist contour sensor, the at least one bioacoustic sensor, and the at least one inertial measurement unit are arranged within a single housing and wherein the first, the second and the third sensor data stream are received concurrently, and
determine, based on at least one of the first, the second and the third sensor data stream, at least one characteristic of an user action,
determine, based on the determined at least one characteristic of an user action, at least one user action, and
generate at least one user interface, UI, command, based at least in part on the determined at least one user action.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein determining the at least one characteristic of an user action comprises providing at least one of the first, the second and the third sensor data stream as an input to at least one neural network trained to determine at least one characteristic of an user action as an output, wherein the output of the at least one neural network comprises a confidence value of the at least one characteristic of an user action.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein determining the at least one user action comprises providing the output of the at least one neural network to an event interpreter configured to determine the user action based on the determined user action characteristics and received contextual information.

### Claim 4 (depends on 2, 3)

The apparatus of claims 2-3, wherein the controller is configured to perform preprocessing on at least one of the first, the second and the third sensor data stream before providing the at least one preprocessed data stream as an input to the at least one neural network.

### Claim 5

The apparatus of any preceding claim, wherein the controller is configured so that at least one at least one of the first, the second and the third sensor data stream is preprocessed in a separate preprocessing sequence from the other data streams.

### Claim 6 (depends on 4, 5)

The apparatus of claims 4-5, wherein the controller is configured so that at least one preprocessing sequence communicates with at least one second preprocessing sequence, wherein the communication comprises adjusting, based on data from the at least one preprocessing sequence, the preprocessing parameters of the second preprocessing sequence.

### Claim 7

The apparatus of any preceding claim, wherein the controller is further configured so that the first sensor data stream is provided to a first neural network, the second sensor data stream is provided to a second neural network, and the third sensor data stream is provided to a third neural network or mathematical model.

### Claim 8 (depends on 2, 3, 4, 5, 6)

The apparatus of any of claims 2-6, wherein the controller is further configured so that the first, the second and the third sensor data stream are provided to the same neural network.

### Claim 9

The apparatus of any preceding claim, wherein the controller is further configured so that the event interpreter comprises at least one of: an inference engine, a decision tree.

### Claim 10

The apparatus of any preceding claim, wherein the wrist contour sensor comprises an array of profilometric sensors.

### Claim 11

The apparatus of any preceding claim, wherein the apparatus comprises a smartwatch.

### Claim 12

The apparatus of any preceding claim, wherein the wrist contour sensor comprises an array of profilometric sensors and wherein said array is arranged with the at least one bioacoustic sensor in an annular form on the surface of the single housing.

### Claim 13

The apparatus of any preceding claim, wherein the wrist contour sensor comprises an array of profilometric sensors comprising concentric or rectangular rings on the surface of the single housing, wherein the array comprises between 1 and 25 rings, preferably 2-6 rings, in particular 3 rings.

### Claim 14

The apparatus of any preceding claim, wherein the apparatus comprises an add-on device, wherein the apparatus is configured to be attached to another wearable by at least one of the following arrangements:
wherein the add-on apparatus is configured to be directly attached to the body of another wearable in such a manner that the sensors of the add-on apparatus may measure the user, for example so that the sensors are oriented towards the user;
wherein the add-on apparatus is configured to be attached to a mounting component of the other wearable, for example wherein the mounting component is a strap, in particular the same strap as another wearable; and/or
wherein the add-on apparatus is configured to be part of or connected to a mounting component of the other wearable, for example wherein the mounting component is a strap, in particular in a manner such that the add-on apparatus is connected on one side to the other wearable and to the strap on the other side.

### Claim 15

A method to generate UI commands, the method comprising:
receiving a first sensor data stream from at least one wrist contour sensor,
receiving a second sensor data stream from at least one bioacoustic sensor,
receiving a third sensor data stream from at least one inertial measurement unit,
wherein the at least one wrist contour sensor, the at least one bioacoustic sensor, the at least one inertial measurement unit are arranged within a single housing and wherein the first, the second and the third sensor data stream are received concurrently, and
determining, based on at least one of the first, the second and the third sensor data stream, at least one characteristic of an user action,
determining, based on the determined at least one characteristic of a user action, at least one user action, and
generating at least one user interface, UI, command, based at least in part on the determined at least one user action.

### Claim 16 (depends on 15)

The method of claim 15, wherein determining the at least one characteristic of an user action comprises providing at least one of the first, the second and the third sensor data stream as an input to at least one neural network trained to determine at least one characteristic of an user action as an output, wherein the output of the at least one neural network comprises a confidence value of the at least one characteristic of an user action.

### Claim 17 (depends on 15, 16)

The method of any of claims 15-16, wherein determining the at least one user action comprises providing the output of the at least one neural network to an event interpreter configured to determine the user action based on the determined user action characteristics.

### Claim 18 (depends on 15, 16, 17)

The method of any of claims 15-17, wherein preprocessing is performed on at least one of the first, the second and the third sensor data stream before providing the at least one preprocessed data stream as an input to the at least one neural network.

### Claim 19 (depends on 15, 16, 17, 18)

The method of any of claims 15-18, wherein at least one at least one of the first, the second and the third sensor data stream is preprocessed in a separate preprocessing sequence from the other data streams.

### Claim 20 (depends on 15, 16, 17, 18, 19)

The method of any of claims 15-19, wherein at least one preprocessing sequence communicates with at least one second preprocessing sequence, wherein the communication comprises adjusting the preprocessing parameters of the second preprocessing sequence.

### Claim 21 (depends on 15, 16, 17, 18, 19, 20)

The method of any of claims 15-20, wherein the first sensor data stream is provided to a first neural network, the second sensor data stream is provided to a second neural network, and the third sensor data stream is provided to a third neural network or mathematical model.

### Claim 22 (depends on 15, 16, 17, 18, 19, 20, 21)

The method of any of claims 15-21, wherein the first, the second and the third sensor data stream are provided to the same neural network.

### Claim 23 (depends on 15, 16, 17, 18, 19, 20, 21, 22)

The method of any of claims 15-22, wherein the controller is further configured so that the event interpreter comprises at least one of: an inference engine, a decision tree.

### Claim 24 (depends on 15, 16, 17, 18, 19, 20, 21, 22, 23)

The method of any of claims 15-23, wherein the single housing comprises a smartwatch.

### Claim 25 (depends on 15, 16, 17, 18, 19, 20, 21, 22, 23, 24)

The method of any of claims 15-24, wherein the wrist contour sensor comprises an array of profilometric sensors and wherein said array is arranged with the at least one bioacoustic sensor in an annular form on the surface of the single housing.

### Claim 26 (depends on 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25)

The method of any of claims 15-25, wherein the apparatus comprises an add-on device, wherein the apparatus is configured to be attached by at least one of the following arrangements:
wherein the add-on apparatus is configured to be directly attached to the body of another wearable in such a manner that the add-on apparatus may measure the user;
wherein the add-on apparatus is configured to be attached to a mounting component such as a strap, for example the same strap as another wearable; and/or
wherein the add-on apparatus is configured to be part of or connected to a mounting component such as a strap, for example in a manner such that the add-on apparatus is connected on one side to the other wearable and to the strap on the other side.

### Claim 27

A non-transitory computer readable medium having stored thereon a set of computer readable instructions that, when executed by at least one processor, cause an apparatus to at least:
receive a first sensor data stream from at least one wrist contour sensor,
receive a second sensor data stream from at least one bioacoustic sensor,
receive a third sensor data stream from at least one inertial measurement unit,
wherein the at least one wrist contour sensor, the at least one bioacoustic sensor, and the at least one inertial measurement unit are arranged within a single housing and wherein the first, the second and the third sensor data stream are received concurrently, and
determine, based on at least one of the first, the second and the third sensor data stream, at least one characteristic of a user action,
determine, based on the determined at least one characteristic of a user action, at least one user action, and
generate at least one user interface, UI, command, based at least in part on the determined at least one user action.
