# US 2019/0317633 A1 · Method and system for identifying tap events on touch panel, and touch-controlled end project

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0317633 A1
- Document ID: US20190317633A1
- Status: Published application
- Kind: A1
- Application number: 16/382,702
- Filed: 2019-04-12
- Published: 2019-10-17
- Earliest claimed: 2018-04-13
- Assignees: Silicon Integrated Systems Corp.
- Applicants: Silicon Integrated Systems Corp
- Inventors: Tsung-Hua TSAI; Ying-Jyh YEH
- Web page: https://patentgrep.com/patent/US20190317633A1

## CPC

- G06F 3/0412: {Digitisers structurally integrated in a display}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06F 3/03547: {Touch pads, in which fingers can move on a surface}
- G06F 3/0416: {Control or interface arrangements specially adapted for digitisers}
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06N 3/02: Neural networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06F 3/0446: {using a grid-like structure of electrodes in at least two directions, e.g. using row and column electrodes}
- G06N 3/048: Activation functions
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method for identifying tap events on a touch panel includes measuring vibration signals for tap events on the touch panel to collect the tap events and record types of the tap events as samples; generating a sample set including a plurality of samples; using the sample set to train a deep neural network to determine an optimized weighting parameter group; taking the deep neural network and the optimized weighting parameter group as a tapping classifier and deploying it to a touch-controlled end product; and obtaining a predicted tap type based on a vibration signal and an image formed by touch sensing values detected by the touch-controlled end product to which a tap operation is performed. The present disclosure also provides a system corresponding to the identifying method, and a touch-controlled end product.

## Claims

### Claim 1

A method for identifying tap events on a touch panel, comprising:
detecting various tap events on the touch panel to obtain a plurality of measured vibration signals and recording types of the tap events as classification labels;
taking the vibration signal and the classification label that correspond to one of the tap events as a sample, and generating a sample set comprising a plurality of samples;
taking the samples of the sample set as inputs and a freely-selected weighting parameter group as adjusting parameters and inputting them into a deep neural network for training the deep neural network by adjusting the weighting parameter group by way of a backpropagation algorithm;
taking out the samples of the sample set in batches to train the deep neural network and fine tune the weighting parameter group to determine an optimized weighting parameter group;
taking the deep neural network and the optimized weighting parameter group as a model and deploying the model to a touch-controlled end product;
detecting a vibration signal generated by a tap operation performed to the touch-controlled end product and generating an image formed by a plurality of touch sensing values corresponding to the tap operation; and
inputting the vibration signal corresponding to the tap operation into the model to obtain a first result and analyzing the image formed by the touch sensing values to obtain a second result, and obtaining a predicted tap type based on the first result and the second result.

### Claim 2 (depends on 1)

The method according to claim 1, wherein after the step of obtaining the predicted tap type, the method further comprises:
executing a predetermined operation corresponding to the predicted tap type.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the predetermined operation is selected from a group consisting of opening or closing a menu, changing a brush color, and changing a brush size.

### Claim 4 (depends on 1)

The method according to claim 1, wherein types of taps are classified by at least one of the number of taps, the number of fingers simultaneously conducting a tap, a portion of a finger conducting the tap, and a tapping angle.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the deep neural network comprises a convolutional neural network.

### Claim 6

A method for identifying tap events on a touch panel, comprising:
detecting various tap events on the touch panel to obtain a plurality of measured vibration signals, using the touch panel to detect each of the tap events to obtain a plurality of touch sensing values and transforming the touch sensing values into an image, and recording types of the tap events as classification labels;
taking the vibration signal, the image, and the classification label that correspond to one of the tap events as a sample, and generating a sample set comprising a plurality of samples;
taking the samples of the sample set as inputs and a freely-selected weighting parameter group as adjusting parameters and inputting them into a deep neural network for training the deep neural network by adjusting the weighting parameter group by way of a backpropagation algorithm; and
taking out the samples of the sample set in batches to train the deep neural network and fine tune the weighting parameter group to determine an optimized weighting parameter group.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the step of training the deep neural network comprises:
inputting features of the vibration signal corresponding to one of the tap events into a first sub network which is a one-dimensional sub network;
inputting features of the image corresponding to the touch sensing values of one of the tap events into a second sub network which is a two-dimensional sub network;
inputting an output of the first sub network and an output of the second sub network into a flat layer for data flattening to transform the outputs of the first sub network and the second sub network into one-dimensional data; and
inputting an output of the flat layer into a multi-layer perceptron to perform classification prediction for the tap events.

### Claim 8 (depends on 7)

The method according to claim 7, wherein the first sub network and the second sub network comprise a convolutional neural network.

### Claim 9 (depends on 6)

The method according to claim 6, wherein the step of transforming the touch sensing values into the image comprises:
transforming the touch sensing values which correspond to a local area where the tap event is located at, into the image.

### Claim 10 (depends on 6)

The method according to claim 6, further comprising:
taking the deep neural network and the optimized weighting parameter group as a model and deploying the model to a touch-controlled end product; and
receiving a vibration signal generated by a tap operation performed to the touch-controlled end product and an image of touch sensing values obtained from the touch-controlled end product, and inputting the vibration signal corresponding to the tap operation and the image from the touch-controlled end product into the model to obtain a predicted tap type.

### Claim 11 (depends on 10)

The method according to claim 10, wherein after the step of obtaining the predicted tap type, the method further comprises:
executing a predetermined operation corresponding to the predicted tap type.

### Claim 12 (depends on 11)

The method according to claim 11, wherein the predetermined operation is selected from a group consisting of opening or closing a menu, changing a brush color, and changing a brush size.

### Claim 13 (depends on 6)

The method according to claim 6, wherein types of taps are classified by at least one of the number of taps, the number of fingers simultaneously conducting a tap, a portion of a finger conducting the tap, and a tapping angle.

### Claim 14

A system for identifying tap events on a touch panel, comprising:
a touch panel;
a vibration sensor, deployed with the touch panel, wherein the vibration sensor is configured to detect various tap events on the touch panel to obtain a plurality of measured vibration signals, and the touch panel is configured to detect each of the tap events to obtain a plurality of touch sensing values and the touch sensing values are transformed into an image;
a processor coupled to the vibration sensor and the touch panel, configured to receive the vibration signals transmitted from the vibration sensor and a plurality of images transmitted from the touch panel; and
a memory, connected to the processor, the memory comprising a plurality of program instructions executable by the processor, the processor executing the program instructions to perform a method comprising:
recording types of the tap events as classification labels in the memory;
taking the vibration signal, the image, and the classification label that correspond to one of the tap events as a sample, and generating a sample set comprising a plurality of samples;
taking the samples of the sample set as inputs and a freely-selected weighting parameter group as adjusting parameters and inputting them into a deep neural network for training the deep neural network by adjusting the weighting parameter group by way of a backpropagation algorithm; and
taking out the samples of the sample set in batches to train the deep neural network and fine tune the weighting parameter group to determine an optimized weighting parameter group.

### Claim 15 (depends on 14)

The system according to claim 14, wherein the deep neural network comprises:
a first sub network, configured to receive features of the vibration signal corresponding to one of the tap events, wherein the first sub network is a one-dimensional sub network;
a second sub network, configured to receive features of the image corresponding to the touch sensing values of one of the tap events, wherein the second sub network is a two-dimensional sub network;
a flat layer, configured to flatten an output of the first sub network and an output of the second sub network to transform the outputs of the first sub network and the second sub network into one-dimensional data; and
a multi-layer perceptron, configured to receive an output of the flat layer to perform classification prediction for the tap events to obtain a predicted classification label.

### Claim 16 (depends on 15)

The system according to claim 15, wherein the first sub network and the second sub network comprise a convolutional neural network.

### Claim 17 (depends on 14)

The system according to claim 14, wherein the method further comprises:
transforming the touch sensing values which correspond to a local area where the tap event is located at, into the image.

### Claim 18 (depends on 14)

The system according to claim 14, wherein types of taps are classified by at least one of the number of taps, the number of fingers simultaneously conducting a tap, a portion of a finger conducting the tap, and a tapping angle.

### Claim 19 (depends on 6)

A touch-controlled end product, comprising:
a touch interface;
a vibration sensor, deployed with the touch interface, the vibration sensor configured to detect a vibration signal generated by a tap operation performed to the touch interface configured to sense the tap operation to obtain a plurality of touch sensing values which are then transformed into an image; and
a controller coupled to the vibration sensor and the touch interface, wherein a deep neural network corresponding to the deep neural network according to claim 6 is deployed in the controller, and the controller is configured to take the corresponding deep neural network and the optimized weighting parameter group obtained according to claim 6 as a model and input the vibration signal corresponding to the tap operation and the image from the touch interface into the model to obtain a predicted tap type.

### Claim 20 (depends on 19)

The touch-controlled end product according to claim 19, wherein the controller is further configured to execute a predetermined operation corresponding to the predicted tap type.
