# US 12,118,153 B2 · Dynamic gesture identification method, gesture interaction method and interaction system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,153 B2
- Document ID: US12118153B2
- Status: Granted
- Kind: B2
- Application number: 18/241,395
- Filed: 2023-09-01
- Published: 2023-12-28
- Granted: 2024-10-15
- Earliest claimed: 2021-03-08
- Assignees: Omnivision Sensor Solution (Shanghai) Co., Ltd
- Applicants: OmniVision Sensor Solution (Shanghai) Co., Ltd
- Inventors: Shunshun Shi; Bin Wu; Rui Jiang; Qinyi Wang; Wei Zhang; Chao Gong
- Web page: https://patentgrep.com/patent/US12118153B2

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 10/82: using neural networks
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06N 3/045: Combinations of networks
- G06N 3/09: Supervised learning
- G06V 2201/07: Target detection
- Y02D 10/00: Energy efficient computing, e.g. low power processors, power management or thermal management
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE

## Abstract

The present disclosure provides a dynamic gesture identification method, a gesture interaction method and an interaction system. The interaction system includes: a dynamic vision sensor configured to trigger an event in accordance with movement of an object in a field of view relative to the dynamic vision sensor, and output an event data flow; a hand detection module configured to process the received event data flow to determine an initial position of a hand; a hand tracking module configured to determine a series of state vectors in accordance with the initial position of the hand; a gesture identification module configured to create an event cloud in accordance with event data corresponding to the obtained state vector, and process the event cloud to identify the gesture; and a command execution module configured to execute a corresponding operating command in accordance with the identified gesture.

## Claims

### Claim 1

A dynamic gesture tracking method, comprising:
processing an event data flow from a dynamic vision sensor at a predetermined time interval so as to generate a series of event cumulative graphs;
determining an initial position of a hand in accordance with the event cumulative graphs;
determining a hand measurement position in an initial frame in accordance with the initial position of the hand, the initial frame being a next event cumulative graph adjacent to an event cumulative graph corresponding to the initial position of the hand; and
determining a state vector indicating a movement state of the hand in each event cumulative graph in accordance with the initial position of the hand and the hand measurement position,
wherein the processing the event data flow from the dynamic vision sensor at the predetermined time interval so as to generate a series of event cumulative graphs comprises: dividing the event data flow from the dynamic vision sensor at the predetermined time interval so as to obtain a plurality of data segments; and generating the event cumulative graph in accordance with event data in each data segment, wherein the event data is triggered by movement of an object in a field of view relative to the dynamic vision sensor, and the event data comprises a coordinate position and a timestamp of a triggered event, wherein the generating the event cumulative graph in accordance with the event data in each data segment comprises: creating an initial image having a predetermined size, and assigning pixel values of the initial image as zero, the predetermined size being determined in accordance with a size of a pixel unit array of the dynamic vision sensor; determining the quantity of times for which an event at a same position as a coordinate position of each pixel in the initial image is triggered within the data segment; and updating a pixel value of each pixel in accordance with the quantity of times, so as to generate the event cumulative graph,
wherein the determining the initial position of the hand in accordance with the event cumulative graphs comprises: determining a high-energy density point in a current event cumulative graph; determining at least one hand detection region in the event cumulative graph through a target detection network; generating at least one rectangular region with the high-energy density point as a center in accordance with a size of the hand detection region; calculating an overlapping ratio of the rectangular region to the hand detection region; in the case that the overlapping ratio is greater than a predetermined value, taking a position where the high-energy density point is located as the initial position of the hand; and in the case that the overlapping ratio is not greater than the predetermined value, calculating a high-energy density point and a hand detection region in a next event cumulative graph, and determining the initial position of the hand in accordance with an overlapping ratio of the hand detection region to the rectangular region.

### Claim 2 (depends on 1)

The dynamic gesture tracking method according to claim 1, wherein the determining the high-energy density point in the current event cumulative graph comprises:
generating an energy density graph in accordance with the current event cumulative graph and at least one event cumulative graph adjacent to the current event cumulative graph;
dividing the energy density graph into a plurality of regions having a same size;
determining an energy density in each region in accordance with a sum of pixel values in the region and the quantity of non-zero pixels;
removing regions where the energy density does not meet a predetermined condition from the regions, so as to obtain remaining regions; and
selecting a region having a largest energy density in the remaining regions, and calculating an average value of coordinate values of the pixels in the region as the high-energy density point.

### Claim 3 (depends on 1)

The dynamic gesture tracking method according to claim 1, wherein the determining the hand measurement position in the initial frame in accordance with the initial position of the hand comprises:
determining a first search region in the initial frame in accordance with the initial position of the hand;
updating the first search region iteratively through determining whether event data in the initial frame is located within the first search region, so as to obtain at least one new first search region;
obtaining a set of first search regions in accordance with the first search region and all the new first search regions; and
determining the hand measurement position in accordance with the quantity of all event data in the set of first search regions and coordinate positions of the event data.

### Claim 4 (depends on 3)

The dynamic gesture tracking method according to claim 3, wherein the updating the first search region iteratively through determining whether the event data in the initial frame is located within the first search region so as to obtain at least one new first search region comprises:
determining whether each piece of event data in the initial frame is located within the first search region in an output sequence of the pieces of event data, and determining in real time the quantity of pieces of event data in the first search region;
in the case that the quantity of pieces of event data in the first search region is greater than an update frequency threshold, updating the first search region so as to obtain the new first search region and a search center; and
determining whether the other pieces of event data in the initial frame is located within the new first search region, and determining the quantity of pieces of event data in the new first search region until all pieces of event data in the initial frame have been traversed.

### Claim 5 (depends on 3)

The dynamic gesture tracking method according to claim 3, wherein the determining the hand measurement position in accordance with the quantity of all event data in the set of first search regions and coordinate positions of the event data comprises determining an average value of the coordinate positions of the event data as the hand measurement position in accordance with the quantity of all event data in the set of first search regions and coordinate positions of the event data.

### Claim 6 (depends on 3)

The dynamic gesture tracking method according to claim 3, wherein the determining the first search region in the initial frame in accordance with the initial position of the hand comprises generating a square region in the initial frame with the initial position of the hand as a center and with a short side or a long side of the hand detection region being magnified by predetermined times as a side, so as to obtain the first search region.

### Claim 7 (depends on 1)

The dynamic gesture tracking method according to claim 1, wherein the determining the state vector indicating the movement state of the hand in each event cumulative graph in accordance with the initial position of the hand and the hand measurement position comprises:
determining a predicted state vector and a hand measurement position in the initial frame in accordance with the initial position of the hand and the hand measurement position;
adjusting the predicted state vector in the initial frame in accordance with the hand measurement position in the initial frame, so as to obtain a state vector in the initial frame;
calculating a predicted state vector and a hand measurement position in a current event cumulative graph with the initial frame as a start point in accordance with a state vector in a previous event cumulative graph; and
adjusting the predicted state vector in the current event cumulative graph in accordance with the hand measurement position in the current event cumulative graph, until the state vector is determined in each event cumulative graph.

### Claim 8 (depends on 1)

A dynamic gesture identification method, comprising:
obtaining a series of state vectors indicating a movement state of a hand through the dynamic gesture tracking method according to claim 1;
creating an event cloud in accordance with event data corresponding to the state vectors; and
processing the event cloud through a point cloud-based neural network so as to identify a gesture.

### Claim 9 (depends on 8)

The dynamic gesture identification method according to claim 8, wherein the creating the event cloud in accordance with the event data corresponding to the obtain state vectors comprises:
generating an event vector corresponding to each event cumulative graph in accordance with the event data corresponding to the state vector; and
creating the event cloud through a first quantity of adjacent event vectors.

### Claim 10 (depends on 9)

The dynamic gesture identification method according to claim 9, wherein the generating the event vector corresponding to each event cumulative graph in accordance with the event data corresponding to the state vector comprises:
generating a second search region with the event data corresponding to the state vector as a center;
sampling event data in the second search region so as to obtain a second quantity of pieces of event data; and
generating the event vector corresponding to each event cumulative graph in accordance with the second quantity of pieces of event data.

### Claim 11 (depends on 8)

A gesture interaction method, comprising:
identifying a current gesture of a user through the dynamic gesture identification method according to claim 8; and
executing a corresponding operation command in accordance with the identified gesture.

### Claim 12 (depends on 11)

An interaction system, which is configured to execute the gesture interaction method according to claim 11, comprising:
a dynamic vision sensor configured to trigger an event in accordance with movement of an object in a field of view relative to the dynamic vision sensor, and output an event data flow;
a hand detection module configured to process the received event data flow to determine an initial position of a hand;
a hand tracking module configured to determine a series of state vectors indicating a movement state of the hand in the event data flow in accordance with the initial position of the hand;
a gesture identification module configured to create an event cloud in accordance with event data corresponding to each state vector, and process the event cloud through a point cloud-based neural network, so as to identify a gesture; and
a command execution module configured to execute a corresponding operation command in accordance with the gesture.

### Claim 13 (depends on 12)

The interaction system according to claim 12, wherein the hand detection module is further configured to generate a detection region for detecting the hand in accordance with the state vector.

### Claim 14 (depends on 13)

The interaction system according to claim 13, further comprising a resetting module configured to, in the case that the hand fails to be detected by the hand detection module within a predetermined time period, indicate the hand detection module to enlarge the detection region, and in the case that the hand has been detected, reset the initial position of the hand.

### Claim 15 (depends on 12)

An intelligent device, comprising the interaction system according to claim 12.

### Claim 16 (depends on 1)

A computing device, comprising one or more processors, a memory, and one or more programs stored in the memory and executed by the one or more processors, wherein the one or more programs is executed by the one or more processors so as to implement the dynamic gesture tracking method according to claim 1.

### Claim 17 (depends on 1)

A non-transitory computer-readable storage medium storing therein one or more programs, wherein the one or more programs is executed by a computer so as to implement the dynamic gesture tracking method according to claim 1.