# US 2024/0004478 A1 · Gesture interaction method, apparatus and electronic device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0004478 A1
- Document ID: US20240004478A1
- Status: Published application
- Kind: A1
- Application number: 18/343,183
- Filed: 2023-06-28
- Published: 2024-01-04
- Earliest claimed: 2022-06-29
- Assignees: Beijing Zitiao Network Technology Co., Ltd.
- Applicants: Beijing Zitiao Network Technology Co.. Ltd.
- Inventors: Chin-Wei LIU
- Web page: https://patentgrep.com/patent/US20240004478A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method for gesture interaction is proposed. The method includes: obtaining a first reality scene image, and in response to a detection of a first gesture that matches a first predefined gesture in the first reality scene image, displaying, at a first position corresponding to the first gesture, an interface identifier; and obtaining at least one frame of reality scene images, and in response to a detection of a second gesture that matches a second predefined gesture in the at least one frame of reality scene images, displaying the interactive interface at the first position, an operation position of the second gesture corresponding to the first position where the interface identifier is located, the interactive interface comprising at least one functional control.

## Claims

### Claim 1

A gesture interaction method, comprising:
obtaining a first reality scene image, and in response to a detection of a first gesture that matches a first predefined gesture in the first reality scene image, displaying, at a first position corresponding to the first gesture, an interface identifier indicating an interactive interface; and
obtaining at least one frame of reality scene images, and in response to a detection of a second gesture that matches a second predefined gesture in the at least one frame of reality scene images, displaying the interactive interface at the first position, an operation position of the second gesture corresponding to the first position where the interface identifier is located, the interactive interface comprising at least one functional control.

### Claim 2 (depends on 1)

The method according to claim 1, further comprising: after the interactive interface is displayed at the first position,
obtaining a second reality scene image, and
in response to a detection of a first touch-control gesture in the second reality scene image, and a touch-control position of the first touch-control gesture corresponding to a position of a target functional control in the interactive interface, performing an operation corresponding to the target functional control, the target functional control being one of the at least one functional control.

### Claim 3 (depends on 1)

The method according to claim 1, wherein
the second gesture is a pull gesture towards a first direction, and the operation position is a pull starting position of the pull gesture, or
the second gesture is a slide gesture towards a second direction, and the operation position is a slide starting position of the slide gesture, or
the second gesture is a click gesture, and the operation position is a click position.

### Claim 4 (depends on 1)

The method according to claim 1, further comprising: after the interactive interface is displayed at the first position,
in a case that the first gesture moves, determining motion parameters of the first gesture, determining a target position corresponding to the first gesture based on the first position and the motion parameters, canceling a display of the interactive interface at the first position, and updating the display of the interactive interface at the target position;
wherein the motion parameters comprise: a motion direction and a motion distance.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the interactive interface further comprises a cancel control, and the method further comprises: after the interactive interface is displayed at the first position,
obtaining a third reality scene image, and in response to a detection of a second touch-control gesture in the third reality scene image, and a touch-control position of the second touch-control gesture corresponding to a position of the cancel control in the interactive interface, canceling the display of the interactive interface.

### Claim 6 (depends on 3)

The method according to claim 3, further comprising: after the interactive interface is displayed at the first position,
obtaining a plurality of consecutive frames of reality scene images, and
in response to a detection of a third gesture that matches a third predefined gesture in the plurality of consecutive frames of reality scene images, canceling the display of the interactive interface, an operation position of the third gesture corresponding to an interface boundary of the interactive interface in a target direction, the target direction being the first direction or the second direction;
wherein the second gesture is a pull gesture towards the first direction, and the third gesture is a pull gesture towards an opposite direction of the first direction; or, the second gesture is a slide gesture towards the second direction, and the third gesture is a slide gesture towards an opposite direction of the second direction.

### Claim 7 (depends on 1)

The method according to claim 1, further comprising: after the interactive interface is displayed at the first position,
obtaining, within a first duration, a plurality of consecutive frames of reality scene images, and in response to a failure of a continuous detection of the first gesture in the plurality of consecutive frames of reality scene images, canceling the display of the interactive interface after the first duration; or
obtaining reality scene images in real-time, in response to that a deflection angle for detecting the first gesture in two adjacent reality scene images obtained in real-time is greater than a predefined angle, canceling the display of the interactive interface; or,
obtaining, within a second duration, a plurality of consecutive frames of reality scene images, and in response to a failure of a detection of a touch-control gesture in the plurality of consecutive frames of reality scene images, canceling the display of the interactive interface after the second duration; or
obtaining, within a third duration, a plurality of consecutive frames of reality scene images, and in response to a failure of a detection, in the plurality of continuous frames of real scene images, of a correspondence between a touch-control position and a touch-control gesture in the interactive interface, canceling the display of the interactive interface after the third duration.

### Claim 8

An electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, causes the processor to perform a gesture interaction method comprising:
obtaining a first reality scene image, and in response to a detection of a first gesture that matches a first predefined gesture in the first reality scene image, displaying, at a first position corresponding to the first gesture, an interface identifier indicating an interactive interface; and
obtaining at least one frame of reality scene images, and in response to a detection of a second gesture that matches a second predefined gesture in the at least one frame of reality scene images, displaying the interactive interface at the first position, an operation position of the second gesture corresponding to the first position where the interface identifier is located, the interactive interface comprising at least one functional control.

### Claim 9 (depends on 8)

The electronic device according to claim 8, wherein the method further comprises: after the interactive interface is displayed at the first position,
obtaining a second reality scene image, and
in response to a detection of a first touch-control gesture in the second reality scene image, and a touch-control position of the first touch-control gesture corresponding to a position of a target functional control in the interactive interface, performing an operation corresponding to the target functional control, the target functional control being one of the at least one functional control.

### Claim 10 (depends on 8)

The electronic device according to claim 8, wherein
the second gesture is a pull gesture towards a first direction, and the operation position is a pull starting position of the pull gesture, or
the second gesture is a slide gesture towards a second direction, and the operation position is a slide starting position of the slide gesture, or
the second gesture is a click gesture, and the operation position is a click position.

### Claim 11 (depends on 8)

The electronic device according to claim 8, wherein the method further comprises: after the interactive interface is displayed at the first position,
in a case that the first gesture moves, determining motion parameters of the first gesture, determining a target position corresponding to the first gesture based on the first position and the motion parameters, canceling a display of the interactive interface at the first position, and updating the display of the interactive interface at the target position;
wherein the motion parameters comprise: a motion direction and a motion distance.

### Claim 12 (depends on 8)

The electronic device according to claim 8, wherein the interactive interface further comprises a cancel control, and the method further comprises: after the interactive interface is displayed at the first position,
obtaining a third reality scene image, and in response to a detection of a second touch-control gesture in the third reality scene image, and a touch-control position of the second touch-control gesture corresponding to a position of the cancel control in the interactive interface, canceling the display of the interactive interface.

### Claim 13 (depends on 10)

The electronic device according to claim 10, wherein the method further comprises: after the interactive interface is displayed at the first position,
obtaining a plurality of consecutive frames of reality scene images, and
in response to a detection of a third gesture that matches a third predefined gesture in the plurality of consecutive frames of reality scene images, canceling the display of the interactive interface, an operation position of the third gesture corresponding to an interface boundary of the interactive interface in a target direction, the target direction being the first direction or the second direction;
wherein the second gesture is a pull gesture towards the first direction, and the third gesture is a pull gesture towards an opposite direction of the first direction; or, the second gesture is a slide gesture towards the second direction, and the third gesture is a slide gesture towards an opposite direction of the second direction.

### Claim 14 (depends on 8)

The electronic device according to claim 8, wherein the method further comprises: after the interactive interface is displayed at the first position,
obtaining, within a first duration, a plurality of consecutive frames of reality scene images, and in response to a failure of a continuous detection of the first gesture in the plurality of consecutive frames of reality scene images, canceling the display of the interactive interface after the first duration; or
obtaining reality scene images in real-time, in response to that a deflection angle for detecting the first gesture in two adjacent reality scene images obtained in real-time is greater than a predefined angle, canceling the display of the interactive interface; or,
obtaining, within a second duration, a plurality of consecutive frames of reality scene images, and in response to a failure of a detection of a touch-control gesture in the plurality of consecutive frames of reality scene images, canceling the display of the interactive interface after the second duration; or
obtaining, within a third duration, a plurality of consecutive frames of reality scene images, and in response to a failure of a detection, in the plurality of continuous frames of real scene images, of a correspondence between a touch-control position and a touch-control gesture in the interactive interface, canceling the display of the interactive interface after the third duration.

### Claim 15

A computer-readable storage medium, storing a computer program thereon that, when executed by a processor, causes the processor to perform a gesture interaction method comprising:
obtaining a first reality scene image, and in response to a detection of a first gesture that matches a first predefined gesture in the first reality scene image, displaying, at a first position corresponding to the first gesture, an interface identifier indicating an interactive interface; and
obtaining at least one frame of reality scene images, and in response to a detection of a second gesture that matches a second predefined gesture in the at least one frame of reality scene images, displaying the interactive interface at the first position, an operation position of the second gesture corresponding to the first position where the interface identifier is located, the interactive interface comprising at least one functional control.

### Claim 16 (depends on 15)

The computer-readable storage medium according to claim 15, wherein the method further comprises: after the interactive interface is displayed at the first position,
obtaining a second reality scene image, and
in response to a detection of a first touch-control gesture in the second reality scene image, and a touch-control position of the first touch-control gesture corresponding to a position of a target functional control in the interactive interface, performing an operation corresponding to the target functional control, the target functional control being one of the at least one functional control.

### Claim 17 (depends on 15)

The computer-readable storage medium according to claim 15, wherein
the second gesture is a pull gesture towards a first direction, and the operation position is a pull starting position of the pull gesture, or
the second gesture is a slide gesture towards a second direction, and the operation position is a slide starting position of the slide gesture, or
the second gesture is a click gesture, and the operation position is a click position.

### Claim 18 (depends on 15)

The computer-readable storage medium according to claim 15, wherein the method further comprises: after the interactive interface is displayed at the first position,
in a case that the first gesture moves, determining motion parameters of the first gesture, determining a target position corresponding to the first gesture based on the first position and the motion parameters, canceling a display of the interactive interface at the first position, and updating the display of the interactive interface at the target position;
wherein the motion parameters comprise: a motion direction and a motion distance.

### Claim 19 (depends on 15)

The computer-readable storage medium according to claim 15, wherein the interactive interface further comprises a cancel control, and the method further comprises: after the interactive interface is displayed at the first position,
obtaining a third reality scene image, and in response to a detection of a second touch-control gesture in the third reality scene image, and a touch-control position of the second touch-control gesture corresponding to a position of the cancel control in the interactive interface, canceling the display of the interactive interface.

### Claim 20 (depends on 17)

The computer-readable storage medium according to claim 17, wherein the method further comprises: after the interactive interface is displayed at the first position,
obtaining a plurality of consecutive frames of reality scene images, and
in response to a detection of a third gesture that matches a third predefined gesture in the plurality of consecutive frames of reality scene images, canceling the display of the interactive interface, an operation position of the third gesture corresponding to an interface boundary of the interactive interface in a target direction, the target direction being the first direction or the second direction;
wherein the second gesture is a pull gesture towards the first direction, and the third gesture is a pull gesture towards an opposite direction of the first direction; or, the second gesture is a slide gesture towards the second direction, and the third gesture is a slide gesture towards an opposite direction of the second direction.
