# US 2025/0148732 A1 · Virtual Operation Method, Electronic Device, and Non-Transitory Readable Storage Medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0148732 A1
- Document ID: US20250148732A1
- Status: Published application
- Kind: A1
- Application number: 19/018,140
- Filed: 2025-01-13
- Published: 2025-05-08
- Earliest claimed: 2022-07-14
- Assignees: Vivo Mobile Communication Co., Ltd.
- Applicants: Vivo Mobile Communication Co., Ltd.
- Inventors: Wei Li; Guorong Li; Lixin Gao
- Web page: https://patentgrep.com/patent/US20250148732A1

## CPC

- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/01: Input arrangements or combined input and output arrangements for interaction between user and computer (G06F3/16 takes precedence)
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A virtual operating method includes displaying a virtual touch interface. The virtual touch interface is associated with a position of a target plane in a physical space in which an electronic device is located; recognizing hand gesture information of a user; and performing a target operation in a case that position information, of a first region of the virtual touch interface, in the physical space and the hand gesture information meet a preset condition.

## Claims

### Claim 1

A virtual operating method, wherein the method comprises:
displaying a virtual touch interface, wherein the virtual touch interface is associated with a position of a target plane in a physical space in which an electronic device is located;
recognizing hand gesture information of a user; and
performing a target operation in a case that position information, of a first region of the virtual touch interface, in the physical space and the hand gesture information meet a preset condition.

### Claim 2 (depends on 1)

The method according to claim 1, wherein before the displaying the virtual touch interface, the method further comprises:
obtaining a first image, wherein the first image is an image of the physical space in which the electronic device is located;
calculating a first point cloud of the physical space in a first coordinate system based on the first image;
determining, from the first point cloud, a target point cloud located in a target coordinate range, wherein the target coordinate range is determined based on coordinate information of the electronic device in the first coordinate system; and
determining a target plane based on the target point cloud; wherein
the first coordinate system is a coordinate system corresponding to the physical space.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the first image is a depth image, and the depth image comprises N pieces of first depth information corresponding to N first pixels and N pieces of third coordinate information corresponding to the N first pixels; and
the calculating the first point cloud of the physical space in the first coordinate system based on the first image comprises:
calculating a second point cloud of the physical space in a second coordinate system based on the N pieces of first depth information and the N pieces of third coordinate information; and
converting the second point cloud into the first point cloud; wherein
the second coordinate system is a coordinate system corresponding to an image sensor of the electronic device.

### Claim 4 (depends on 2)

The method according to claim 2, wherein the determining the target plane based on the target point cloud comprises:
determining at least one first plane based on the target point cloud;
determining a plane meeting a first preset condition in the at least one first plane as a second plane; and
determining a corresponding position of a point cloud of the second plane in the physical space as the target plane; wherein the first preset condition comprises at least one of the following:
a plane whose included angle between its normal vector and a normal vector of a horizontal plane in the physical space is less than a preset value in the at least one first plane;
a plane perpendicular to a target direction in the at least one first plane; or
a plane whose plane size is in a preset range in the at least one first plane.

### Claim 5 (depends on 2)

The method according to claim 2, wherein the target coordinate range comprises at least one first coordinate range; and
the determining the target plane based on the target point cloud comprises:
determining at least one point cloud subset in the target point cloud based on the at least one first coordinate range, wherein each point cloud subset corresponds to each first coordinate range, and a point cloud subset comprises at least one point cloud;
determining at least one first plane based on a target point cloud subset in the at least one point cloud subset, wherein the target point cloud subset is a point cloud subset whose quantity of point clouds is greater than or equal to a preset quantity in the at least one point cloud subset; and
fusing a third plane in the at least one first plane and a fourth plane to obtain the target plane, wherein the fourth plane is a plane stored in the electronic device, and spatial positions of the third plane and the fourth plane in the physical space meet a second preset condition.

### Claim 6 (depends on 1)

The method according to claim 1, wherein before the displaying the virtual touch interface, the method further comprises:
obtaining a third image of the physical space, wherein the third image comprises an image of a user hand;
based on the third image, calculating a third point cloud of the physical space in a first coordinate system, wherein the third point cloud comprises position and posture information of the user hand;
determining a third plane in the third point cloud based on the position and posture information of the user hand; and
determining a screen corresponding to the third plane in the physical space as the target plane; wherein
the first coordinate system is a coordinate system corresponding to the physical space.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the recognizing the hand gesture information of the user comprises:
obtaining a fourth image of the physical space, wherein the fourth image comprises an image of a user hand; and
performing user hand gesture recognition on the fourth image to obtain at least one piece of hand joint point information.

### Claim 8 (depends on 7)

The method according to claim 7, wherein the at least one piece of hand joint point information comprises at least one first coordinate of at least one hand joint in a second coordinate system; and
the performing the target operation in the case that the position information, of the first region of the virtual touch interface, in the physical space and the hand gesture information meet the preset condition comprises:
determining the first region of the virtual touch interface in a case that a spatial distance in the physical space between the at least one first coordinate and the virtual touch interface meets the preset condition; and
performing the target operation based on operation information corresponding to the first region.

### Claim 9

An electronic device, comprising a processor and a memory, wherein a program or instructions are stored in the memory and executable on the processor, and the program or instructions, when executed by the processor, cause the electronic device to perform:
displaying a virtual touch interface, wherein the virtual touch interface is associated with a position of a target plane in a physical space in which the electronic device is located;
recognizing hand gesture information of a user; and
performing a target operation in a case that position information, of a first region of the virtual touch interface, in the physical space and the hand gesture information meet a preset condition.

### Claim 10 (depends on 9)

The electronic device according to claim 9, wherein the program or instructions, when executed by the processor, cause the electronic device to further perform:
obtaining a first image, wherein the first image is an image of the physical space in which the electronic device is located;
calculating a first point cloud of the physical space in a first coordinate system based on the first image;
determining, from the first point cloud, a target point cloud located in a target coordinate range, wherein the target coordinate range is determined based on coordinate information of the electronic device in the first coordinate system; and
determining a target plane based on the target point cloud; wherein the first coordinate system is a coordinate system corresponding to the physical space.

### Claim 11 (depends on 10)

The electronic device according to claim 10, wherein the first image is a depth image, and the depth image comprises N pieces of first depth information corresponding to N first pixels and N pieces of third coordinate information corresponding to the N first pixels; and
the program or instructions, when executed by the processor, cause the electronic device to perform:
calculating a second point cloud of the physical space in a second coordinate system based on the N pieces of first depth information and the N pieces of third coordinate information; and
converting the second point cloud into the first point cloud; wherein
the second coordinate system is a coordinate system corresponding to an image sensor of the electronic device.

### Claim 12 (depends on 10)

The electronic device according to claim 10, wherein the program or instructions, when executed by the processor, cause the electronic device to perform:
determining at least one first plane based on the target point cloud;
determining a plane meeting a first preset condition in the at least one first plane as a second plane; and
determining a corresponding position of a point cloud of the second plane in the physical space as the target plane; wherein
the first preset condition comprises at least one of the following:
a plane whose included angle between its normal vector and a normal vector of a horizontal plane in the physical space is less than a preset value in the at least one first plane;
a plane perpendicular to a target direction in the at least one first plane; or a plane whose plane size is in a preset range in the at least one first plane.

### Claim 13 (depends on 10)

The electronic device according to claim 10, wherein the target coordinate range comprises at least one first coordinate range; and
the program or instructions, when executed by the processor, cause the electronic device to perform:
determining at least one point cloud subset in the target point cloud based on the at least one first coordinate range, wherein each point cloud subset corresponds to each first coordinate range, and a point cloud subset comprises at least one point cloud;
determining at least one first plane based on a target point cloud subset in the at least one point cloud subset, wherein the target point cloud subset is a point cloud subset whose quantity of point clouds is greater than or equal to a preset quantity in the at least one point cloud subset; and
fusing a third plane in the at least one first plane and a fourth plane to obtain the target plane, wherein the fourth plane is a plane stored in the electronic device, and spatial positions of the third plane and the fourth plane in the physical space meet a second preset condition.

### Claim 14 (depends on 9)

The electronic device according to claim 9, wherein the program or instructions, when executed by the processor, cause the electronic device to further perform:
obtaining a third image of the physical space, wherein the third image comprises an image of a user hand;
based on the third image, calculating a third point cloud of the physical space in a first coordinate system, wherein the third point cloud comprises position and posture information of the user hand;
determining a third plane in the third point cloud based on the position and posture information of the user hand; and
determining a screen corresponding to the third plane in the physical space as the target plane; wherein
the first coordinate system is a coordinate system corresponding to the physical space.

### Claim 15 (depends on 9)

The electronic device according to claim 9, wherein the program or instructions, when executed by the processor, cause the electronic device to perform:
obtaining a fourth image of the physical space, wherein the fourth image comprises an image of a user hand; and
performing user hand gesture recognition on the fourth image to obtain at least one piece of hand joint point information.

### Claim 16 (depends on 15)

The electronic device according to claim 15, wherein the at least one piece of hand joint point information comprises at least one first coordinate of at least one hand joint in a second coordinate system; and
the program or instructions, when executed by the processor, cause the electronic device to perform:
determining the first region of the virtual touch interface in a case that a spatial distance in the physical space between the at least one first coordinate and the virtual touch interface meets the preset condition; and
performing the target operation based on operation information corresponding to the first region.

### Claim 17

A non-transitory readable storage medium, wherein a program or instructions are stored in the non-transitory readable storage medium, and the program or instructions, when executed by a processor of an electronic device, cause the electronic device to perform:
displaying a virtual touch interface, wherein the virtual touch interface is associated with a position of a target plane in a physical space in which the electronic device is located;
recognizing hand gesture information of a user; and
performing a target operation in a case that position information, of a first region of the virtual touch interface, in the physical space and the hand gesture information meet a preset condition.

### Claim 18 (depends on 17)

The non-transitory readable storage medium according to claim 17, wherein the program or instructions, when executed by the processor, cause the electronic device to further perform:
obtaining a first image, wherein the first image is an image of the physical space in which the electronic device is located;
calculating a first point cloud of the physical space in a first coordinate system based on the first image;
determining, from the first point cloud, a target point cloud located in a target coordinate range, wherein the target coordinate range is determined based on coordinate information of the electronic device in the first coordinate system; and
determining a target plane based on the target point cloud; wherein
the first coordinate system is a coordinate system corresponding to the physical space.

### Claim 19 (depends on 18)

The non-transitory readable storage medium according to claim 18, wherein the first image is a depth image, and the depth image comprises N pieces of first depth information corresponding to N first pixels and N pieces of third coordinate information corresponding to the N first pixels; and
the program or instructions, when executed by the processor, cause the electronic device to perform:
calculating a second point cloud of the physical space in a second coordinate system based on the N pieces of first depth information and the N pieces of third coordinate information; and
converting the second point cloud into the first point cloud; wherein
the second coordinate system is a coordinate system corresponding to an image sensor of the electronic device.

### Claim 20 (depends on 18)

The non-transitory readable storage medium according to claim 18, wherein the program or instructions, when executed by the processor, cause the electronic device to perform:
determining at least one first plane based on the target point cloud;
determining a plane meeting a first preset condition in the at least one first plane as a second plane; and
determining a corresponding position of a point cloud of the second plane in the physical space as the target plane; wherein the first preset condition comprises at least one of the following:
a plane whose included angle between its normal vector and a normal vector of a horizontal plane in the physical space is less than a preset value in the at least one first plane;
a plane perpendicular to a target direction in the at least one first plane; or
a plane whose plane size is in a preset range in the at least one first plane.
