# US 12,555,334 B2 · Three-dimensional positioning and motion-detecting system for human body and object status in mixed reality and method thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,555,334 B2
- Document ID: US12555334B2
- Status: Granted
- Kind: B2
- Application number: 18/427,608
- Filed: 2024-01-30
- Published: 2025-05-01
- Granted: 2026-02-17
- Earliest claimed: 2024-01-30
- Assignees: Smart Surgery
- Applicants: Smart Surgery
- Inventors: Yu-Chieh Lee
- Web page: https://patentgrep.com/patent/US12555334B2

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G06T 2219/2016: Rotation, translation, scaling
- G06V 2201/07: Target detection
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A three-dimensional positioning and motion-detecting system for human body and object status in mixed reality and a method thereof are disclosed. A virtualization computing device performs gesture recognition for scaling, moving or 3D rotating a virtual posture model, to make posture model aligning points of the virtual posture model correspond to the posture model positioning points one by one, a virtual-and-real display device displays the adjusted virtual posture model in real time, when the virtual posture model is fixed, the virtualization computing device recognizes a hand position on the physical environment image, and detects a motion on the hand position based on the posture model positioning points, to generate operating position information and hand motion information. The virtual-and-real display device displays virtual-and-real integration of operating position information, hand motion information and a physical space.

## Claims

### Claim 1

A three-dimensional positioning and motion-detecting system for human body and object status in mixed reality, comprising
a camera device, configured to obtain and provide a physical environment image of a physical space;
a virtualization computing device, coupled to the camera device, wherein the virtualization computing device comprises:
a non-transitory computer-readable storage medium, configured to store computer readable instructions;
a hardware processor, electrically connected to the non-transitory computer-readable storage medium, and configured to execute the computer readable instructions to make the virtualization computing device execute:
receiving the physical environment image from the camera device;
setting posture model positioning points in the physical environment image;
generating a virtual posture model having posture model aligning points, based on a preset size;
performing gesture recognition on the physical environment image to obtain an adjustment gesture for scaling, moving or three-dimensional (3D) rotating the virtual posture model, to make the posture model aligning points of the virtual posture model correspond and align to the posture model positioning points one by one;
performing gesture recognition on the physical environment image to obtain a position-fixing gesture for fixing the virtual posture model; and
when the virtual posture model is fixed, recognizing a hand position on the physical environment image and detecting a motion on the hand position based on the posture model positioning points, to generate operating position information and hand motion information; and
a virtual-and-real display device, coupled to the virtualization computing device, and configured to display virtual-and-real integration of the posture model positioning points, the virtual posture model, and the physical space, display virtual-and-real integration of the adjusted virtual posture model in real time, and display the operating position information, the hand motion information, and the physical space.

### Claim 2 (depends on 1)

The three-dimensional positioning and motion-detecting system for human body and object status in mixed reality according to claim 1, wherein the virtualization computing device executes:
generating a virtual operation machine in the physical environment image, wherein the virtual operation machine comprises a machine identification point;
performing gesture recognition on the physical environment image to obtain a machine selection gesture for operating the virtual operation machine; and
performing position and motion recognition on the machine identification point based on the posture model positioning points, to generate the operating position information and machine motion information; and
wherein the virtual-and-real display device displays a change of the virtual operation machine during operation in real time, and displays virtual-and-real integration of the operating position information, the machine motion information, and the physical space.

### Claim 3 (depends on 1)

The three-dimensional positioning and motion-detecting system for human body and object status in mixed reality according to claim 1, wherein the physical environment image comprises an operation machine, the operation machine comprises a machine identification point and a machine sensor, the virtualization computing device is coupled to the operation machine, and the virtualization computing device executes:
obtaining a machine sensing signal from the machine sensor; and
recognizing a position of the machine identification point based on the posture model positioning points, and recognizing machine motion information based on the machine sensing signal;
wherein the virtual-and-real display device displays a change of the operation machine during operation in real time, displays virtual-and-real integration of the operating position information, the machine motion information, a physical object, and the physical space.

### Claim 4

A three-dimensional positioning and motion-detecting method for human body and object status in mixed reality, comprising:
obtaining and providing a physical environment image of a physical space to a virtualization computing device, by a camera device, wherein the virtualization computing device is coupled to the camera device;
setting posture model positioning points on the physical environment image, by the virtualization computing device;
generating a virtual posture model having posture model aligning points based on a preset size, by the virtualization computing device;
displaying the posture model positioning points, the virtual posture model and the physical space through a virtual-and-real display device, by the virtualization computing device, wherein the virtual-and-real display device is coupled to the virtualization computing device;
performing gesture recognition on the physical environment image to obtain an adjustment gesture for scaling, moving, or three-dimensional (3D) rotating the virtual posture model, to make the posture model aligning points of the virtual posture model correspond and align to the posture model positioning points one by one, by the virtualization computing device, wherein the virtual-and-real display device displays the virtual posture model in accordance with the adjustment gesture;
performing gesture recognition on the physical environment image to obtain a position-fixing gesture for fixing the virtual posture model, by the virtualization computing device;
when the virtual posture model is fixed, recognizing a hand position on the physical environment image, and detecting a motion on the hand position based on the posture model positioning points to generate operating position information and hand motion information, by the virtualization computing device; and
displaying virtual-and-real integration of the operating position information, the hand motion information and the physical space, by the virtual-and-real display device.

### Claim 5 (depends on 4)

The three-dimensional positioning and motion-detecting method for human body and object status in mixed reality according to claim 4, further comprising:
generating a virtual operation machine in the physical environment image, by the virtualization computing device, wherein the virtual operation machine comprises a machine identification point;
performing gesture recognition on the physical environment image to obtain a machine selection gesture for operating the virtual operation machine, by the virtualization computing device; and
performing position and motion recognition on the machine identification point based on the posture model positioning points, to generate the operating position information and machine motion information, by the virtualization computing device; and
displaying a change of the virtual operation machine during operation in real time, and displaying virtual-and-real integration of the operating position information, the machine motion information, and the physical space, by the virtual-and-real display device.

### Claim 6 (depends on 4)

The three-dimensional positioning and motion-detecting method for human body and object status in mixed reality according to claim 4, further comprising:
providing an operation machine in the physical environment image, wherein the operation machine comprises a machine identification point and a machine sensor;
coupling the virtualization computing device to the operation machine, and obtaining a machine sensing signal from the machine sensor;
recognizing a position of the machine identification point based on the posture model positioning points, and recognizing machine motion information based on the machine sensing signal, by the virtualization computing device;
displaying a change of the operation machine during operation in real time, and displaying virtual-and-real integration of the operating position information, the machine motion information, a physical object, and the physical space, by the virtual-and-real display device.

### Claim 7

A three-dimensional positioning and motion-detecting system for human body and object status in mixed reality, comprising:
a camera device, configured to obtain and provide a physical environment image of a physical space, wherein the physical environment image comprises a physical object having posture model positioning points, and the physical object comprises a sensor disposed correspondingly to a particular position;
a virtualization computing device, coupled to the camera device and the physical object, wherein the virtualization computing device comprises:
a non-transitory computer-readable storage medium, configured to store computer readable instructions; and
a hardware processor, electrically connected to the non-transitory computer-readable storage medium, and configured to execute the computer readable instructions to make the virtualization computing device execute:
receiving the physical environment image from the camera device;
generating a virtual posture model having posture model aligning points based on a preset size;
performing gesture recognition on the physical environment image to obtain an adjustment gesture for scaling, moving or three-dimensional (3D) rotating the virtual posture model to make the posture model aligning points of the virtual posture model correspond to the posture model positioning points of the physical object one by one;
performing gesture recognition on the physical environment image to obtain a position-fixing gesture for fixing the virtual posture model at a position of the physical object; and
when the virtual posture model is fixed, obtaining a sensing signal of the sensor from the physical object, to recognize operating position information and hand motion information based on the sensing signal; and
a virtual-and-real display device, coupled to the virtualization computing device, and configured to display virtual-and-real integration of the virtual posture model, the physical object, and the physical space, display the adjusted virtual posture model in real time, and display virtual-and-real integration of the operating position information, the hand motion information, the physical object, and the physical space.

### Claim 8 (depends on 7)

The three-dimensional positioning and motion-detecting system for human body and object status in mixed reality according to claim 7, wherein the physical environment image comprises an operation machine, the operation machine comprises a machine identification point and a machine sensor, the virtualization computing device is coupled to the operation machine, and the virtualization computing device executes:
obtaining a machine sensing signal from the machine sensor; and
recognizing a position of the machine identification point based on the posture model positioning points, and recognizing machine motion information based on the machine sensing signal;
wherein the virtual-and-real display device displays a change of the operation machine during operation in real time, displays virtual-and-real integration of the operating position information, the machine motion information, the physical object, and the physical space.

### Claim 9 (depends on 7)

The three-dimensional positioning and motion-detecting system for human body and object status in mixed reality according to claim 7, wherein the virtualization computing device executes:
generating a virtual operation machine in the physical environment image, wherein the virtual operation machine comprises a machine identification point;
performing gesture recognition on the physical environment image to obtain a machine selection gesture for operating the virtual operation machine; and
performing position and motion recognition on the machine identification point based on the posture model positioning points, to generate the operating position information and machine motion information; and
wherein the virtual-and-real display device displays a change of the virtual operation machine during operation in real time, and displays virtual-and-real integration of the operating position information, the machine motion information, and the physical space.