# US 12,005,344 B2 · Motion mirroring system that incorporates virtual environment constraints

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,005,344 B2
- Document ID: US12005344B2
- Status: Granted
- Kind: B2
- Application number: 17/878,835
- Filed: 2022-08-01
- Published: 2022-11-24
- Granted: 2024-06-11
- Earliest claimed: 2017-05-23
- Assignees: Blast Motion Inc.
- Applicants: Blast Motion Inc.
- Inventors: Juergen Haas; Bhaskar Bose; Michael Bentley; Ryan Kaps
- Web page: https://patentgrep.com/patent/US12005344B2

## CPC

- A63F 13/211: using inertial sensors, e.g. accelerometers or gyroscopes
- A63F 13/235: using a wireless connection, e.g. infrared or piconet
- A63F 13/25: Output arrangements for video game devices
- A63F 13/42: by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
- A63F 13/525: Changing parameters of virtual cameras
- A63F 13/833: Hand-to-hand fighting, e.g. martial arts competition
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- A63F 13/812: Ball games, e.g. soccer or baseball
- A63F 2300/105: using inertial sensors, e.g. accelerometers, gyroscopes
- A63F 2300/8082: Virtual reality
- G06T 15/20: Perspective computation
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A system that mirrors motion of a physical object by displaying a virtual object moving in a virtual environment. The mirroring display may be used for example for feedback, coaching, or for playing virtual games. Motion of the physical object is measured by motion sensors that may for example include an accelerometer, a gyroscope, and a magnetometer. Sensor data is transmitted to a computer that calculates the position and orientation of the physical object and generates a corresponding position and orientation of the virtual object. The computer may correct or adjust the calculations using sensor data redundancies. The virtual environment may include constraints on the position, orientation, or motion of the virtual object. These constraints may be used to compensate for accumulating errors in position and orientation. The system may for example use proportional error feedback to adjust position and orientation based on sensor redundancies and virtual environment constraints.

## Claims

### Claim 1

A motion mirroring system that incorporates virtual environment constraints,
comprising
a motion capture element configured to couple with a moveable object located in a real environment, wherein said motion capture element comprises
a sensor configured to capture one or more values associated with an orientation, position, velocity, acceleration, angular velocity, and angular acceleration of said motion capture element;
a first communication interface;
a microprocessor coupled with said sensor and said first communication interface,
wherein said microprocessor is configured to
collect data that comprises said one or more values from said sensor;
transmit said data via said first communication interface;
a computer that comprises
a display;
a memory that contains
a virtual environment state comprising a position and orientation of a virtual object in a virtual environment, wherein said virtual object represents said moveable object in said virtual environment;
one or more constraints on said position and orientation of said virtual object in said virtual environment;
a second communication interface configured to communicate with said first communication interface to obtain said data;
wherein said computer is configured to receive said data via said second communication interface;
calculate a position and orientation of said moveable object in said real environment from said data;
apply one or more corrections to said position and orientation of said moveable object in said real environment based on one or more redundancies in said data;
transform said position and orientation of said moveable object in said real environment into said position and orientation of said virtual object in said virtual environment;
determine whether said position and orientation of said virtual object in said virtual environment satisfies said one or more constraints;
when said position and orientation of said virtual object in said virtual environment does not satisfy one or more of said one or more constraints, modify said position and orientation of said virtual object in said virtual environment to satisfy said one or more constraints;
display said virtual environment and said virtual object on said display.

### Claim 2 (depends on 1)

The system of claim 1, wherein said moveable object is one or more of a piece of equipment, an article of clothing, or a body part of a person.

### Claim 3 (depends on 1)

The system of claim 1, wherein said sensor comprises a 3-axis accelerometer, a 3-axis rate gyroscope, and a 3-axis magnetometer.

### Claim 4 (depends on 3)

The system of claim 3, wherein said redundancies in said data comprise
a calculation of said orientation of said moveable object in said real environment by integration of angular velocity data from said 3-axis rate gyroscope;
a measurement of said orientation of said moveable object in said real environment relative to an Earth magnetic field from said 3-axis magnetometer;
a measurement of said orientation of said moveable object in said real environment relative to an Earth gravitational field from said 3-axis accelerometer, when said data indicates that said moveable object is substantially stationary.

### Claim 5 (depends on 4)

The system of claim 4, wherein
said calculate a position and orientation of said moveable object in said real environment from said data comprises said integration of angular velocity data from said 3-axis rate gyroscope;
said apply one or more corrections to said position and orientation of said moveable object in said real environment comprises one or more of
calculate a first rotational error between
a predicted magnetic field vector based on said integration of angular velocity data, and
a measured magnetic field vector from said 3-axis magnetometer;
apply a first fraction of said first rotational error to said orientation of said moveable object in said real environment;
calculate a second rotational error between
a predicted gravitational field vector based on said integration of angular velocity data, and
a measured gravitational field vector from said 3-axis accelerometer;
apply a second fraction of said second rotational error to said orientation of said moveable object in said real environment.

### Claim 6 (depends on 1)

The system of claim 1, wherein
said virtual environment comprises a virtual camera that forms an image of said virtual environment;
said display said virtual environment and said virtual object on said display transmits said image of said virtual environment to said display;
a position and orientation of said virtual camera in said virtual environment is modifiable by a user.

### Claim 7 (depends on 1)

The system of claim 1, wherein a change in said position and orientation of said moveable object in said real environment results in a corresponding change within a predefined time period of said position and orientation of said virtual object in said virtual environment shown on said display.

### Claim 8 (depends on 1)

The system of claim 1, wherein said one or more constraints comprise a region in said virtual environment that said virtual object must be located within.

### Claim 9 (depends on 1)

The system of claim 1, wherein said one or more constraints comprise a location in said virtual environment that said virtual object must be located near.

### Claim 10 (depends on 9)

The system of claim 9, wherein said modify said position and orientation of said virtual object in said virtual environment to satisfy said one or more constraints comprises calculate an error vector between said position of said virtual object in said virtual environment and said location;
add a fraction of said error vector to said position of said virtual object in said virtual environment.

### Claim 11 (depends on 1)

The system of claim 1, wherein said one or more constraints comprise one or more of a maximum speed, a minimum speed, a maximum acceleration, a minimum acceleration, a maximum angular velocity, a minimum angular velocity, a maximum angular acceleration, and a minimum angular acceleration of said virtual object.

### Claim 12 (depends on 1)

The system of claim 1, wherein said one or more constraints comprise one or more virtual barriers that said virtual object cannot pass through.

### Claim 13 (depends on 12)

The system of claim 12, wherein said one or more virtual barriers comprise a ground surface.

### Claim 14 (depends on 1)

The system of claim 1, wherein
said moveable object is a piece of sporting equipment used in a sports activity;
said virtual environment is a virtual game for said sports activity;
said virtual object in said virtual environment is a representation of said piece of sporting equipment.

### Claim 15 (depends on 14)

The system of claim 14, wherein said sports activity is one or more of golf, tennis, badminton, racquetball, table tennis, squash, baseball, softball, cricket, hockey, field hockey, croquet, football, rugby, Australian rules football, soccer, volleyball, water polo, polo, basketball, lacrosse, billiards, horseshoes, shuffleboard, handball, bocce, bowling, dodgeball, kick ball, track and field events, curling, martial arts, boxing, archery, pistol shooting, rifle shooting, ice skating, gymnastics, surfing, skateboarding, snowboarding, skiing, windsurfing, roller blading, bicycling, and racing.

### Claim 16 (depends on 14)

The system of claim 14, wherein
said virtual object is used to contact and launch a virtual game piece in said virtual game;
said computer calculates a launch velocity of said game piece based on a motion of said virtual object at a time of impact between said virtual object and said virtual game piece.

### Claim 17 (depends on 14)

The system of claim 14, wherein said computer relocates said virtual object to a new position in said virtual environment in response to one or more events in said virtual game.

### Claim 18 (depends on 17)

The system of claim 17, wherein said relocates said virtual object to a new position occurs gradually to prevent a user from observing a discontinuous change in position on said display.

### Claim 19 (depends on 17)

The system of claim 17, wherein
said virtual game is a virtual golf game;
said virtual object is a virtual golf club;
said relocates said virtual object to a new position comprises moves said virtual golf club to a location of a virtual golf ball after a shot, or to a starting position for a new hole after completion of a previous hole.

### Claim 20 (depends on 19)

The system of claim 19, wherein said moves said virtual golf club further comprises modifies an orientation of said virtual golf club to aim at a hole from said new position.