# US 2011/0298824 A1 · System and method of virtual interaction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2011/0298824 A1
- Document ID: US20110298824A1
- Status: Published application
- Kind: A1
- Application number: 12/979,636
- Filed: 2010-12-28
- Published: 2011-12-08
- Earliest claimed: 2009-12-31
- Assignees: Sony Computer Entertainment Europe Limited
- Inventors: Richard Lee; Sam Coates; Paul Kevan Grenfell
- Web page: https://patentgrep.com/patent/US20110298824A1

## CPC

- A63F 13/31: Communication aspects specific to video games, e.g. between several handheld game devices at close range
- A63F 13/211: using inertial sensors, e.g. accelerometers or gyroscopes
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/327: using wireless networks, e.g. Wi-Fi® or piconet
- A63F 13/92: Video game devices specially adapted to be hand-held while playing
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- A63F 13/2145: the surface being also a display device, e.g. touch screens
- A63F 13/812: Ball games, e.g. soccer or baseball
- A63F 2300/1075: using a touch screen
- A63F 2300/204: the platform being a handheld device
- A63F 2300/405: being a wireless ad hoc network, e.g. Bluetooth®, Wi-Fi®, piconet
- A63F 2300/64: for computing dynamical parameters of game objects, e.g. motion determination or computation of frictional forces for a virtual car
- A63F 2300/8082: Virtual reality
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system for virtual interaction, comprising two or more portable electronic devices, is provided. Each device comprises, in turn, coordinate referencing means operable to define a coordinate system common to the portable electronic devices with respect to a physically defined reference position, position estimation means operable to detect a the physical position of its respective portable electronic device with respect to the reference position, virtual environment generation means operable to generate a virtual environment, and communication means operable to transmit positional data using the common coordinate system from that portable electronic device to another portable electronic device. The virtual environment is shared in common between the portable electronic devices. The virtual environment uses the common co-ordinate system within which each portable electronic device defines a position for itself responsive to its physical position with respect to the reference position.

## Claims

### Claim 1

A system for virtual interaction, comprising
two or more portable electronic devices, each device comprising in turn:
a coordinate referencer operable to define a coordinate system common to said portable electronic devices with respect to a physically defined reference position;
a position estimator operable to detect a physical position of its respective portable electronic device with respect to said reference position;
a virtual environment generator operable to generate a virtual environment; and
a communicator operable to transmit positional data using said common coordinate system from that portable electronic device to another portable electronic device; and in which
said virtual environment is shared in common between said portable electronic devices, said virtual environment using said common co-ordinate system within which each portable electronic device defines a position for itself responsive to its physical position with respect to said reference position.

### Claim 2 (depends on 1)

A system for virtual interaction according to claim 1, in which the position estimator of at least one of said portable electronic devices comprises:
one or more accelerometers; and
a motion tracker operable to estimate a physical position with respect to said reference position from cumulative changes in physical position detected using said one or more accelerometers.

### Claim 3 (depends on 1)

A system for virtual interaction according to claim 1, in which said position estimator of at least one of said portable electronic devices comprises:
a video camera operable to capture a sequence of video images; and
an optical flow estimator operable to estimate a change in physical position of a respective portable electronic device as a motion that would account for apparent motion in said sequence of video images captured by the video camera, and to estimate a physical position with respect to said reference position from cumulative estimated changes in physical position.

### Claim 4 (depends on 1)

A system for virtual interaction according to claim 1, in which the position estimator of at least one of said portable electronic devices comprises:
a video camera operable to capture a sequence of video images; and
an image processor operable to estimate a physical position of the portable electronic device based upon distortion characteristics of a captured video image of a reference object at said reference position.

### Claim 5 (depends on 1)

A system for virtual interaction according to claim 1, in which the position estimator of at least one of the portable electronic devices comprises:
a wireless identification signal transmitter; and
a wireless signal triangulator; and in which
the at least one of said portable electronic devices triangulates its relative physical position with respect to another one of said portable electronic devices.

### Claim 6 (depends on 1)

A system for virtual interaction according to claim 1, in which at least one of said portable electronic devices comprises two or more position estimators; and in which the at least one of said portable electronic devices is operable to use a second position estimator to adjust the operation of a first position estimator.

### Claim 7 (depends on 1)

A system for virtual interaction according to claim 1, in which at least one of said portable electronic device comprises:
a video camera operable to capture a video image; and in which
the at least one said portable electronic device is operable to detect an error in the correspondence between the real relative position of another portable electronic device and the relative position of the other portable electronic device within said common coordinate system, by analysing a captured video image to detect whether said other portable electronic device lies within a region of said captured video image predicted to encompass to the position of the other portable electronic device based upon its position within said common coordinate system.

### Claim 8 (depends on 1)

A system for virtual interaction according to claim 1, in which at least one of said portable electronic devices comprises:
a video camera operable to capture a video image; and
a display screen; and in which
the at least one said portable electronic device is operable to display a captured video image of the physical environment on said display screen in conjunction with one or more virtual objects included in said shared virtual environment.

### Claim 9 (depends on 1)

A system for virtual interaction according to claim 1, in which control of at least one virtual object within said shared virtual environment is shared between said two or more portable electronic devices.

### Claim 10 (depends on 1)

A system for virtual interaction according to claim 1, in which boundaries of said shared virtual environment are defined by moving one or more of said portable electronic devices to one or more boundaries of a physical environment and noting the corresponding position of the or each portable electronic device within said shared virtual environment.

### Claim 11

A method of virtual interaction between two or more portable electronic devices comprises the steps of:
defining a coordinate system common to said portable electronic devices with respect to a physically defined reference position;
estimating a physical position of at least one portable electronic device with respect to said reference position;
generating a virtual environment;
communicating positional data between said portable electronic devices using said common coordinate system; and in which
said virtual environment is shared in common between said portable electronic devices, the virtual environment using the common co-ordinate system;
and for each portable electronic device,
defining its position in said shared virtual environment responsive to its physical position with respect to said reference position.

### Claim 12 (depends on 11)

A method of virtual interaction between two or more portable electronic devices according to claim 11, in which the step of estimating the physical position of the at least one respective portable electronic device comprises one or more selected from the list consisting of:
i. estimating the physical position with respect to said reference position from cumulative changes in physical position calculated from detected acceleration of said respective portable electronic device;
ii. estimating a physical motion that accounts for apparent motion in a sequence of video images captured by a video camera of said respective portable electronic device, and estimating the physical position with respect to said reference position from cumulative estimated motion;
iii. estimating the physical position with respect to said reference position based upon distortion characteristics of a captured video image of a reference object at said reference position, captured by a video camera of said respective portable electronic device; and
iv. estimating a relative physical position with respect to another portable electronic device based upon triangulated wireless identification signals.

### Claim 13 (depends on 11)

A method of virtual interaction between two or more portable electronic devices according to claim 11, comprising the step of:
passing control of at least one virtual object within said shared virtual environment between said two or more portable electronic devices, so as to allocate control of said virtual object to a portable electronic device which most recently interacted with said virtual object.

### Claim 14 (depends on 11)

A method of virtual interaction between two or more portable electronic devices according to claim 11, comprising the step of:
defining boundaries of said shared virtual environment by moving one or more of said portable electronic devices to one or more boundaries of a physical environment and noting the corresponding position of the or each portable electronic device within said shared virtual environment.

### Claim 15

A tangible computer-readable storage medium on which computer readable instructions of a computer program are stored, the instructions, when executed by a computer, cause the computer to perform a method of virtual interaction between two or more portable electronic devices comprising:
defining a coordinate system common to said portable electronic devices with respect to a physically defined reference position;
estimating a physical position of at least one portable electronic device with respect to said reference position;
generating a virtual environment;
communicating positional data between said portable electronic devices using said common coordinate system; and in which
said virtual environment is shared in common between said portable electronic devices, the virtual environment using the common co-ordinate system;
and for each portable electronic device,
defining its position in said shared virtual environment responsive to its physical position with respect to said reference position.
