# US 2014/0235311 A1 · Maintaining multiple views on a shared stable virtual space

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0235311 A1
- Document ID: US20140235311A1
- Status: Published application
- Kind: A1
- Application number: 14/260,216
- Filed: 2014-04-23
- Published: 2014-08-21
- Earliest claimed: 2010-03-05
- Assignees: Sony Computer Entertainment America LLC
- Applicants: Sony Computer Entertainment America LLC
- Inventors: George Weising; Thomas Miller
- Web page: https://patentgrep.com/patent/US20140235311A1

## CPC

- A63F 13/92: Video game devices specially adapted to be hand-held while playing
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- A63F 3/02: Chess; Similar board games
- A63F 9/24: Games using electronic circuits not otherwise provided for
- A63F 13/00: Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F 13/211: using inertial sensors, e.g. accelerometers or gyroscopes
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/40: Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
- A63F 13/50: Controlling the output signals based on the game progress
- A63F 13/65: automatically by game devices or servers from real world data, e.g. measurement in live racing competition
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G09G 5/08: Cursor circuits
- H04N 23/6812: {based on additional sensors, e.g. acceleration sensors}
- A63F 2009/2457: {Display screens, e.g. monitors, video displays}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G09G 2320/0261: in the context of movement of objects on the screen or movement of the observer relative to the screen
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Methods, apparatus, and computer programs for controlling a view of a virtual scene with a handheld device are presented. In one method, images of a real world scene are captured using a device. The method further includes operations for creating an augmented view for presentation on a display of the device by augmenting the images with virtual reality objects, and for detecting a hand in the images as extending into the real world scene. In addition, the method includes operations for showing the hand in the screen as detected in the images, and for generating interaction data, based on an interaction of the hand with a virtual reality object, when the hand makes virtual contact in the augmented view with the virtual reality object. The augmented view is updated based on the interaction data, which simulates on the screen that the hand is interacting with the virtual reality object.

## Claims

### Claim 1

A method comprising:
capturing images of a real world scene using a handheld device;
creating an augmented view for presentation on a display screen of the handheld device by augmenting the captured images with one or more virtual reality objects;
detecting a user hand in the captured images as extending into the real world scene;
showing the user hand in the display screen as detected in the captured images;
generating interaction data based on an interaction of the user hand with a first virtual reality object from the one or more virtual reality objects when the user hand makes virtual contact in the augmented view with the first virtual reality object; and
updating the augmented view based on the interaction data, wherein the interaction data simulates on the display screen that the user hand is interacting with the first virtual reality object.

### Claim 2 (depends on 1)

The method as recited in claim 1, wherein the interaction of the user hand is an action on the first virtual reality object selected from a group consisting of holding, pushing, pulling, moving, hitting, throwing, opening, closing, turning on or off, pushing a button, firing a weapon, or eating the first virtual reality object.

### Claim 3 (depends on 1)

The method as recited in claim 1, wherein creating the augmented view further includes:
determining a location of the handheld device utilizing one or more of image analysis of the captured, or data obtained from inertial systems within the handheld device, or GPS, or ultrasonic triangulation, or WiFi communications, or dead reckoning.

### Claim 4 (depends on 1)

The method as recited in claim 1, wherein in the augmented view the one or more virtual reality objects change with a geometrical perspective as if the one or more virtual reality objects were in the real world scene.

### Claim 5 (depends on 1)

The method as recited in claim 1, wherein creating the augmented view further includes:
detecting a table in the real world scene; and
placing a second virtual reality object on top of the table as if the second virtual reality object was resting on the table.

### Claim 6 (depends on 1)

The method as recited in claim 1, wherein the interaction data is associated with the user hand grabbing the first virtual reality object.

### Claim 7 (depends on 6)

The method as recited in claim 6, wherein the interaction includes moving the first virtual reality object after the user hand grabs the first virtual reality object.

### Claim 8 (depends on 1)

The method as recited in claim 1, wherein the user hand is able to manipulate the first virtual reality object to change a position or a property of the first virtual reality object as if the first virtual reality object were a real object.

### Claim 9 (depends on 1)

The method as recited in claim 1, wherein the augmented view simulates that a camera in a back of the handheld device moves within the real world scene.

### Claim 10 (depends on 1)

The method as recited in claim 1, wherein the first virtual reality object is a chess piece, wherein the interaction of the user hand includes one or more of grabbing the chess piece, or moving the chess piece, or releasing the chess piece.

### Claim 11

A method comprising:
capturing images of a real world scene using a plurality of handheld devices associated with a plurality of users;
creating augmented views for presentation on respective display screens of the handheld devices by augmenting the captured images with one or more virtual reality objects;
detecting a user hand in first captured images of a first handheld device, the user hand extending into the real world scene;
showing the user hand in a first display screen of the first handheld device as detected in the first captured images;
generating interaction data based on an interaction of the user hand with a first virtual reality object from the one or more virtual reality objects when the user hand makes virtual contact with the first virtual reality object in a first augmented view of the first handheld device; and
updating the first augmented view based on the interaction data, wherein the interaction data simulates on the first display screen that the user hand is interacting with the first virtual reality object.

### Claim 12 (depends on 11)

The method as recited in claim 11, further including:
synchronizing the plurality of handheld devices to a reference point in the real world scene.

### Claim 13 (depends on 11)

The method as recited in claim 11, wherein each handheld device is operable to detect a user hand of the respective user and generate interaction data based on the interaction of the respective user with any of the one or more virtual reality objects.

### Claim 14 (depends on 11)

The method as recited in claim 11, wherein the interaction of the user hand is an action on the first virtual reality object selected from a group consisting of holding, or opening, or smashing.

### Claim 15 (depends on 11)

The method as recited in claim 11, further including:
exchanging among the plurality of handheld devices image information, relative motion, and ping data.

### Claim 16 (depends on 11)

The method as recited in claim 11, exchanging among the plurality of handheld devices image information, relative motion, and positional information to synchronize calibrated positions for the plurality of handheld devices in the augmented views.

### Claim 17 (depends on 11)

The method as recited in claim 11, wherein augmented views in respective displays from the plurality of handheld devices are independent from each other and the respective augmented views are based on a position of the respective handheld device.

### Claim 18 (depends on 11)

The method as recited in claim 11, wherein the augmented view includes a virtual board game, wherein players holding respective handheld devices play a virtual board game.

### Claim 19

A non-transitory computer-readable storage medium storing a computer program, the computer-readable storage medium comprising:
program instructions for capturing images of a real world scene using a handheld device;
program instructions for creating an augmented view for presentation on a display screen of the handheld device by augmenting the captured images with one or more virtual reality objects;
program instructions for detecting a user hand in the captured images as extending into the real world scene;
program instructions for showing the user hand in the display screen as detected in the captured images;
program instructions for generating interaction data based on an interaction of the user hand with a first virtual reality object from the one or more virtual reality objects when the user hand makes virtual contact in the augmented view with the first virtual reality object; and
program instructions for updating the augmented view based on the interaction data, wherein the interaction data simulates on the display screen that the user hand is interacting with the first virtual reality object.

### Claim 20 (depends on 19)

The storage medium as recited in claim 19, wherein creating augmented views further includes:
adding a shadow of the user hand over the one or more virtual reality objects according to lighting conditions in the real world scene.
