# US 2012/0075430 A1 · Computer-readable storage medium, information processing apparatus, information processing system, and information processing method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0075430 A1
- Document ID: US20120075430A1
- Status: Published application
- Kind: A1
- Application number: 13/243,622
- Filed: 2011-09-23
- Published: 2012-03-29
- Earliest claimed: 2010-09-27
- Assignees: HAL Laboratory Inc.; Nintendo Co., Ltd.
- Inventors: Yuichiro ITO; Tetsuya Noge
- Web page: https://patentgrep.com/patent/US20120075430A1

## CPC

- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/2145: the surface being also a display device, e.g. touch screens
- A63F 13/25: Output arrangements for video game devices
- A63F 13/26: having at least one additional display device, e.g. on the game controller or outside a game booth
- A63F 13/426: involving on-screen location information, e.g. screen coordinates of an area at which the player is aiming with a light gun
- A63F 13/525: Changing parameters of virtual cameras
- A63F 13/5258: by dynamically adapting the position of the virtual camera to keep a game object or game character in its viewing frustum, e.g. for tracking a character or a ball
- A63F 13/65: automatically by game devices or servers from real world data, e.g. measurement in live racing competition
- A63F 13/818: Fishing
- A63F 13/92: Video game devices specially adapted to be hand-held while playing
- G06T 7/73: using feature-based methods
- H04N 13/156: Mixing image signals
- H04N 13/275: from three-dimensional [3D] object models, e.g. computer-generated stereoscopic image signals
- A63F 2300/1075: using a touch screen
- A63F 2300/1087: comprising photodetecting means, e.g. a camera
- A63F 2300/1093: using visible light
- A63F 2300/204: the platform being a handheld device
- A63F 2300/301: using an additional display connected to the game console, e.g. on the controller
- A63F 2300/6045: for mapping control signals received from the input arrangement into game commands
- A63F 2300/66: for rendering three dimensional images
- A63F 2300/6684: by dynamically adapting its position to keep a game object in its viewing frustrum, e.g. for tracking a character or a ball
- A63F 2300/6692: using special effects, generally involving post-processing, e.g. blooming
- A63F 2300/69: Involving elements of the real world in the game world, e.g. measurement in live races, real video
- A63F 2300/8029: Fighting without shooting
- A63F 2300/8035: Virtual fishing
- G06T 2207/10012: Stereo images
- G06T 2207/30204: Marker
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A marker present in a real space is detected from each of a real image for a left eye and a real image for a right eye. The position and the orientation of each of imaging sections relative to the marker are calculated. A left virtual camera and a right virtual camera are set in accordance with the positions and the orientations of the imaging sections. A virtual model having a planar shape is placed in a marker coordinate system, and areas in the real image for a left eye and the real image for a right eye that correspond to the virtual model are set as a left texture area and a right texture area, respectively. Each of images included in a left texture area and a right texture area is applied, as a texture, to a deformation object into which the virtual model is deformed.

## Claims

### Claim 1

A computer-readable storage medium having stored therein an information processing program executed by a computer of an information processing apparatus connected to a left real camera and a right real camera for shooting a real space, and to display means capable of displaying a stereoscopic visible image, the information processing program causing the computer to function as:
real image obtaining means for obtaining a real image for a left eye shot by the left real camera, and a real image for a right eye shot by the right real camera;
virtual camera setting means for setting a left virtual camera and a right virtual camera in a virtual space such that the left virtual camera and the right virtual camera are placed in accordance with a position and an orientation of the left real camera in the real space, and a position and an orientation of the right real camera in the real space;
left texture area setting means for setting, as a left texture area, an area in the real image for a left eye that corresponds to a predetermined virtual model in the virtual space when the area is looked at from the left virtual camera set by the virtual camera setting means;
right texture area setting means for setting, as a right texture area, an area in the real image for a right eye that corresponds to a predetermined virtual model in the virtual space when the area is looked at from the right virtual camera set by the virtual camera setting means;
left virtual camera image obtaining means for applying an image included in the left texture area set by the left texture area setting means, to a drawing model that is the same as the virtual model or has a predetermined relationship with the virtual camera, and for obtaining a left virtual camera image of the drawing model shot by the left virtual camera;
right virtual camera image obtaining means for applying an image included in the right texture area set by the right texture area setting means, to the drawing model, and obtaining a right virtual camera image of the drawing model shot by the right virtual camera; and
display control means for displaying the left virtual camera image and the right virtual camera image on the display means such that the left virtual camera image is viewed by the left eye of a user and the right virtual camera image is viewed by the right eye of the user.

### Claim 2 (depends on 1)

The computer-readable storage medium according to claim 1, wherein
the left texture area setting means sets, as a left texture area, an area corresponding to the position and the contour of the virtual model in a rendered image obtained by shooting the virtual model by the left virtual camera, and
the right texture area setting means sets, as a right texture area, an area corresponding to the position and the contour of the virtual model in a rendered image obtained by shooting the virtual model by the right virtual camera.

### Claim 3 (depends on 1)

The computer-readable storage medium according to claim 1, the information processing program further causing the computer to function as superimposed image generation means for generating a superimposed image for a left eye obtained by superimposing the left virtual camera image onto the real image for a left eye, and a superimposed image for a right eye obtained by superimposing the right virtual camera image onto the real image for a right eye,
wherein, the display control means displays the superimposed image for a left eye and the superimposed image for a right eye on the display means.

### Claim 4 (depends on 3)

The computer-readable storage medium according to claim 3, the information processing program further causing the computer to function as deformation object setting means for generating a deformation object into which the virtual model is deformed, and setting the deformation object in the virtual space, wherein
the left virtual camera image obtaining means applies an image included in the left texture area set by the left texture area setting means, to the deformation object, and obtains the left virtual camera image of the deformation object shot by the left virtual camera, and
the right virtual camera image obtaining means applies an image included in the right texture area set by the right texture area setting means, to the deformation object, and obtains the right virtual camera image of the deformation object shot by the right virtual camera.

### Claim 5 (depends on 4)

The computer-readable storage medium according to claim 4, wherein
the virtual model has a planar shape or a curved-surface shape, and is set in the virtual space so as to follow the shape of a plane or a curved surface in the real space, and
the deformation object setting means generates the deformation object into which the virtual model having a planar shape or a curved-surface shape, and sets the deformation object in the virtual space.

### Claim 6 (depends on 4)

The computer-readable storage medium according to claim 4, wherein the deformation object setting means generates the deformation object into which the virtual model is deformed without its end portion being deformed.

### Claim 7 (depends on 4)

The computer-readable storage medium according to claim 4, the information processing program further causing the computer to function as second object setting means for setting a second object different from the deformation object in the virtual space such that the second object has a predetermined position relationship with the deformation object, wherein
the left virtual camera image obtaining means obtains a left virtual camera image including the deformation object and the second object shot by the left virtual camera, and
the right virtual camera image obtaining means obtains a right virtual camera image of the deformation object and the second object shot by the right virtual camera.

### Claim 8 (depends on 7)

The computer-readable storage medium according to claim 7,
wherein the second object is placed so as to be away from the deformation object in the shooting direction of each of the left virtual camera and the right virtual camera,
the information processing program further causing the computer to function as mask object setting means for setting a predetermined mask object for masking the second object, at an end portion of the deformation object.

### Claim 9 (depends on 3)

The computer-readable storage medium according to claim 3, wherein
each of the left real camera and the right real camera shoots a marker, having a predetermined planar shape, that is placed in the real space,
the virtual camera setting means sets the left virtual camera and the right virtual camera in the virtual space such that the left virtual camera and the right virtual camera are placed in accordance with the position and the orientation of the left real camera in the real space relative to the position and the orientation of the marker, and the position and the orientation of the right real camera in the real space relative to the position and the orientation of the marker,
the virtual model is a model, having a planar shape, that is placed on a plane on which the marker is placed, in the marker coordinate system set based on the marker,
the information processing program further causes the computer to function as deformation object setting means for generating the deformation object into which the virtual model is deformed, and setting the deformation object in the virtual space,
the left virtual camera image obtaining means applies, as a texture, an image included in the left texture area set by the left texture area setting means, to the deformation object, and obtains a left virtual camera image of the deformation object shot by the left virtual camera, and
the right virtual camera image obtaining means applies, as a texture, an image included in the right texture area set by the right texture area setting means, to the deformation object, and obtains a right virtual camera image of the deformation object shot by the right virtual camera.

### Claim 10 (depends on 1)

The computer-readable storage medium according to claim 1, wherein
the virtual camera setting means
detects a specific subject included in the real image for a left eye, obtain relative positions and relative orientations of the left real camera and the specific subject present in the real space with respect to each other, based on a result of the detection, and set the left virtual camera, based on the relative positions and the relative orientations, and
detects the specific subject included in the real image for a right eye, obtain relative positions and relative orientations of the right real camera and the specific subject present in the real space with respect to each other, based on a result of the detection, and set the right virtual camera, based on the relative positions and the relative orientations.

### Claim 11 (depends on 10)

The computer-readable storage medium according to claim 10, wherein the virtual model is placed based on a position, in the virtual space, that corresponds to the position of the specific subject in the real space.

### Claim 12

An information processing apparatus connected to a left real camera and a right real camera for shooting a real space, and to display means capable of displaying a stereoscopic visible image, the information processing apparatus comprising:
real image obtaining means for obtaining a real image for a left eye shot by the left real camera, and a real image for a right eye shot by the right real camera;
virtual camera setting means for setting a left virtual camera and a right virtual camera in a virtual space such that the left virtual camera and the right virtual camera are placed in accordance with a position and an orientation of the left real camera in the real space, and a position and an orientation of the right real camera in the real space;
left texture area setting means for setting, as a left texture area, an area in the real image for a left eye that corresponds to a predetermined virtual model in the virtual space when the area is looked at from the left virtual camera set by the virtual camera setting means;
right texture area setting means for setting, as a right texture area, an area in the real image for a right eye that corresponds to a predetermined virtual model in the virtual space when the area is looked at from the right virtual camera set by the virtual camera setting means;
left virtual camera image obtaining means for applying an image included in the left texture area set by the left texture area setting means, to a drawing model that is the same as the virtual model or has a predetermined relationship with the virtual camera, and for obtaining a left virtual camera image of the drawing model shot by the left virtual camera;
right virtual camera image obtaining means for applying an image included in the right texture area set by the right texture area setting means, to the drawing model, and obtaining a right virtual camera image of the drawing model shot by the right virtual camera; and
display control means for displaying the left virtual camera image and the right virtual camera image on the display means such that the left virtual camera image is viewed by the left eye of a user and the right virtual camera image is viewed by the right eye of the user.

### Claim 13

An information processing system comprising:
a left real camera and a right real camera for shooting a real space;
display means capable of displaying a stereoscopic visible image;
real image obtaining means for obtaining a real image for a left eye shot by the left real camera, and a real image for a right eye shot by the right real camera;
virtual camera setting means for setting a left virtual camera and a right virtual camera in a virtual space such that the left virtual camera and the right virtual camera are placed in accordance with a position and an orientation of the left real camera in the real space, and a position and an orientation of the right real camera in the real space;
left texture area setting means for setting, as a left texture area, an area in the real image for a left eye that corresponds to a predetermined virtual model in the virtual space when the area is looked at from the left virtual camera set by the virtual camera setting means;
right texture area setting means for setting, as a right texture area, an area in the real image for a right eye that corresponds to a predetermined virtual model in the virtual space when the area is looked at from the right virtual camera set by the virtual camera setting means;
left virtual camera image obtaining means for applying an image included in the left texture area set by the left texture area setting means, to a drawing model that is the same as the virtual model or has a predetermined relationship with the virtual camera, and for obtaining a left virtual camera image of the drawing model shot by the left virtual camera;
right virtual camera image obtaining means for applying an image included in the right texture area set by the right texture area setting means, to the drawing model, and obtaining a right virtual camera image of the drawing model shot by the right virtual camera; and
display control means for displaying the left virtual camera image and the right virtual camera image on the display means such that the left virtual camera image is viewed by the left eye of a user and the right virtual camera image is viewed by the right eye of the user.

### Claim 14

An information processing method comprising:
a real image obtaining step of obtaining a real image for a left eye shot by a left real camera for shooting a real space, and a real image for a right eye shot by a right real camera for shooting a real space;
a virtual camera setting step of setting a left virtual camera and a right virtual camera in a virtual space such that the left virtual camera and the right virtual camera are placed in accordance with a position and an orientation of the left real camera in the real space, and a position and an orientation of the right real camera in the real space;
a left texture area setting step of setting, as a left texture area, an area in the real image for a left eye that corresponds to a predetermined virtual model in the virtual space when the area is looked at from the left virtual camera set in the virtual camera setting step;
a right texture area setting step of setting, as a right texture area, an area in the real image for a right eye that corresponds to a predetermined virtual model in the virtual space when the area is looked at from the right virtual camera set in the virtual camera setting step;
a left virtual camera image obtaining step of applying an image included in the left texture area set in the left texture area setting step, to a drawing model that is the same as the virtual model or has a predetermined relationship with the virtual camera, and for obtaining a left virtual camera image of the drawing model shot by the left virtual camera;
a right virtual camera image obtaining step of applying an image included in the right texture area set in the right texture area setting step, to the drawing model, and obtaining a right virtual camera image of the drawing model shot by the right virtual camera; and
a display control step of displaying the left virtual camera image and the right virtual camera image, on display means capable of displaying a stereoscopic visible image, such that the left virtual camera image is viewed by the left eye of a user and the right virtual camera image is viewed by the right eye of the user.
