# US 2020/0402308 A1 · Image generation apparatus and image generation method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0402308 A1
- Document ID: US20200402308A1
- Status: Published application
- Kind: A1
- Application number: 16/770,420
- Filed: 2017-12-19
- Published: 2020-12-24
- Earliest claimed: 2017-12-19
- Assignees: Sony Interactive Entertainment Inc.
- Applicants: Sony Interactive Entertainment Inc.
- Inventors: Masakazu SUZUOKI; Yuki KARASAWA
- Web page: https://patentgrep.com/patent/US20200402308A1

## CPC

- G06T 15/20: Perspective computation
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 7/50: Depth or shape recovery
- G06T 15/04: Texture mapping
- G06T 15/503: {Blending, e.g. for anti-aliasing}
- G06T 15/506: {Illumination models}
- H04N 9/3179: {Video signal processing therefor}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06T 2215/12: Shadow map, environment map
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

For a space including an object to be displayed, images of the space viewed from reference points of view are created in advance as reference images, and the reference images are combined according to a position of an actual point of view to draw a display image. In this case, a reference image not displaying reflection is used to determine the color of the object (S50). In a case of expressing reflection of another object (Y in S52), a position of the reflected object is estimated in a three-dimensional space (S54), a position on the reference image corresponding to the position is acquired (S56), and a color of the position is combined with the color of the object (S60).

## Claims

### Claim 1

An image generation apparatus comprising:
a reference image data storage unit that stores data of a reference image representing an image of a space viewed from a predetermined reference point of view, the space including an object to be displayed;
a point-of-view information acquisition unit that acquires information regarding a point of view;
a projection unit that displays, on a plane of a display image, a position and a shape of an image of the object obtained by viewing the space from the point of view;
a pixel value determination unit that determines, for each pixel in the display image, a color of the image of the object by using a color of an image of the same object in the reference image, specifies a position of an image of another object reflected on the object in the reference image, acquires a color at the position, and combines the colors to thereby determine a final value of the pixel representing the reflection; and
an output unit that outputs data of the display image.

### Claim 2 (depends on 1)

The image generation apparatus according to claim 1, wherein
the pixel value determination unit estimates a point on the other object, which is a point where a vector in a regular reflection direction of a gaze vector defined by the point of view reaches in the space, by using information regarding the space acquired in generating the reference image and specifies a position of the image at the point in the reference image based on the point.

### Claim 3 (depends on 2)

The image generation apparatus according to claim 2, wherein
the reference image data storage unit stores, as the data of the reference image, data of an image representing a space in all directions around the reference point of view, and
the pixel value determination unit specifies, as the position of the image of the point, a position on the reference image corresponding to a direction parallel to the vector in the regular reflection direction.

### Claim 4 (depends on 2)

The image generation apparatus according to claim 2, wherein
the reference image data storage unit stores, as the data of the reference image, data of an image representing a space in all directions around the reference point of view, and
the pixel value determination unit rotates a vector parallel to the vector in the regular reflection direction about the reference point of view and specifies a position on the object, which is a position where the vector reaches, based on a depth image corresponding to the reference image to thereby search the point.

### Claim 5 (depends on 2)

The image generation apparatus according to claim 2, wherein
the pixel value determination unit obtains, based on a positional relationship between the reference point of view and the point of view, a weighted average of the position of the other object, which is reflected on the object as viewed from the reference point of view, acquired as the information regarding the space to thereby estimate the point.

### Claim 6 (depends on 5)

The image generation apparatus according to claim 5, wherein
when a difference between a position of the estimated point and a position of the object indicated in the depth image corresponding to the reference image is equal to or greater than a predetermined threshold, the pixel value determination unit uses a weighted average of the position of the object indicated in the depth image to estimate the point again.

### Claim 7 (depends on 1)

The image generation apparatus according to claim 1, wherein the pixel value determination unit applies a predetermined filter to pixels of a region in a predetermined range around the position of the image of the other object in the reference image to thereby determine the color to be combined.

### Claim 8 (depends on 1)

The image generation apparatus according to claim 1, wherein the reference image data storage unit stores, for each reference point of view, data of a plurality of the reference images with different resolutions, and the pixel value determination unit switches the resolution at a location of acquiring the color to be combined according to a state of a surface of the object for which the value of the pixel is to be determined.

### Claim 9

An image generation method of generating a display image representing a space including an object to be displayed, the image generation method comprising:
acquiring information regarding a point of view;
displaying, on a plane of a display image, a position and a shape of an image of the object obtained by viewing the space from the point of view;
reading, from a memory storing data of a reference image representing an image of the space viewed from a predetermined reference point of view, the data of the reference image, determining, for each pixel in the display image, a color of the image of the object by using a color of an image of the same object in the reference image, specifying a position of an image of another object reflected on the object in the reference image, acquiring a color at the position, and combining the colors to thereby determine a final value of the pixel representing the reflection; and
outputting data of the display image.

### Claim 10

A computer program for a computer that generates a display image representing a space including an object to be displayed to realize, the computer program comprising:
by a point-of-view information acquisition unit, acquiring information regarding a point of view;
by a projection unit, displaying, on a plane of a display image, a position and a shape of an image of the object obtained by viewing the space from the point of view; and
by a pixel value determination unit, reading, from a memory storing data of a reference image representing an image of the space viewed from a predetermined reference point of view, the data of the reference image, determining, for each pixel in the display image, a color of the image of the object by using a color of an image of the same object in the reference image, specifying a position of an image of another object reflected on the object in the reference image, acquiring a color at the position, and combining the colors to thereby determine a final value of the pixel representing the reflection.
