# US 10,871,822 B2 · Image generation apparatus, image generation system, and image generation method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,871,822 B2
- Document ID: US10871822B2
- Status: Granted
- Kind: B2
- Application number: 16/309,587
- Filed: 2017-06-14
- Published: 2019-10-17
- Granted: 2020-12-22
- Earliest claimed: 2016-06-21
- Assignees: Sony Interactive Entertainment Inc.
- Applicants: Sony Interactive Entertainment Inc.
- Inventors: Daisuke Nakata
- Web page: https://patentgrep.com/patent/US10871822B2

## CPC

- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 27/0172: {characterised by optical features}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
- H04N 13/332: Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
- H04N 13/344: with head-mounted left-right displays
- H04N 13/383: for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An image processing section 750 draws an image viewed in a line-of-sight direction from a viewpoint position at a certain time. An image storage section 760 holds the image drawn by the image processing section 750 as past frames. A past frame selection section 790 selects, from among the past frames held in the image storage section 760, a substitute image to be viewed in the line-of-sight direction from a current viewpoint position. An image providing section 770 outputs the substitute image before providing the image viewed in the line-of-sight direction from the current viewpoint position, drawn by the image processing section 750.

## Claims

### Claim 1

An image generation apparatus for compensating for drawing delays, the image generating apparatus comprising:
an image generation section that draws a display image viewed in a line-of-sight direction from a viewpoint position at a certain time,
wherein the display image displays a virtual three-dimensional space from the viewpoint position on a display of a non-transmissive head mounted display (HMD),
a storage section that holds already displayed display images as past frames for a predetermined period of time;
a selection section that selects a substitute image to be viewed in the line-of-sight direction from a current viewpoint position from among the past frames held in the storage section; and
an image providing section that displays the substitute image on the display, to compensate for drawing delays, before displaying a new image viewed in the line-of-sight direction from the current viewpoint position drawn by the image generation section,
wherein the display image, the past frames, and the substitute image are all full frame images covering an entirety of the display.

### Claim 2 (depends on 1)

The image generation apparatus according to claim 1, wherein
the selection section compares at least one of the viewpoint position and line-of-sight direction of the past frame held in the storage section and at least one of the current viewpoint position and the line-of-sight direction and selects, as the substitute image, a past frame in which at least one of a difference between the viewpoint positions and an angle difference between the line-of-sight directions is smallest.

### Claim 3 (depends on 2)

The image generation apparatus according to claim 2, wherein
when at least one of the difference between the viewpoint positions and the angle difference between the line-of-sight directions is present within a predetermined threshold, the selection section selects the substitute image from among the past frames held in the storage section, and when at least one of the difference between the viewpoint positions and the angle difference between the line-of-sight directions exceeds the predetermined threshold, the selection section does not select the substitute image and the image providing section does not output the substitute image and provides the display image viewed in the line-of-sight direction from the current viewpoint position drawn by the image generation section.

### Claim 4 (depends on 1)

The image generation apparatus according to claim 1, further comprising:
a discarding section that compares, in the past frames held in the storage section, at least one of the viewpoint position and line-of-sight direction of the past frame held in the storage section and at least one of the current viewpoint position and the line-of-sight direction and deletes, from the storage section, a past frame in which at least one of the difference between the viewpoint positions and the angle difference between the line-of-sight directions is largest.

### Claim 5

An image generation system comprising:
a server; and
a client that is connected to the server via a network, wherein
the server includes
an image generation section that draws a display image viewed in a line-of-sight direction from a viewpoint position at a certain time, a storage section that holds a plurality of already displayed display images as past frames,
a selection section that selects a substitute image to be viewed in the line-of-sight direction from a current viewpoint position from among the past frames held in the storage section, and
an image providing section that transmits the substitute image to the client before providing a new display image viewed in the line-of-sight direction from the current viewpoint position drawn by the image generation section,
wherein the display image displays a virtual three-dimensional space from the viewpoint position, and
wherein the display image, the past frames, and the substitute image are all full frame images covering an entirety of a display of a non-transmissive head mounted display (HMD),
and
the client includes
a reprojection section that performs processing of deviating the substitute image received from the server in accordance with a motion amount of at least one of the viewpoint position and the line-of-sight direction and displaying the deviated image on the display.

### Claim 6

An image generation method comprising:
drawing a display image viewed in a line-of-sight direction from a viewpoint position at a certain time,
wherein the display image displays a virtual three-dimensional space from the viewpoint position on a display of a non-transmissive head mounted display (HMD);
storing already displayed display images as past frames for a predetermined period of time;
selecting a substitute image to be viewed in the line-of-sight direction from a current viewpoint position from among the past frames; and
outputting the substitute image to the display before providing a new image viewed in the line-of-sight direction from the current viewpoint position drawn to the display,
wherein the display image, the past frames, and the substitute image are all full frame images covering an entirety of the display.