# US 2017/0257609 A1 · Reconstructing viewport upon user viewpoint misprediction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0257609 A1
- Document ID: US20170257609A1
- Status: Published application
- Kind: A1
- Application number: 15/602,442
- Filed: 2017-05-23
- Published: 2017-09-07
- Earliest claimed: 2015-01-22
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: David Chiyuan Chu; Eduardo Alberto Cuervo Laffaye; Johannes Peter Kopf; Alastair Wolman; Yury Degtyarev; Kyungmin Lee; Sergey Grizan
- Web page: https://patentgrep.com/patent/US20170257609A1

## CPC

- H04N 9/3179: {Video signal processing therefor}
- G02B 27/0172: {characterised by optical features}
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06T 15/20: Perspective computation
- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- H04N 13/128: Adjusting depth or disparity
- H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597)
- H04N 13/194: Transmission of image signals
- H04N 13/279: the virtual viewpoint locations being selected by the viewers or determined by tracking
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A head mounted display device including a processor configured to compute a rendered rendering surface of a predicted scene having a predicted user viewpoint, the predicted user viewpoint being a prediction of a viewpoint that a user will have at a point in time that was predicted for the user of the head mounted display device prior to the point in time, receive, from the user input device, a subsequent user navigation input near the point in time in the stream of user input, determine an actual user viewpoint based on the subsequent user navigation input, determine a user viewpoint misprediction based on the predicted user viewpoint and the actual user viewpoint, and reconstruct a viewport for the actual user viewpoint from the rendered rendering surface.

## Claims

### Claim 1

A head mounted display device comprising:
a user input device that receives a stream of user input;
a display device; and
a processor configured to:
compute a rendered rendering surface of a predicted scene having a predicted user viewpoint, the predicted user viewpoint being a prediction of a viewpoint that a user will have at a point in time that was predicted for the user of the head mounted display device prior to the point in time;
receive, from the user input device, a subsequent user navigation input near the point in time in the stream of user input;
determine an actual user viewpoint based on the subsequent user navigation input;
determine a user viewpoint misprediction based on the predicted user viewpoint and the actual user viewpoint;
reconstruct a viewport for the actual user viewpoint from the rendered rendering surface of the predicted scene having the predicted user viewpoint; and
display the viewport on the display device.

### Claim 2 (depends on 1)

The head mounted display device of claim 1, wherein reconstructing the viewport for the actual user viewpoint includes:
reprojecting from a portion of the rendered rendering surface to the viewport; and
interpolating one or more pixel data items to fill one or more visual holes in the reconstructed viewport.

### Claim 3 (depends on 2)

The head mounted display device of claim 2, wherein interpolating one or more pixel data items includes:
projecting a screen-space grid mesh onto a depth buffer for a depth map corresponding to the rendered rendering surface of the predicted scene, wherein the screen-space grid mesh includes an edge that extends between a foreground object and a background object in the predicted scene;
casting a ray from the actual user viewpoint to the screen-space grid mesh;
determining that the ray intersects with the edge in the screen-space grid mesh; and
calculating a pixel color for the interpolated one or more pixel data items based on a color of the background object in the predicted scene.

### Claim 4 (depends on 3)

The head mounted display device of claim 3, wherein the edge in the screen-space grid mesh projected onto the depth buffer is dilated by one or more pixels in width.

### Claim 5 (depends on 2)

The head mounted display device of claim 2, wherein interpolating one or more pixel data items includes reprojecting from a portion of a previously rendered viewport.

### Claim 6 (depends on 2)

The head mounted display device of claim 2, wherein the rendered rendering surface is a composite of a plurality of depth layers.

### Claim 7 (depends on 1)

The head mounted display device of claim 1, the processor being further configured to:
compute a plurality of alternative impulse input sequences and a plurality of rendered rendering surfaces corresponding to the plurality of alternative impulse input sequences;
receive a subsequent impulse input sequence from the stream of user input;
select an alternative impulse input sequence from the plurality of alternative impulse input sequences based on the subsequent impulse input sequence; and
wherein the viewport for the actual user viewpoint is reconstructed from the rendered rendering surface corresponding to the selected alternative impulse input sequence.

### Claim 8 (depends on 7)

The head mounted display device of claim 7, wherein selecting an alternative impulse input sequence includes determining which alternative impulse input sequence of the plurality of alternative impulse input sequences matches the subsequent impulse input sequence.

### Claim 9 (depends on 8)

The head mounted display device of claim 8, wherein selecting an alternative impulse input sequence further includes time-shifting each impulse input of the subsequent impulse input sequence to a nearest subsampled tick in the plurality of alternative impulse input sequences.

### Claim 10 (depends on 8)

The head mounted display device of claim 8, further configured in an augmented reality head mounted display device configuration.

### Claim 11

A method for reconstructing a viewport from a rendered rendering surface of a predicted user viewpoint comprising:
computing a rendered rendering surface of a predicted scene having a predicted user viewpoint, the predicted user viewpoint being a prediction of a viewpoint that a user will have at a point in time that was predicted for the user prior to the point in time;
receiving a subsequent user navigation input near the point in time in a stream of user input;
determining an actual user viewpoint based on the subsequent user navigation input;
determining a user viewpoint misprediction based on the predicted user viewpoint and the actual user viewpoint; and
reconstructing a viewport for the actual user viewpoint from the rendered rendering surface, wherein the viewport for the actual user viewpoint includes a portion of the plurality of non-displayed rendered pixel data items.

### Claim 12 (depends on 11)

The method of claim 11, wherein reconstructing the viewport for the actual user viewpoint includes:
reprojecting from a portion of the rendered rendering surface to the viewport; and
interpolating one or more pixel data items to fill one or more visual holes in the reconstructed viewport.

### Claim 13 (depends on 12)

The method of claim 12, wherein interpolating one or more pixel data items includes:
projecting a screen-space grid mesh onto a depth buffer for a depth map corresponding to the rendered rendering surface of the predicted scene, wherein the screen-space grid mesh includes an edge that extends between a foreground object and a background object in the predicted scene;
casting a ray from the actual user viewpoint to the screen-space grid mesh;
determining that the ray intersects with the edge in the screen-space grid mesh; and
calculating a pixel color for the interpolated one or more pixel data items based on a color of the background object in the predicted scene.

### Claim 14 (depends on 13)

The method of claim 13, wherein the edge in the screen-space grid mesh projected onto the depth buffer is dilated by one or more pixels in width.

### Claim 15 (depends on 12)

The method of claim 12, wherein interpolating one or more pixel data items includes reprojecting from a portion of a previously rendered viewport.

### Claim 16 (depends on 12)

The method of claim 12, wherein the rendered rendering surface is a composite of a plurality of depth layers, wherein a number of depth layers in the plurality of depth layers does not exceed an adjustable threshold value.

### Claim 17 (depends on 11)

The method of claim 11, further comprising:
computing a plurality of alternative impulse input sequences and a plurality of rendered rendering surfaces corresponding to the plurality of alternative impulse input sequences;
receiving a subsequent impulse input sequence in the stream of user input;
selecting an alternative impulse input sequence from the plurality of alternative impulse input sequences based on the subsequent impulse input sequence; and
wherein the viewport for the actual user viewpoint is reconstructed from the rendered rendering surface corresponding to the selected alternative impulse input sequence.

### Claim 18 (depends on 17)

The method of claim 17, wherein selecting an alternative impulse input sequence further includes time-shifting each impulse input of the subsequent impulse input sequence to a nearest subsampled tick of the plurality of alternative impulse input sequences.

### Claim 19

A head mounted display device comprising:
a user input device that receives a stream of user input;
a display device; and
a processor configured to:
compute a plurality of alternative impulse input sequences and a corresponding plurality of rendered rendering surfaces of a predicted scene having a predicted user viewpoint, the predicted user viewpoint being a prediction of a viewpoint that a user will have at a point in time that was predicted for the user of the head mounted display device prior to the point in time;
receive, from the user input device, a subsequent user navigation input and a subsequent impulse input sequence near the point in time in the stream of user input;
select an alternative impulse input sequence from the plurality of alternative impulse input sequences based on the subsequent impulse input sequence;
determine an actual user viewpoint based on the subsequent user navigation input;
determine a user viewpoint misprecliction based on the predicted user viewpoint and the actual user viewpoint;
select a rendered rendering surface from the plurality of rendered rendering surfaces that corresponds to the selected alternative impulse input sequence;
reconstruct a viewport for the actual user viewpoint from the selected rendered rendering surface; and
display the viewport on the display device.

### Claim 20 (depends on 19)

The head mounted display device of claim 19, further configured in an augmented reality head mounted display configuration.