# US 11,307,655 B2 · Multi-stream foveal display transport

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,307,655 B2
- Document ID: US11307655B2
- Status: Granted
- Kind: B2
- Application number: 16/575,804
- Filed: 2019-09-19
- Published: 2021-03-25
- Granted: 2022-04-19
- Earliest claimed: 2019-09-19
- Assignees: ATI Technologies ULC
- Applicants: ATI Technologies ULC
- Inventors: Guennadi Riguer; Syed Athar Hussain
- Web page: https://patentgrep.com/patent/US11307655B2

## CPC

- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 3/04: Context-preserving transformations, e.g. by using an importance map (panospheric to cylindrical image transformations G06T3/12)
- G06T 3/4092: Image resolution transcoding, e.g. by using client-server architectures
- H04L 65/61: for supporting one-way streaming services, e.g. Internet radio
- H04L 65/70: Media network packetisation
- H04L 65/764: {at the destination (reformatting of additional data in video clients H04N21/4355)}
- H04L 65/80: Responding to QoS
- H04L 67/131: Protocols for games, networked simulations or virtual reality
- H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
- H04N 19/162: User input
- H04N 19/164: Feedback from the receiver or from the transmission channel
- H04N 19/167: Position within a video image, e.g. region of interest [ROI]
- H04N 19/17: the unit being an image region, e.g. an object
- H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence)
- H04N 19/59: involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
- H04N 21/234345: {the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment}
- H04N 21/234363: {by altering the spatial resolution, e.g. for clients with a lower screen resolution}
- H04N 21/234381: {by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping}
- H04N 21/2365: Multiplexing of several video streams
- H04N 21/43072: {of multiple content streams on the same device}
- H04N 21/4347: {Demultiplexing of several video streams}
- H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)}
- H04N 21/816: {involving special video data, e.g 3D video}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Systems, apparatuses, and methods for using a multi-stream foveal display transport layer are disclosed. A virtual reality (VR) system includes a transmitter sending a plurality of streams over a display transport layer to a receiver coupled to a display. Each stream corresponds to a different image to be blended together by the receiver. The images include at least a foveal region image corresponding to a gaze direction of the eye and a background image which is a lower-resolution image with a wider field of view than the foveal region image. The phase timing of the foveal region stream being sent over the transport layer is adjusted with respect to the background stream to correspond to the location of the foveal region within the overall image. This helps to reduce the amount of buffering needed at the receiver for blending the images together to create a final image to be driven to the display.

## Claims

### Claim 1

A system comprising:
a rendering unit, comprising circuitry, configured to render a plurality of regions of a frame, wherein each region of the plurality of regions is rendered at a different resolution from other regions; and
a transmitter, comprising circuitry, configured to:
transmit each region of the plurality of regions as a separate image in a separate stream to a receiver; and
transmit a first stream by shifting a phase of the first stream with respect to a second stream by changing a time at which the first stream is transmitted, wherein the phase is shifted by an amount which corresponds to a location of a foveal region within the frame, wherein the first stream contains the foveal region, and wherein the second stream contains a background image.

### Claim 2 (depends on 1)

The system as recited in claim 1, wherein:
the foveal region is updated at a first refresh rate;
the background image is updated at a second refresh rate; and
the second refresh rate is different from the first refresh rate.

### Claim 3 (depends on 1)

The system as recited in claim 1, wherein the rendering unit is further configured to increase a number of regions being rendered at different resolutions responsive to receiving an indication that an eye saccade has been detected.

### Claim 4 (depends on 3)

The system as recited in claim 3, wherein the rendering unit is further configured to:
render two separate regions for a first frame prior to receiving the indication; and
render more than two separate regions for a second frame subsequent to receiving the indication.

### Claim 5 (depends on 3)

The system as recited in claim 3, wherein the rendering unit is further configured to adjust a refresh rate of the foveal region responsive to receiving the indication.

### Claim 6 (depends on 1)

The system as recited in claim 1, wherein the transmitter is further configured to convey a blend mask to the receiver to specify how to combine the plurality of regions to create the composite image.

### Claim 7 (depends on 1)

The system as recited in claim 1, wherein the transmitter is further configured to convey a background image scaling attribute to the receiver to specify how to scale the background image.

### Claim 8

A method comprising:
rendering, by a rendering unit comprising circuitry, a plurality of regions of a frame, wherein each region of the plurality of regions is rendered at a different resolution from other regions;
transmitting, by a transmitter comprising circuitry, each region as a separate image in a separate stream to a receiver; and
transmitting a first stream by shifting a phase of the first stream with respect to a second stream by changing a time at which the first stream is transmitted, wherein the phase is shifted by an amount which corresponds to a location of a foveal region within the frame, wherein the first stream contains the foveal region, and wherein the second stream contains a background image.

### Claim 9 (depends on 8)

The method as recited in claim 8, wherein:
the foveal region is updated at a first refresh rate;
the background image is updated at a second refresh rate; and
the second refresh rate is different from the first refresh rate.

### Claim 10 (depends on 8)

The method as recited in claim 8, further comprising increasing a number of regions being rendered at different resolutions responsive to receiving an indication that an eye saccade has been detected.

### Claim 11 (depends on 10)

The method as recited in claim 10, further comprising:
rendering two separate regions for a first frame prior to receiving the indication; and
rendering more than two separate regions for a second frame subsequent to receiving the indication.

### Claim 12 (depends on 10)

The method as recited in claim 10, further comprising adjusting a refresh rate of the foveal region responsive to receiving the indication.

### Claim 13 (depends on 8)

The method as recited in claim 8, further comprising conveying a blend mask to the receiver to specify how to combine the plurality of regions to create the composite image.

### Claim 14 (depends on 8)

The method as recited in claim 8, further comprising conveying a background image scaling attribute to the receiver to specify how to scale the background image.

### Claim 15

A system comprising:
a transmitter, comprising circuitry, configured to:
render a plurality of regions of a frame, wherein each region of the plurality of regions is rendered at a different resolution from other regions;
transmit each region as a separate image in a separate stream to a receiver; and
transmit a first stream by shifting a phase of the first stream with respect to a second stream by changing a time at which the first stream is transmitted, wherein the phase is shifted by an amount which corresponds to a location of a foveal region within the frame, wherein the first stream contains the foveal region, and wherein the second stream contains a background image; and
the receiver configured to:
combine the plurality of regions from a plurality of streams to create a composite image; and
drive the composite image to a display.

### Claim 16 (depends on 15)

The system as recited in claim 15, wherein:
the foveal region is updated at a first refresh rate;
the background image is updated at a second refresh rate; and
the second refresh rate is different from the first refresh rate.

### Claim 17 (depends on 15)

The system as recited in claim 15, wherein the transmitter is further configured to increase a number of regions being rendered at different resolutions responsive to receiving an indication that an eye saccade has been detected.

### Claim 18 (depends on 17)

The system as recited in claim 17, wherein the transmitter is further configured to:
render two separate regions for a first frame prior to receiving the indication; and
render more than two separate regions for a second frame subsequent to receiving the indication.

### Claim 19 (depends on 17)

The system as recited in claim 17, wherein the transmitter is further configured to adjust a refresh rate of the foveal region responsive to receiving the indication.

### Claim 20 (depends on 15)

The system as recited in claim 15, wherein the transmitter is further configured to convey a blend mask to the receiver to specify how to combine the plurality of regions to create the composite image.