# US 2021/0281912 A1 · Client-side offload of graphics effects processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0281912 A1
- Document ID: US20210281912A1
- Status: Published application
- Kind: A1
- Application number: 17/193,484
- Filed: 2021-03-05
- Published: 2021-09-09
- Earliest claimed: 2020-03-06
- Assignees: ATI Technologies ULC
- Applicants: Ati Technologies Ulc
- Inventors: Ihab AMER; Guennadi RIGUER; Thomas PERRY; Mehdi SAEEDI; Gabor SINES; Yang LIU
- Web page: https://patentgrep.com/patent/US20210281912A1

## CPC

- H04N 21/23614: {Multiplexing of additional data and video streams (arrangements for simultaneous broadcast of plural pieces of information H04H20/28)}
- H04N 21/44012: {involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs}
- H04N 21/4307: {Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen}
- H04N 21/43074: {of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV programme}
- H04N 21/4312: {involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations}
- H04N 21/4318: {by altering the content in the rendering process, e.g. blanking, blurring or masking an image region (image enhancement or restoration in general G06T5/00)}
- H04N 21/438: Interfacing the downstream path of the transmission network originating from a server, e.g. retrieving encoded video stream packets from an IP network
- H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/4781: {Games}
- H04N 21/8146: {involving graphical data, e.g. 3D object, 2D graphics}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A server offloads graphics effects processing to a client device with graphics processing resources by determining a modification to a graphics effects operation, generating a portion of a rendered video stream using the modification to the graphics effects operation, and providing an encoded representation of the portion of the rendered video stream to the client device, along with metadata representing the modification implemented. The client device decodes the encoded representation to recover the portion of the rendered video stream and selectively performs a graphics effects operation on the recovered portion to at least partially revert the resulting graphics effects for the portion to the intended effects without the modification implemented by the server.

## Claims

### Claim 1

A computer-implemented method comprising:
determining, at a server, a modification to a graphics effects operation to be implemented by the server in generating a portion of a first stream of rendered video frames for encoding and transmission to a client device via at least one network; and
transmitting, from the server to the client device via the at least one network, metadata representing the modification to the graphics effects operation for the portion of the first stream.

### Claim 2 (depends on 1)

The method of claim 1, wherein:
determining the modification to the graphics effects operation comprises determining a change to one or more parameters of the graphics effects operation that increases an encoding efficiency ; and
the metadata includes a representation of the change to the one or more parameters.

### Claim 3 (depends on 1)

The method of claim 1, wherein:
determining the modification to the graphics effects operation comprises selecting to bypass use of the graphics effects operation in generating the portion of the first stream; and
the metadata includes an indication that the graphics effects operation was bypassed in generating the portion of the first stream.

### Claim 4 (depends on 1)

The method of claim 1, wherein generating the first stream of rendered video frames comprises performing a rendering process that uses the modification to the graphics effects operation to generate the portion of the first stream.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
rendering, at the server, a portion of a second stream of rendered video frames using an unmodified version of the graphics effects operation; and
wherein generating the portion of the first stream comprises processing, at the server, the portion of the second stream of rendered video frames to modify, based on the modification to the graphics effects operation, a result of use of the unmodified version of the graphics effects operation present in the portion of the second stream to generate the portion of the first stream.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
receiving, from the client device, client status information indicating one or more of an indication of graphics processing resources available at the client device, one or more current status parameters of the client device, and one or more indications of user experience expectations; and
wherein determining the modification to the graphics effects operation comprises determining the modification to the graphics effects operation based on the client status information.

### Claim 7 (depends on 6)

The method of claim 6, wherein determining the modification to the graphics effects operation is further based on at least one of a current network status of the at least one network or a current resource status of the server.

### Claim 8 (depends on 1)

The method of claim 1, wherein determining the modification to the graphics effects operation comprises determining the modification to the graphics effects operation based on a negotiation with the client device to identify one or more graphics effects operations that are available for at least partial offloading from the server to the client device.

### Claim 9 (depends on 1)

The method of claim 1, the metadata includes an indication of one or more parameters for use by the client device during processing of the portion of the first stream to compensate for the modification to the graphics effects operation applied to the portion of the first stream.

### Claim 10

A server comprising:
a network interface configured to couple to at least one network;
at least one processor coupled to the network interface; and
a memory coupled to the at least one processor, the memory storing executable instructions configured to manipulate the at least one processor to:
determine a modification to a graphics effects operation to be implemented by the at least one processor in generating a portion of a first stream of rendered video frames for encoding and transmission to a client device via the network interface; and
provide metadata representing the modification to the graphics effects operation for the portion of the first stream to the network interface for transmission to the client device.

### Claim 11 (depends on 10)

The server of claim 10, wherein:
the executable instructions configured to manipulate the at least one processor to determine the modification to the graphics effects operation manipulate the at least one processor to determine a change to one or more parameters of the graphics effects operation that increases an encoding efficiency; and
the metadata includes a representation of the change to the one or more parameters.

### Claim 12 (depends on 10)

The server of claim 10, wherein:
the executable instructions configured to manipulate the at least one processor to determine the modification to the graphics effects operation manipulate the at least one processor to select to bypass use of the graphics effects operation in generating the portion of the first stream; and
the metadata includes an indication that the graphics effects operation was bypassed in generating the portion of the first stream.

### Claim 13 (depends on 10)

The server of claim 10, wherein the executable instructions configured to manipulate the at least one processor to generate the first stream of rendered video frames manipulate the at least one processor to perform a rendering process that uses the modification to the graphics effects operation to generate the portion of the first stream.

### Claim 14 (depends on 10)

The server of claim 10, wherein the executable instructions further comprise executable instructions configured to manipulate the at least one processor to:
render a portion of a second stream of rendered video frames using an unmodified version of the graphics effects operation; and
wherein the executable instructions configured to manipulate the at least one processor to generate the portion of the first stream comprises executable instructions to manipulate the at least one processor to process the portion of the second stream of rendered video frames to modify, based on the modification to the graphics effects operation, a result of use of the unmodified version of the graphics effects operation present in the portion of the second stream to generate the portion of the first stream.

### Claim 15 (depends on 10)

The server of claim 10, wherein the executable instructions further comprise executable instructions configured to manipulate the at least one processor to:
receive, from the client device, an indication of graphics processing resources available at the client device; and
wherein the executable instructions configured to manipulate the at least one processor to determine the modification to the graphics effects operation comprise executable instructions to manipulate the at least one processor to determine the modification to the graphics effects operation based on the indication of the graphics processing resources available at the client device.

### Claim 16 (depends on 15)

The server of claim 15, wherein the executable instructions to manipulate the at least one processor to determine the modification to the graphics effects operation manipulate the at least one processor to determine the modification further based on at least one of a current network status of the at least one network or a current resource status of the server.

### Claim 17 (depends on 10)

The server of claim 10, wherein the executable instructions to manipulate the at least one processor to determine the modification to the graphics effects operation manipulate the at least one processor to determine the modification to the graphics effects operation based on a negotiation with the client device to identify one or more graphics effects operations that are available for at least partial offloading from the server to the client device.

### Claim 18 (depends on 10)

The server of claim 10, wherein the metadata includes an indication of one or more parameters for use by the client device during processing of the portion of the first stream to compensate for the modification to the graphics effects operation applied to the portion of the first stream.

### Claim 19

A computer-implemented method comprising:
receiving, at a client device, an encoded representation of a portion of a first stream of rendered video frames from a server via at least one network;
receiving, at the client device, metadata from the server via the at least one network, the metadata including a representation of a modification to a graphics effects operation implemented by the server in generating the portion of the first stream;
decoding, at the client device, the encoded representation of the portion of the first stream to recover the portion of the first stream;
performing, at the client device, a graphics effects operation on the recovered portion of the first stream based on the modification to the graphics effects operation implemented by the server to generate a portion of a second stream of rendered video frames; and
providing the portion of the second stream for display at a display device associated with the client device.

### Claim 20 (depends on 19)

The method of claim 19, wherein:
the metadata includes a representation of a change to one or more parameters of the graphics effects operation implemented by the server as the modification to the graphics effects operation; and
performing the graphics effects operation comprises performing the graphics effects operation with one or more parameters that compensate for the change to one or more parameters represented in the metadata.

### Claim 21 (depends on 19)

The method of claim 19, wherein:
the metadata includes an indication that the graphics effects operation was bypassed by the server in generating the portion of the first stream; and
performing the graphics effects operation comprises performing one of the same graphics effects operation bypassed by the server or a graphics effects operation that is an alternative to, or compensates for, the graphics effects operation bypassed by the server.

### Claim 22 (depends on 19)

The method of claim 19, further comprising:
negotiating, with the server, a strategy for offloading graphics effects processing to the client device.

### Claim 23 (depends on 22)

The method of claim 22, wherein negotiating the strategy comprises:
providing, to the server, client status information indicating one or more of an indication of graphics processing resources available at the client device, one or more current status parameters of the client device, and one or more indications of user experience expectations.

### Claim 24 (depends on 19)

The method of claim 19, wherein performing, at the client device, the graphics effects operation comprises selectively performing the graphics effects operation at the client device based on one or both of: a current status of one or more parameters of the client device and a user-indicated preference regarding the graphics effects operation.

### Claim 25

A client device comprising:
a network interface configured to couple to at least one network;
at least one processor coupled to the network interface; and
a memory coupled to the at least one processor, the memory storing executable instructions configured to manipulate the at least one processor to:
receive an encoded representation of a portion of a first stream of rendered video frames from a server via the network interface;
receive metadata from the server via the network interface, the metadata including a representation of a modification to a graphics effects operation implemented by the server in generating the portion of the first stream;
decode the encoded representation of the portion of the first stream to recover the portion of the first stream;
perform a graphics effects operation on the recovered portion of the first stream based on the modification to the graphics effects operation implemented by the server to generate a portion of a second stream of rendered video frames; and
provide the portion of the second stream for display at a display device associated with the client device.

### Claim 26 (depends on 25)

The client device of claim 25, wherein:
the metadata includes a representation of a change to one or more parameters of the graphics effects operation implemented by the server as the modification to the graphics effects operation; and
the executable instructions to manipulate the at least one processor to perform the graphics effects operation manipulate the at least one processor to perform the graphics effects operation with one or more parameters that compensate for the change to one or more parameters represented in the metadata.

### Claim 27 (depends on 25)

The client device of claim 25, wherein:
the metadata includes an indication that the graphics effects operation was bypassed by the server in generating the portion of the first stream; and
the executable instructions to manipulate the at least one processor to perform the graphics effects operation manipulate the at least one processor to perform one of the same graphics effects operation bypassed by the server or a graphics effects operation that is an alternative to, or compensates for, the graphics effects operation bypassed by the server.

### Claim 28 (depends on 25)

The client device of claim 25, wherein the executable instructions further are configured to manipulate the at least one processor to:
negotiate, with the server, a strategy for offloading graphics effects processing to the client device.

### Claim 29 (depends on 28)

The client device of claim 28, wherein the executable instructions configured to manipulate the at least one processor to negotiate the strategy manipulate the at least one processor to provide, to the server, client status information indicating one or more of an indication of graphics processing resources available at the client device, one or more current status parameters of the client device, and one or more indications of user experience expectations.

### Claim 30 (depends on 25)

The client device of claim 25, wherein the executable instructions configured to manipulate the at least one processor to perform the graphics effects operation manipulate the at least one processor to selectively perform the graphics effects operation based on one or both of: a current status of one or more parameters of the client device and a user-indicated preference regarding the graphics effects operation.
