# US 12,604,057 B2 · Streaming system and method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,604,057 B2 - Document ID: US12604057B2 - Status: Granted - Kind: B2 - Application number: 17/964,989 - Filed: 2022-10-13 - Published: 2023-04-27 - Granted: 2026-04-14 - Earliest claimed: 2021-10-25 - Assignees: Sony Interactive Entertainment Inc. - Applicants: Sony Interactive Entertainment Inc. - Inventors: Matthew William Sanders; Justin Lim; Sahin Serdar Kocdemir; Marina Villanueva Barreiro - Web page: https://patentgrep.com/patent/US12604057B2 ## CPC - H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)} - H04N 19/42: characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation (H04N19/635 takes precedence) - H04N 21/4223: Cameras (H04N23/00 takes precedence) - H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/4402: involving reformatting operations of video signals for household redistribution, storage or real-time display {(details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764)} - H04N 21/440227: {by decomposing into layers, e.g. base layer and one or more enhancement layers} - H04N 21/4436: {Power management, e.g. shutting down unused components of the receiver (power management in computer systems G06F1/3203; hibernate or awake process in computer systems G06F9/4418)} - H04N 21/4621: {Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen (arrangements for using the results of monitoring on user's side in broadcast systems H04H60/65; flow control in packet networks H04L47/10)} - H04N 21/658: Transmission by the client directed to the server - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method of displaying streamed content includes receiving streamed content from a server, decoding the streamed content, outputting the decoded streamed content for display, and estimating a degree of attention of a user on the streamed content, where the decoding step comprises decoding the content to a quality that is dependent upon the estimated degree of attention of the use. ## Claims ### Claim 1 A client device for displaying streamed content, comprising: a receiver to receive streamed content from a server; a variable decoder configured to decode the streamed content in a decoding process, with successive stages of the decoding process decoding the streamed content to a higher quality from decoded outputs of prior stages; an output for displaying the decoded streamed content; and an attention estimation processor configured to estimate a degree of attention of a user on the streamed content; wherein the streamed content has been encoded using an encoding process with a trained multi-stage neural network autoencoder configured for preserving size and spatial resolution of the streamed content in a plurality of convolution layers, the variable decoder is further configured to decode the streamed content by performing at least one stage of the decoding process in a reversal of the encoding process with the trained multi-stage neural network autoencoder based only on the streamed content, and the variable decoder is configured to selectively decode the streamed content in accordance with the estimated degree of attention of the user. ### Claim 2 (depends on 1) The client device of claim 1, wherein the variable decoder is further configured to decode the streamed content to a higher quality including a higher spatial resolution representation of the streamed content, in accordance with detection of a high estimated degree of attention of the user. ### Claim 3 (depends on 1) The client device of claim 1, further comprising: a transmitter configured to transmit to the server data relating to the degree of attention of the user on the streamed content. ### Claim 4 (depends on 3) The client device of claim 3, wherein if the user is estimated to be fully engaged with the streamed content, then the variable decoder performs all stages of the decoding process. ### Claim 5 (depends on 1) The client device of claim 1, wherein if the user is estimated to be temporarily disengaged from the streamed content, then the variable decoder performs fewer than all of the stages of the decoding process. ### Claim 6 (depends on 1) The client device of claim 1, wherein if the user is estimated to be one of absent and disengaged from the streamed content for a threshold period, then the variable decoder selectively decodes the streamed content to a low quality. ### Claim 7 (depends on 1) The client device of claim 1, further comprising: a gaze tracker configured to identify a region of the streamed content corresponding to a gaze of the user, wherein the variable decoder is further configured to decode the region of the streamed content corresponding to the gaze of the user to a quality higher than a portion of the streamed content outside of the region. ### Claim 8 (depends on 1) The client device of claim 1, in which a highest decoder quality is set responsive to one or more of: i. battery level; and ii. processor load. ### Claim 9 (depends on 1) The client device of claim 1, in which: the output of the variable decoder is post-processed to generate an output image for viewing by the user. ### Claim 10 A method of displaying streamed content at a client device, comprising: receiving streamed content from a server; decoding the streamed content using a variable decoder in a decoding process, wherein successive stages of the decoding process decode the streamed content to a higher quality from decoded outputs of prior stages; displaying the decoded streamed content; and estimating a degree of attention of a user on the streamed content; wherein the streamed content has been encoded using an encoding process with a trained multi-stage neural network autoencoder configured for preserving size and spatial resolution of the streamed content in a plurality of convolution layers, decoding includes decoding the streamed content by performing at least one stage of the decoding process in a reversal of the encoding process with the trained multi-stage neural network autoencoder based only on the streamed content, and decoding further comprises selectively decoding the streamed content in accordance with the estimated degree of attention of the user. ### Claim 11 A non-transitory, computer readable storage medium containing computer program comprising computer executable instructions adapted to cause a computer system to perform operations comprising: receiving streamed content from a server; decoding the streamed content using a variable decoder in a decoding process, wherein successive stages of the decoding process decode the streamed content to a higher quality from decoded outputs of prior stages; displaying the decoded streamed content; and estimating a degree of attention of a user on the streamed content; wherein the streamed content has been encoded using an encoding process with a trained multi-stage neural network autoencoder configured for preserving size and spatial resolution of the streamed content in a plurality of convolution layers, decoding includes decoding the streamed content by performing at least one stage of the decoding process in a reversal of the encoding process with the trained multi-stage neural network autoencoder based only on the streamed content, and decoding further comprises selectively decoding the streamed content in accordance with the estimated degree of attention of the user.