# US 11,998,840 B2 · Determining targeted display time of a video frame for a client based on server side timing information > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,998,840 B2 - Document ID: US11998840B2 - Status: Granted - Kind: B2 - Application number: 17/960,020 - Filed: 2022-10-04 - Published: 2023-01-26 - Granted: 2024-06-04 - Earliest claimed: 2019-10-01 - Assignees: Sony Interactive Entertainment Inc. - Applicants: Sony Interactive Entertainment Inc. - Inventors: Roelof Roderick Colenbrander; Mark E. Cerny - Web page: https://patentgrep.com/patent/US11998840B2 ## CPC - A63F 13/355: Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client - A63F 13/358: Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients - A63F 13/52: involving aspects of the displayed game scene - A63F 13/533: for prompting the player, e.g. by displaying a game menu - A63F 13/537: using indicators, e.g. showing the condition of a game character on screen - H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) - H04N 21/2343: involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements {(video transcoding H04N19/40; media packet handling at the source H04L65/762)} - H04N 21/25825: {involving client display capabilities, e.g. screen resolution of a mobile phone (optimising the visualisation of content during browsing in the Internet G06F16/9577; processing of terminal status or physical abilities in wireless networks H04W8/22; authentication in wireless network security H04W12/06)} - H04N 21/25833: {involving client hardware characteristics, e.g. manufacturer, processing or storage capabilities (allocation of resources considering hardware capabilities in multiprogramming arrangements G06F9/5044; allocation of resources considering software capabilities in multiprogramming arrangements G06F9/5055)} - A63F 2300/534: for network load management, e.g. bandwidth optimization, latency reduction - A63F 2300/538: for performing operations on behalf of the game client, e.g. rendering - A63F 2300/5593: involving scheduling aspects - H04L 65/61: for supporting one-way streaming services, e.g. Internet radio - H04L 67/12: specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks - H04N 21/4781: {Games} - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method for cloud gaming. The method including generating a video frame when executing a video game at a server. The method including performing a scan-out process to deliver the video frame to an encoder configured to compress the video frame, wherein the scan-out process begins at a flip-time of the video frame. The method including transmitting the video frame that is compressed to a client. The method including determining at the client a target display time for the video frame. The method including scheduling at the client a display time for the video frame based on the target display time. ## Claims ### Claim 1 A method of cloud gaming, comprising: aligning a server VSYNC signal of a server and a client VSYNC signal of a client; receiving at the client from the server a video frame that is encoded; receiving at the client from the server timing information for the video frame; determining a target display time of the video frame based on the timing information, wherein the target display time is a targeted occurrence of the server VSYNC signal; translating the target display time to a targeted occurrence of the client VSYNC signal; and scheduling a display time for the video frame based on the targeted occurrence of the client VSYNC signal. ### Claim 2 (depends on 1) The method of claim 1, wherein the timing information includes a time at which scan-out of the video frame started at the server. ### Claim 3 (depends on 1) The method of claim 1, wherein the timing information identifies the targeted occurrence of the server VSYNC signal. ### Claim 4 (depends on 1) The method of claim 1, wherein the display time is a fractional occurrence of the targeted occurrence of the client VSYNC signal. ### Claim 5 (depends on 1) The method of claim 1, further comprising: receiving in the timing information a flip-time and a simulation time associated with the video frame; and determining the targeted occurrence of the server VSYNC signal based on the flip-time and the simulation time, wherein scan-out of the video frame to an encoder occurs at the flip-time, wherein the simulation time indicates a time taken to generate the video frame. ### Claim 6 (depends on 1) The method of claim 1, further comprising: providing an offset between the server VSYNC signal and the client VSYNC signal, wherein the offset is determined based on a predetermined number of video frames that are received right before a corresponding next occurrence of the client VSYNC signal, wherein the server VSYNC signal and the client VSYNC signal are aligned to a frequency. ### Claim 7 (depends on 1) The method of claim 1, further comprising: sending the timing information in a data control packet using a graphics pipeline unit (GPU) application programming interface (API). ### Claim 8 A non-transitory computer-readable medium storing a computer program for cloud gaming, the computer-readable medium comprising: program instructions for aligning a server VSYNC signal of a server and a client VSYNC signal of a client; program instructions for receiving at the client from the server a video frame that is encoded; program instructions for receiving at the client from the server timing information for the video frame; program instructions for determining a target display time of the video frame based on the timing information, wherein the target display time is a targeted occurrence of the server VSYNC signal; program instructions for translating the target display time to a targeted occurrence of the client VSYNC signal; and program instructions for scheduling a display time for the video frame based on the targeted occurrence of the client VSYNC signal. ### Claim 9 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein in the method the timing information includes a time at which scan-out of the video frame started at the server. ### Claim 10 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein in the method the timing information identifies the targeted occurrence of the server VSYNC signal. ### Claim 11 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein in the method the display time is a fractional occurrence of the targeted occurrence of the client VSYNC signal. ### Claim 12 (depends on 8) The non-transitory computer-readable medium of claim 8, further comprising: program instructions for receiving in the timing information a flip-time and a simulation time associated with the video frame; and program instructions for determining the targeted occurrence of the server VSYNC signal based on the flip-time and the simulation time, wherein scan-out of the video frame to an encoder occurs at the flip-time, wherein the simulation time indicates a time taken to generate the video frame. ### Claim 13 (depends on 8) The non-transitory computer-readable medium of claim 8, further comprising: program instructions for providing an offset between the server VSYNC signal and the client VSYNC signal, wherein the offset is determined based on a predetermined number of video frames that are received right before a corresponding next occurrence of the client VSYNC signal, wherein the server VSYNC signal and the client VSYNC signal are aligned to a frequency. ### Claim 14 (depends on 8) The non-transitory computer-readable medium of claim 8, further comprising: program instructions for sending the timing information in a data control packet using a graphics pipeline unit (GPU) application programming interface (API). ### Claim 15 A computer system comprising: a processor; and memory coupled to the processor and having stored therein instructions that, if executed by the computer system, cause the computer system to execute a method for cloud gaming comprising: aligning a server VSYNC signal of a server and a client VSYNC signal of a client; receiving at the client from the server a video frame that is encoded; receiving at the client from the server timing information for the video frame; determining a target display time of the video frame based on the timing information, wherein the target display time is a targeted occurrence of the server VSYNC signal; translating the target display time to a targeted occurrence of the client VSYNC signal; and scheduling a display time for the video frame based on the targeted occurrence of the client VSYNC signal. ### Claim 16 (depends on 15) The computer system of claim 15, wherein in the method the timing information includes a time at which scan-out of the video frame started at the server. ### Claim 17 (depends on 15) The computer system of claim 15, wherein in the method the timing information identifies the targeted occurrence of the server VSYNC signal. ### Claim 18 (depends on 15) The computer system of claim 15, wherein in the method the display time is a fractional occurrence of the targeted occurrence of the client VSYNC signal. ### Claim 19 (depends on 15) The computer system of claim 15, the method further comprising: receiving in the timing information a flip-time and a simulation time associated with the video frame; and determining the targeted occurrence of the server VSYNC signal based on the flip-time and the simulation time, wherein scan-out of the video frame to an encoder occurs at the flip-time, wherein the simulation time indicates a time taken to generate the video frame. ### Claim 20 (depends on 15) The computer system of claim 15, the method further comprising: providing an offset between the server VSYNC signal and the client VSYNC signal, wherein the offset is determined based on a predetermined number of video frames that are received right before a corresponding next occurrence of the client VSYNC signal, wherein the server VSYNC signal and the client VSYNC signal are aligned to a frequency.