# US 11,077,363 B2 · Video game overlay > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,077,363 B2 - Document ID: US11077363B2 - Status: Granted - Kind: B2 - Application number: 16/811,519 - Filed: 2020-03-06 - Published: 2020-07-02 - Granted: 2021-08-03 - Earliest claimed: 2009-06-01 - Assignees: Sony Interactive Entertainment Inc. - Applicants: Sony Interactive Entertainment America LLC - Inventors: Andrew Buchanan Gault; David Perry; Rui Filipe Andrade Pereira - Web page: https://patentgrep.com/patent/US11077363B2 ## CPC - A63F 13/10 - A63F 13/355: Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client - A63F 13/12 - A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) - A63F 13/2145: the surface being also a display device, e.g. touch screens - A63F 13/30: Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers - A63F 13/32: using local area network [LAN] connections - A63F 13/327: using wireless networks, e.g. Wi-Fi® or piconet - A63F 13/33: using wide area network [WAN] connections - A63F 13/335: using Internet - A63F 13/45: Controlling the progress of the video game - G06F 9/45537: {Provision of facilities of other operating environments, e.g. WINE (I/O emulation G06F13/105)} - H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) - H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) - A63F 2300/407: Data transfer via internet - A63F 2300/409: Data transfer via television network - A63F 2300/538: for performing operations on behalf of the game client, e.g. rendering - A63F 2300/552: for downloading to client devices, e.g. using OS version, hardware or software profile of the client device - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A video server is configured to provide streaming video to players of computer games over a computing network. The video server can provided video of different games to different players simultaneously. This is accomplished by rendering several video streams in parallel using a single GPU (Graphics Processing Unit). The output of the GPU is provided to graphics processing pipelines that are each associated with a specific client/player and are dynamically allocated as needed. A client qualifier may be used to assure that only clients capable of presenting the streaming video to a player at a minimum level of quality receive the video stream. Video frames provided by the video server optionally include overlays added to the output of the GPU. These overlays can include voice data received from another game player. ## Claims ### Claim 1 An online video game delivery system, comprising: a video server system including a processor for executing game logic of an online video game in accordance with game commands received from a first client device and in accordance with a game state associated with the first client device to generate video data defining a video stream representing game play of the online video game by a user of the first client device, wherein the game logic is stored in a first portion of computer memory, wherein the first client device includes a handheld game playing device, and wherein the video stream is displayable on a display associated with the first client device; overlay logic executable by the video server system to combine overlay data with the video data defining the video stream to create overlay modified video data defining an output video stream, wherein the overlay logic is stored in a second portion of computer memory, the overlay data including a separately generated video stream corresponding to game play of the online video game by a user of a second client device; encoding logic executable by the video server system to generate an encoded version of the overlay modified video data defining an output video stream, wherein the encoding logic is stored in a third portion of computer memory; and an output device for transmitting the encoded version of the overlay modified video data over a network for decoding and rendering on the display associated with the first client device. ### Claim 2 (depends on 1) The online video game delivery system as recited in claim 1, further comprising: an input device for receiving game commands over the network from the handheld game playing device of the first client device. ### Claim 3 (depends on 1) The online video game delivery system as recited in claim 1, further comprising: a graphics processing unit configured to generate the video data defining the video stream representing game play of the online video game by the user of the first client device in accordance with output from the game logic executing on the processor. ### Claim 4 (depends on 3) The online video game delivery system as recited in claim 3, wherein the graphics processing unit is configured to generate the overlay data. ### Claim 5 (depends on 1) The online video game delivery system as recited in claim 1, wherein the overlay data is defined to augment a video game virtual environment generated by the game logic executing on the processor. ### Claim 6 (depends on 1) The online video game delivery system as recited in claim 1, wherein the overlay data is defined to replace a portion of a video game virtual environment generated by the game logic executing on the processor. ### Claim 7 (depends on 1) The online video game delivery system as recited in claim 1, wherein the overlay data defines an overlay graphic that is responsive to game commands received from the handheld game playing device of the first client device. ### Claim 8 (depends on 1) The online video game delivery system as recited in claim 1, wherein the overlay logic is configured to map the overlay data to specified pixels within the video data defining the video stream. ### Claim 9 (depends on 1) The online video game delivery system as recited in claim 1, wherein the separately generated video stream is generated by execution of the game logic of the online video game in accordance with game commands received from the second client device and in accordance with a game state associated with the second client device. ### Claim 10 (depends on 9) The online video game delivery system as recited in claim 9, wherein the game state associated with the second client device is separate from the game state associated with the first client device. ### Claim 11 (depends on 9) The online video game delivery system as recited in claim 9, wherein the game state associated with the second client device is the game state associated with the first client device. ### Claim 12 (depends on 1) The online video game delivery system as recited in claim 1, wherein the online game delivery system is configured to receive captured data from the first client device, the captured data including one or more of image data, video data, and audio data, wherein the overlay data includes at least a portion of the captured data received from the first client device. ### Claim 13 (depends on 1) The online video game delivery system as recited in claim 1, wherein the overlay logic is configured to modify the overlay data to resize an image defined by the overlay data prior to combining the overlay data with the video data defining the video stream. ### Claim 14 A method for providing an online video game, comprising: executing game logic of an online video game in accordance with game commands received from a first client device and in accordance with a game state associated with the first client device to generate video data defining a video stream representing game play of the online video game by a user of the first client device, wherein the first client device includes a handheld game playing device, and wherein the video stream is displayable on a display associated with the first client device; combining overlay data with the video data defining the video stream to create overlay modified video data defining an output video stream, the overlay data including a separately generated video stream corresponding to game play of the online video game by a user of a second client device; generating an encoded version of the overlay modified video data defining an output video stream; and transmitting the encoded version of the overlay modified video data over a network for decoding and rendering on the display associated with the first client device. ### Claim 15 (depends on 14) The method as recited in claim 14, wherein the overlay data is defined to augment a video game virtual environment generated by the executing the game logic. ### Claim 16 (depends on 14) The method as recited in claim 14, wherein the overlay data is defined to replace a portion of a video game virtual environment generated by the executing game logic. ### Claim 17 (depends on 14) The method as recited in claim 14, wherein the overlay data defines an overlay graphic that is responsive to game commands received from the handheld game playing device of the first client device. ### Claim 18 (depends on 14) The method as recited in claim 14, wherein the overlay data defines a separately generated video stream.