# US 11,445,246 B1 · Activity-based selection of replacement content > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,445,246 B1 - Document ID: US11445246B1 - Status: Granted - Kind: B1 - Application number: 16/834,060 - Filed: 2020-03-30 - Granted: 2022-09-13 - Earliest claimed: 2020-03-30 - Assignees: Twitch Interactive, Inc. - Applicants: Twitch Interactive, Inc. - Inventors: Josiah Eatedali - Web page: https://patentgrep.com/patent/US11445246B1 ## CPC - A63F 13/35: Details of game servers - H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)} - A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) - A63F 13/655: by importing photos, e.g. of the player - A63F 13/86: Watching games played by other players - H04N 21/2187: Live feed - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/4223: Cameras (H04N23/00 takes precedence) - H04N 21/4316: {for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window} - H04N 21/4781: {Games} - H04N 21/2668: Creating a channel for a dedicated end-user group, e.g. insertion of targeted commercials based on end-user profiles {(information retrieval from the Internet by querying with filtering and personalisation G06F16/9535; arrangements for replacing or switching information during the broadcast H04H20/10; push services over packet-switching network H04L12/1859; adaptation of message content in packet-switching networks H04L51/063)} - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An input video stream may include primary video content, such as output from a first video game. An indication of a decrease in player activity in the first video game may be detected. A determination may be made, based at least in part on the indication of the decrease in player activity, to replace a portion of the primary video content with secondary video content. The decrease in player activity in the first video game may coincide with an increase in player activity in a second video game. An indication of this increase in player activity may be detected. The second video game may be selected, based at least in part on the indication of the increase in player activity, to provide the secondary video content. An output video stream is provided in which the selected portion of the primary video content is replaced by the secondary video content. ## Claims ### Claim 1 A computing system comprising: one or more processors; and one or more memories having stored therein computing instructions that, upon execution by the one or more processors, cause the computing system to perform operations comprising: receiving a live input video stream of live primary video content; receiving a first live additional video stream corresponding to a video game; detecting an indication of a temporary increase in player activity associated with the video game; determining, by one or more computing devices, based at least in part on the indication of the temporary increase in player activity, to replace a selected portion of the live primary video content with live secondary video content from the first live additional video stream, wherein the selected portion of the live primary video content is between a first portion of the live primary video content and a second portion of the live primary video content; and providing an output video stream including the first portion of the live primary video content, the live secondary video content, and the second portion of the live primary video content, wherein the live secondary video content replaces the selected portion of the live primary video content in the output video stream. ### Claim 2 (depends on 1) The computing system of claim 1, wherein the first live additional video stream is selected from a plurality of live additional video streams to provide the live secondary video content based at least in part on a comparison of player activity levels for the plurality of live additional video streams. ### Claim 3 (depends on 1) The computing system of claim 1, wherein the indication of the temporary increase in player activity is detected by performing an image analysis on the live primary video content to identify a visual cue within the live primary video content. ### Claim 4 (depends on 1) The computing system of claim 1, wherein the indication of the temporary increase in player activity is detected based on video game state data provided by the video game. ### Claim 5 A computer-implemented method comprising: receiving, by a video streaming service, an input video stream of primary video content; receiving, by the video streaming service, a first additional video stream corresponding to an application; detecting an indication of a temporary increase in user activity associated with the application; determining, by one or more computing devices, based at least in part on the indication of the temporary increase in user activity, to replace a selected portion of the primary video content with secondary video content from the first additional video stream, wherein the selected portion of the primary video content is between a first portion of the primary video content and a second portion of the primary video content; and providing an output video stream including the first portion of the primary video content, the secondary video content, and the second portion of the primary video content, wherein the secondary video content replaces the selected portion of the primary video content in the output video stream. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein the first additional video stream is selected from a plurality of additional video streams to provide the secondary video content based at least in part on a comparison of user activity levels for the plurality of additional video streams. ### Claim 7 (depends on 5) The computer-implemented method of claim 5, wherein the first additional video stream is selected to provide the secondary video content based at least in part on a predicted duration of the temporary increase in user activity. ### Claim 8 (depends on 5) The computer-implemented method of claim 5, wherein the indication of the temporary increase in user activity corresponds at least one of an increase in user movement, a user joining the application, or a start of a match or battle. ### Claim 9 (depends on 5) The computer-implemented method of claim 5, wherein the indication of the temporary increase in user activity is detected by performing an image analysis on the first additional video stream to identify a visual cue within the first additional video stream. ### Claim 10 (depends on 5) The computer-implemented method of claim 5, wherein the indication of the temporary increase in user activity is detected based on application state information provided by the application. ### Claim 11 (depends on 5) The computer-implemented method of claim 5, further comprising determining to stop replacement of the primary video content with the secondary video content based at least in part on a detection of a decrease in user activity associated with the application. ### Claim 12 (depends on 7) The computer-implemented method of claim 7, wherein the predicted duration of the temporary increase in user activity is estimated based at least in part on a type of event to which the temporary increase in user activity corresponds. ### Claim 13 (depends on 9) The computer-implemented method of claim 9, wherein the visual cue comprises text or graphics within video captured from the application. ### Claim 14 (depends on 9) The computer-implemented method of claim 9, wherein the visual cue comprises a user entering into a field of view of a webcam capturing video included in the first additional video stream. ### Claim 15 One or more non-transitory computer-readable storage media having stored thereon computing instructions that, upon execution by one or more computing devices, cause the one or more computing devices to perform operations comprising: receiving, by a video streaming service, an input video stream of primary video content; receiving, by the video streaming service, a first additional video stream corresponding to an application; detecting an indication of a temporary increase in user activity associated with the application; determining, by one or more computing devices, based at least in part on the indication of the temporary increase in user activity, to replace a selected portion of the primary video content with secondary video content from the first additional video stream, wherein the selected portion of the primary video content is between a first portion of the primary video content and a second portion of the primary video content; and providing an output video stream including the first portion of the primary video content, the secondary video content, and the second portion of the primary video content, wherein the secondary video content replaces the selected portion of the primary video content in the output video stream. ### Claim 16 (depends on 15) The one or more non-transitory computer-readable storage media of claim 15, wherein the first additional video stream is selected from a plurality of additional video streams to provide the secondary video content based at least in part on a comparison of user activity levels for the plurality of additional video streams. ### Claim 17 (depends on 15) The one or more non-transitory computer-readable storage media of claim 15, wherein the selected portion of the primary video content comprises one or more complete frames of the primary video content. ### Claim 18 (depends on 15) The one or more non-transitory computer-readable storage media of claim 15, wherein the selected portion of the primary video content comprises one or more portions of one or more frames of the primary video content. ### Claim 19 (depends on 18) The one or more non-transitory computer-readable storage media of claim 18, wherein the indication of the temporary increase in user activity is detected based on service data related to the first additional video stream. ### Claim 20 (depends on 19) The one or more non-transitory computer-readable storage media of claim 19, wherein the service data indicates an increase in chat messages related to the application.