# US 2025/0168217 A1 · Augmenting multimedia streaming by anticipating events using artificial intelligence

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0168217 A1
- Document ID: US20250168217A1
- Status: Published application
- Kind: A1
- Application number: 19/025,621
- Filed: 2025-01-16
- Published: 2025-05-22
- Earliest claimed: 2021-10-27
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Prabindh Sundareson; Sachin Dattatray Pandhare; Shyam Raikar; Toshant Sharma; Prakash Yadav
- Web page: https://patentgrep.com/patent/US20250168217A1

## CPC

- H04L 65/611: for multicast or broadcast (systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06)
- H04L 65/764: {at the destination (reformatting of additional data in video clients H04N21/4355)}
- A63F 13/35: Details of game servers
- A63F 13/355: Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
- A63F 13/358: Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
- A63F 13/497: Partially or entirely replaying previous game actions
- A63F 13/67: adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use
- A63F 13/85: Providing additional services to players
- A63F 13/86: Watching games played by other players
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 20/00: Machine learning
- H04L 65/65: Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
- H04L 65/765: {intermediate}
- H04N 21/2187: Live feed
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/234345: {the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment}
- H04N 21/2408: {Monitoring of the upstream path of the transmission network, e.g. client requests (monitoring data switching networks utilization H04L43/0876; scheduling or organising the servicing of application requests H04L67/60)}
- H04N 21/251: {Learning process for intelligent management, e.g. learning user preferences for recommending movies (details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08)}
- H04N 21/26603: {for automatically generating descriptors from content, e.g. when it is not made available by its provider, using content analysis techniques}
- H04N 21/2668: Creating a channel for a dedicated end-user group, e.g. insertion of targeted commercials based on end-user profiles {(information retrieval from the Internet by querying with filtering and personalisation G06F16/9535; arrangements for replacing or switching information during the broadcast H04H20/10; push services over packet-switching network H04L12/1859; adaptation of message content in packet-switching networks H04L51/063)}
- H04N 21/2743: Video hosting of uploaded data from client
- H04N 21/4781: {Games}
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/10: Interfaces, programming languages or software development kits, e.g. for simulating neural networks
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Apparatuses, systems, and techniques to predict events within a gameplay session and modify a game broadcast based at least in part on the predicted events. In at least one embodiment, events within the gameplay session are predicted, which once predicted, cause assets to be generated and once the event is detected the assets are included in the game broadcast.

## Claims

### Claim 1

A method comprising:
predicting an interval of time based at least in part on user activity data associated with subject matter of a stream;
modifying one or more frames of the stream that correspond to the interval of time; and
causing the modified one or more frames to be broadcast in the stream.

### Claim 2 (depends on 1)

The method of claim 1, wherein the subject matter of the stream comprises a game, and the interval of time is a prediction of an occurrence of at least one in-game event.

### Claim 3 (depends on 1)

The method of claim 1, wherein the user activity data includes at least one of video data, audio data, device data, or biometric data.

### Claim 4 (depends on 1)

The method of claim 1, wherein the subject matter of the stream comprises a game, and the user activity data comprises at least one input obtained from a controller device associated with the game.

### Claim 5 (depends on 4)

The method of claim 4, wherein the controller device includes at least one of: a gaming console controller, a track pad, a mouse, a keyboard, or a motion controller.

### Claim 6 (depends on 1)

The method of claim 1, wherein modifying the one or more frames comprises applying an effect to the one or more frames that is not applied to one or more other frames of the stream.

### Claim 7 (depends on 1)

The method of claim 1, wherein predicting the interval of time is based, at least in part, on information associated with the stream that includes at least one of: data obtained from a remote client, one or more image frames corresponding to the stream, audio of the stream, or audio captured from the remote client.

### Claim 8 (depends on 1)

The method of claim 1, wherein modifying the one or more frames comprises displaying an overlay in the one or more frames.

### Claim 9 (depends on 1)

The method of claim 1, wherein modifying the one or more frames comprises including an audio cue in the one or more frames.

### Claim 10

A processor comprising:
one or more circuits to at least:
predict an event will occur within an interval of time based at least in part on user activity data;
modify a stream by at least applying an effect to one or more frames of the stream corresponding the interval of time; and
cause the one or more frames with the applied effect to be broadcast in the stream.

### Claim 11 (depends on 10)

The processor of claim 10, wherein the one or more circuits are to at least:
generate an asset; and
cause a state of the asset to be modified after the event is determined to have occurred.

### Claim 12 (depends on 11)

The processor of claim 11, wherein the event is to be predicted to occur within the interval of time based, at least in part, on an event type associated with the event and a type associated with the asset.

### Claim 13 (depends on 11)

The processor of claim 11, wherein the one or more circuits are to at least:
apply another effect corresponding to the asset to at least one frame of the stream.

### Claim 14 (depends on 13)

The processor of claim 13, wherein the other effect adds a delay between two or more successive frames of the at least one frame of the stream.

### Claim 15 (depends on 10)

The processor of claim 10, wherein the event is to be predicted to occur within the interval of time based, at least in part, on audio or video captured at a host computer system associated with the stream.

### Claim 16 (depends on 10)

The processor of claim 10, wherein the user activity data comprises inputs obtained from a controller device logged at a host computer system associated with the stream.

### Claim 17 (depends on 10)

The processor of claim 10, wherein the event is an in-game event.

### Claim 18

A system comprising:
one or more processors; and
memory storing instructions that, as a result of being executed by the one or more processors, cause the system to:
modify a streaming session by at least applying an effect to one or more frames corresponding a predicted interval of time, wherein the predicted interval of time is to predict when an event is to occur in the streaming session, and the predicted interval of time is to be generated based at least in part on an analysis of user inputs associated with the streaming session; and
cause the modified streaming session to be broadcast.

### Claim 19 (depends on 18)

The system of claim 18, wherein the memory further stores instructions that, as a result of being executed by the one or more processors, cause the system to:
during the predicted interval of time, obtain a set of images of the streaming session; and
generate a determination that the event occurred prior to an expiration of the predicted interval of time.

### Claim 20 (depends on 18)

The system of claim 18, wherein the memory further stores instructions that, as a result of being executed by the one or more processors, cause the system to:
cause the effect to no longer be applied to the streaming session in response to an expiration of the predicted interval of time.

### Claim 21 (depends on 18)

The system of claim 18, wherein the user inputs associated with the streaming session includes keystroke data obtained from a remote client.

### Claim 22 (depends on 18)

The system of claim 18, wherein the instructions further include instructions that cause the system to:
cause a machine learning model to generate the predicted interval of time.

### Claim 23 (depends on 18)

The system of claim 18, wherein the memory further stores instructions that, as a result of being executed by the one or more processors, cause the system to:
continue causing the effect to be applied beyond expiration of the predicted interval of time as a result of a determination that the event has occurred.

### Claim 24 (depends on 18)

The system of claim 18, wherein the streaming session is associated with a game, and the event is an in-game event.
