# US 12,015,806 B2 · Method and data processing system for making predictions during a live event stream > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,015,806 B2 - Document ID: US12015806B2 - Status: Granted - Kind: B2 - Application number: 17/017,925 - Filed: 2020-09-11 - Published: 2022-03-17 - Granted: 2024-06-18 - Earliest claimed: 2020-09-11 - Assignees: Noice Entertainment Oy - Applicants: Noice Entertainment Oy - Inventors: Jussi Laakkonen; Florian Ziegler - Web page: https://patentgrep.com/patent/US12015806B2 ## CPC - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - A63F 13/86: Watching games played by other players - G06N 20/00: Machine learning - G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/42: {of sport video content} - G07F 17/3225: {Data transfer within a gaming system, e.g. data sent between gaming machines and users} - G07F 17/3232: {wherein the operator is informed} - G07F 17/326: {Game play aspects of gaming systems} - G07F 17/3288: {Betting, e.g. on live events, bookmaking} - H04N 21/2187: Live feed - H04N 21/251: {Learning process for intelligent management, e.g. learning user preferences for recommending movies (details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08)} - H04N 21/472: End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content {(end-user interfaces for retrieving video data from a database G06F16/739; network services for supporting unicast streaming H04L65/612)} - G06V 20/44: {Event detection} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G07F: COIN-FREED OR LIKE APPARATUS (coin sorting G07D3/00; coin testing G07D5/00; {handling coins or paper currencies apart from payment activated apparatus G07D; payment architectures, schemes or protocols G06Q20/00}) ## Abstract A method of operating a data processing system and data processing system for enabling a spectator to make predictions during a live event stream before the predictable events take place, wherein the method is performed by the data processing system comprising a server configured to communicate with a means for transferring the live event stream over a communication network and to provide the prediction environment to the spectator devices for making predictions. ## Claims ### Claim 1 A method of operating a data processing system for making predictions during a video game stream, wherein the video game stream comprises a video stream and the video game stream is received from a means capturing the video game in a first server, the method comprising the steps of: defining a sub-category of the video game to be streamed from the first server to at least one spectator device; pre-defining a set of predictable event actions of the video game according to the defined sub-category of the video game before streaming the video game corresponding to the defined sub-category of the video game; associating each predictable event action of the set of defined predictable event actions with an electronic prediction means; providing the electronic prediction means associated with the set of defined predictable event actions to at least one spectator device; generating a data stream of the predictable event actions from event data from participant devices of video game event participants, wherein the generated data stream is derived from one or more of: video streams of the video game streams, originating from the devices of one or more participants, who are competing individually or against others; video or data streams from one or more participant devices having a game client running game client software; or video or data streams from one or more game servers coupled to the game client; and recording in the first server a data of the predictable event actions from the at least one spectator device; streaming the video game corresponding to the defined sub-category of the video game received from the means capturing the video game by the first server to the at least one spectator device; detecting event actions from the generated data stream by a real time object and event detection engine; comparing in real time the recorded data of the event actions being predicted and the detected event actions from the generated data stream to identify the prediction results; displaying the identified prediction result in a user interface of the at least one spectator device; wherein the first server is configured to determine delays in distributing the generated data stream and to accordingly synchronize distributing the generated data stream and engagement in predictions. ### Claim 2 (depends on 1) The method according to claim 1, wherein generating the data stream of predictable event actions from the event data source comprises generating the data stream from the video of the video game to be streamed, an application program interface, a software development kit or combination thereof. ### Claim 3 (depends on 1) The method according to claim 1, wherein generating the data stream is performed by analyzing the video game stream using a machine learning algorithm based on at least one object or event detection technique. ### Claim 4 (depends on 3) The method according to claim 3, wherein using the machine learning algorithm further comprises training the machine learning algorithm by using an event replay data for generating the data and video streams. ### Claim 5 (depends on 1) The method according to claim 1, wherein generating the data stream is performed by analyzing in the first server the event data received from the one or more participant devices, from one or more video game servers or from the means capturing the video game stream. ### Claim 6 (depends on 1) The method according to claim 1, wherein generating the data stream further comprises receiving one or more of: event-based analysis, event-based statistics, and event-based commentary. ### Claim 7 (depends on 1) The method according to claim 1, wherein detecting the event actions comprises detecting triggering actions by implementing a software interface library configured to interface with the video game being played and configured to analyze the data or the state of the video game being played. ### Claim 8 (depends on 1) The method of according to claim 1, wherein detecting the event actions further comprises detecting triggering actions by at least one of: performing an audio and video recognition analysis of the video game stream; or using a real time statistics feed of the video game. ### Claim 9 (depends on 1) The method according to claim 1, wherein detecting the event actions further comprises detecting sub-event attributes by receiving feedback on the generated sub-event attribute from one or more spectators of the video game stream; and updating the generated sub-event attribute for subsequent use based on the received feedback. ### Claim 10 (depends on 1) The method according to claim 1, wherein generating the data stream comprises generating information about a type of action, location where the event action is predicted to take place, identity of a performer of the action, a time of occurrence of the corresponding predictable event actions in the video game stream or dependency on a preceding predictable event action. ### Claim 11 (depends on 1) The method according to claim 1, wherein providing the electronic prediction means comprise providing one or more defined selectable objects, and wherein each of the one or more defined selectable objects presents at least one defined predictable event action. ### Claim 12 (depends on 1) The method according to claim 1, wherein generating the data stream of predictable event actions further comprises: receiving at least one prediction input related to the defined predictable event actions from the at least one spectator device of the at least one spectator in response to the provided electronic prediction means associated with the defined predictable event actions; and processing the received at least one prediction input to generate data of event actions being predicted. ### Claim 13 (depends on 1) The method according to claim 1, wherein defining the sub-category of the video game comprises defining a type of the video game, or a specific game mode of the video game. ### Claim 14 (depends on 1) The method according to claim 1, wherein the method further comprises providing an award based on the identified prediction result. ### Claim 15 A data processing system for making predictions during a video game stream, wherein the video game stream comprises a video stream and the video game stream is received from a means capturing the video game, the data processing system comprising: at least one spectator device operable by the spectator; and at least one first server in communication with the at least one spectator device over a communications network, wherein at least one of the first servers is configured to: define a sub-category of video game to be streamed from the first server to at least one spectator device; pre-define a set of predictable event actions of the video game according to the defined sub-category of the video game before streaming the video game corresponding to the defined sub-category of the video game; associate each predictable event action of the set of defined predictable event actions with an electronic prediction means; provide the electronic prediction means associated with the set of defined predictable event actions to the at least one spectator device for selection by the at least one spectator; generate a data stream of the predictable event actions from event data from participant devices of video game event participants, wherein the generated data stream is derived from one or more of: video streams of the video game streams, originating from the devices of one or more participants, who are competing individually or against others; video or data streams from one or more participant devices having a game client running game client software; or video or data streams from one or more game servers coupled to the game client; and record a data of the predictable event actions from the at least one spectator device; stream the video game corresponding to the defined sub-category of the video game received from the means capturing the video game to the at least one spectator device; detect event actions from the generated data stream by a real time object and event detection engine; compare in real time the recorded data of the event actions being predicted and the detected event actions from the generated data stream to identify the prediction results; display the identified prediction result in a user interface of the at least one spectator device; wherein the first server is configured to determine delays in distributing the generated data stream and to accordingly synchronize distributing the generated data stream and engagement in predictions. ### Claim 16 (depends on 15) The data processing system according to claim 15, wherein the event data source comprises the data stream from the video of the video game to be streamed, an application program interface, a software development kit or combination thereof. ### Claim 17 (depends on 15) The data processing system according to claim 15, wherein at least one of the first servers or the spectator devices is operable to generate the data stream by analyzing the video game stream using a machine learning algorithm residing thereon, the machine learning algorithm based on at least one object and event detection technique. ### Claim 18 (depends on 15) The data processing system according to claim 15, wherein the at least one first server is further configured to train the machine learning algorithm by using an event replay data for generating the data and video streams. ### Claim 19 (depends on 15) The data processing system according to claim 15, wherein the at least one first server is further configured to receive the data stream comprising one or more of: event-based analysis, event-based statistics, and event-based commentary. ### Claim 20 (depends on 15) The data processing system according to claim 15, wherein the at least one first server is further configured to detect the event actions by implementing a software interface library. ### Claim 21 (depends on 15) The data processing system according to claim 15, wherein the at least one of the first server is further configured to detect the event actions comprising triggering actions by at least one of: performing an audio and video recognition analysis of the video game stream; or using a real time statistics feed of the video game. ### Claim 22 (depends on 15) The data processing system according to claim 15, wherein the at least one of the first server or the spectator device is configured to generate the sub-event attributes by: receiving feedback on the generated sub-event attribute from one or more participants of the video game stream; and updating the generated sub-event attribute for subsequent use based on the received feedback. ### Claim 23 (depends on 15) The data processing system according to claim 15, wherein the data stream for at least one of the predictable event actions, provided by at least one of the first server or the spectator device, comprises information about a type of action, location where the event action is predicted to take place, identity of a performer of the action, and a time of occurrence of the corresponding predictable event actions in the video game stream. ### Claim 24 (depends on 15) The data processing system according to claim 15, wherein the electronic prediction means comprise one or more defined selectable objects, and wherein each of the one or more defined selectable objects presents at least one defined predictable event.