System and method for low-latency communication over unreliable networks

A cloud game interactive system for low-latency communication from a game execution system and a video stream decoder over an unreliable network using at least one predictive machine learning model, wherein the game execution system and the video stream decoder comprises one or more processors, one or more non-transitory computer-readable mediums storing one or more sequences of instructions, which when executed by the one or more processors, cause:

• representing at least one frame of time series data at the game execution system, wherein the at least one frame of time series data is a series of data points indexed in time order;
• recording at least one output stream, a metadata associated with the at least one output stream, and a plurality of external inputs from the game execution system in an interaction recorder of the video stream decoder, wherein the at least one output stream comprises the at least one frame of time series data;
• segmenting an image into at least one background area stream and at least one character-centered portion, wherein the at least one background area stream is captured from a plurality of users;
• compressing the at least one character-centered portion of the image into a character focus stream;
• detecting, at the video stream decoder, at least one lost frame of time series data using a frame of time series data from previously received frames;
• training the at least one predictive machine learning model at the game execution system for a predictive frame regeneration by providing the at least one output stream from the interaction recorder or an instantaneous application state, comprising interactive application state-variables, as an input, wherein the interactive application state-variables describe a state space representation of a closed-loop control defined by a user's interaction with an application that drives creation of the audio and video;
• transmitting results of the training or interactions, between the game execution system and the video stream decoder, in a time series to the video stream decoder, from the game execution system;
• regenerating the at least one lost frame of time series data, at video stream decoder, using the at least one predictive machine learning model based on the at least one output stream to obtain at least one regenerated frame of time series data;
• and
• combining an output stream from an application stream with the at least one regenerated frame of time series data obtained from the at least one predictive machine learning model, at the video stream decoder to obtain a modified output stream, wherein the application stream comprises the at least one frame of time series data.

An adaptive model selection system, for low-latency communication from a first device to a second device over an unreliable network using at least one predictive machine learning model, wherein the first device and the second device comprises one or more processors, one or more non-transitory computer-readable mediums storing one or more sequences of instructions, which when executed by the one or more processors, cause:

• representing at least one frame of time series data at the first device, wherein the at least one frame of time series data is a series of data points indexed in time order;
• recording at least one output stream, a metadata associated with the at least one output stream, and a plurality of external inputs from the first device in an interaction recorder of the second device, wherein the at least one output stream comprises the at least one frame of time series data;
• segmenting an image into at least one background area stream and at least one character-centered portion, wherein the at least one background area stream is captured from a plurality of users;
• compressing the at least one character-centered portion of the image into a character focus stream;
• detecting, at the second device, at least one lost frame of time series data using a frame of time series data from previously received frames;
• pushing, from the second device to the first device, a decoded frame of time series data;
• selecting, at the first device, a best set of predictive model package to obtain bundle models, and sending, from the first device to the second device, the best set of predictive model package;
• training the at least one predictive machine learning model, from the best set of predictive model package, at the first device for a predictive frame regeneration by providing the at least one output stream from the interaction recorder as an input;
• transmitting results of the training or interactions, between the first device and the second device, in a time series to the second device, from the first device;
• regenerating the at least one lost frame of time series data, at the second device, using the at least one predictive machine learning model based on the at least one output stream to obtain at least one regenerated frame of time series data;
• and
• combining an output stream from an application stream with the at least one regenerated frame of time series data obtained from the at least one predictive machine learning model, at the second device to obtain a modified output stream, wherein the application stream comprises the at least one frame of time series data.