# US 11,310,349 B1 · Transforming multivariate time series data into image data to generate image-based predictions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,310,349 B1 - Document ID: US11310349B1 - Status: Granted - Kind: B1 - Application number: 16/878,506 - Filed: 2020-05-19 - Granted: 2022-04-19 - Earliest claimed: 2020-05-19 - Assignees: Amazon Technologies, Inc. - Applicants: Amazon Technologies, Inc. - Inventors: David L Pan; Roger Thomas John Moffatt - Web page: https://patentgrep.com/patent/US11310349B1 ## CPC - G06F 16/2477: {Temporal data queries} - H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP] - H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) - H04L 69/329: in the application layer [OSI layer 7] - H04N 19/103: Selection of coding mode or of prediction mode - H04N 19/159: Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction - H04N 19/176: the region being a block, e.g. a macroblock - H04W 4/38: for collecting sensor information - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) ## Abstract An image-based prediction service of a provider network may receive multivariate time series data (e.g., from different sensors for a machine) from a remote network of the client. The multivariate time series data includes different time series of data that are obtained from different data sources (e.g., sensors) over a time period. The image-based prediction service may transform the multivariate time series data into image data that represents an image. The image-based prediction service may then process the image data using one or more image processing models to generate a prediction (e.g., machine failure within a week). The image-based prediction service may send the prediction to the remote network. ## Claims ### Claim 1 A system, comprising: one or more processors; and one or more memories, wherein the one or more memories have stored thereon instructions, which when executed by the one or more processors, cause the one or more processors to implement an image-based prediction service for a plurality of clients of a provider network, wherein the image-based prediction service is configured to, for a given client: receive, from a remote network of the client, multivariate time series data, wherein the multivariate time series data comprises different time series of data obtained from respective data sources over a time period; transform the multivariate time series data into image data, wherein the image data comprises a set of data representing an image; process the image data using one or more image processing models to generate prediction associated with the remote network; and send the prediction to the remote network or to another endpoint. ### Claim 2 (depends on 1) The system as recited in claim 1, wherein to transform the multivariate time series data, the image-based prediction service is configured to: transform the multivariate time series data into image data comprising a set of data representing a heat map, wherein the set of data comprises different values that represent one or more of different colors or different intensities at different locations of a heat map, wherein at least some of the values are lower than a portion of the values and higher than another portion of the values. ### Claim 3 (depends on 1) The system as recited in claim 1, wherein to transform the multivariate time series data into image data, the image-based prediction service is configured to: transform each of the time series of data into a subset of the image data to generate a plurality of subsets of the image data, wherein a given subset of the image data comprises different values that represent varying amounts of one or more of color or intensity; and generate the set of data representing the image, based on the plurality of subsets of the image data. ### Claim 4 (depends on 1) The system as recited in claim 1, wherein to transform the multivariate time series data, the image-based prediction service is configured to: analyze the multivariate time series data; and based at least on the analysis, select, from among a plurality of available types of image data, a type of image data to transform the multivariate time series data into; and transform the multivariate time series data into the image data of the selected type. ### Claim 5 (depends on 1) The system as recited in claim 1, wherein the image-based prediction service is further configured to: select, from among a plurality of available image processing models configured to process different types of image data, the one or more image processing models to process the image data based on the type of the image data to be processed. ### Claim 6 A method, comprising: performing, by one or more computing devices: receiving multivariate time series data, wherein the multivariate time series data comprises different time series of data obtained from respective data sources over a time period; transforming the multivariate time series data into image data, wherein the image data comprises a set of data representing an image; processing the image data using one or more image processing models to generate a prediction; and generating a response based on the prediction. ### Claim 7 (depends on 6) The method as recited in claim 6, wherein the set of data representing the image comprises different values that represent one or more of different colors or different intensities at different locations of a heat map, wherein at least some of the values are lower than a portion of the values and higher than another portion of the values. ### Claim 8 (depends on 6) The method as recited in claim 6, wherein transforming the multivariate time series data comprises: transforming each of the time series of data into a subset of the image data to generate a plurality of subsets of the image data, wherein a given subset of the image data comprises different values that represent varying amounts of one or more of color or intensity; and generating the set of data representing the image, based on the plurality of subsets of the image data. ### Claim 9 (depends on 8) The method as recited in claim 8, wherein each of the subsets of the image data represents a different spectrogram or a different layer of a lasagna plot. ### Claim 10 (depends on 8) The method as recited in claim 8, wherein each of the subsets of the image data represents one of a plurality of layers of a lasagna plot, and wherein processing the image data using one or more image processing models comprises: applying one dimensional convolution along each of the subsets of the image data that represent a layer of the lasagna plot. ### Claim 11 (depends on 8) The method as recited in claim 8, wherein each of the subsets of the image data represents one of a plurality of layers of a lasagna plot that are arranged in a particular order, and further comprising: analyzing one or more sets of image data that each represent a lasagna plot that was based on data obtained from the respective data sources over different time periods, wherein the analysis comprises analyzing at least the set of data representing the lasagna plot; and based on the analysis, transforming data subsequently received from the respective data sources into another set of image data that represents another lasagna plot with a plurality of layers that are arranged in a different order than the particular order. ### Claim 12 (depends on 8) The method as recited in claim 8, wherein each of the subsets of the image data represents one of a plurality of layers of a lasagna plot, and further comprising: analyzing one or more sets of image data that each represent a lasagna plot that was based on data obtained from the respective data sources over different time periods, wherein the analysis comprises analyzing at least the set of data representing the lasagna plot; and based on the analysis, transforming data subsequently received from the respective data sources into another set of image data that represents another lasagna plot with a smaller number of layers than the lasagna plot. ### Claim 13 (depends on 6) The method as recited in claim 6, further comprising: for individual ones of the different time series of data, determine one or more of a type of data collected for the time series of data or a type of sensor used to collect data for the time series of data; based at least on the determination, selecting, from among a plurality of available types of image data, a type of image data to transform the multivariate time series data into; and transforming the multivariate time series data into the image data of the selected type. ### Claim 14 (depends on 1) The method as recited in claim 1, further comprising: obtaining training data, wherein the training data comprises different image data and corresponding predictions, wherein the predictions comprise one or more of: failure of a device within a particular period of time, or a type of failure of the device; and training the one or more image processing models based on the training data. ### Claim 15 One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more processors cause the one or more processors to: receive multivariate time series data, wherein the multivariate time series data comprises different time series of data obtained from respective data sources over a time period; transform the multivariate time series data into image data, wherein the image data comprises a set of data representing an image; process the image data using one or more image processing models to generate a prediction; and send the prediction to an endpoint. ### Claim 16 (depends on 15) The one or more storage media as recited in claim 15, wherein the set of data representing the image comprises different values that represent one or more of different colors or different intensities at different locations of a heat map, wherein at least some of the values are lower than a portion of the values and higher than another portion of the values. ### Claim 17 (depends on 15) The one or more storage media as recited in claim 15, wherein to transform the multivariate time series data into image data, the program instructions when executed on or across the one or more processors cause the one or more processors to: transform each of the time series of data into a subset of the image data to generate a plurality of subsets of the image data, wherein a given subset of the image data comprises different values that represent varying amounts of one or more of color or intensity; and generate the image data based on the plurality of subsets of the image data. ### Claim 18 (depends on 15) The one or more storage media as recited in claim 15, wherein the different time series of data comprise sensor measurements associated with a device over the time period, and further comprising program instructions that when executed on or across the one or more processors cause the one or more processors to: receive additional device data indicating one or more of a configuration or a location of the device; and wherein to process the image data, the program instructions when executed on or across the one or more processors cause the one or more processors to: process the image data and the additional device data using the one or more image processing models to generate the prediction. ### Claim 19 (depends on 15) The one or more storage media as recited in claim 15, wherein the different time series of data comprise different types of data obtained from different sensors associated with a device, and wherein the prediction indicates one or more of: failure of a device within a particular period of time, or a type of failure of the device. ### Claim 20 (depends on 15) The one or more storage media as recited in claim 15, and further comprising program instructions that when executed on or across the one or more processors cause the one or more processors to: analyzing one or more sets of image data and corresponding predictions, wherein the one or more sets of image data are based on time series of data obtained from the respective data sources over different time periods, and wherein the analysis comprises analyzing at least the set of image data and the corresponding prediction; and based on the analysis, remove one or more time series of data from additional multivariate time series data subsequently received from the respective data sources to generate filtered multivariate time series data; transform the filtered multivariate time series data into additional image data, wherein the additional image data comprises a set of data representing an additional image; and process the additional image data using the one or more image processing models to generate an additional prediction.