Methods and systems to extract signals from large and imperfect datasets

A system for improving processing of data from a plurality of data sources by extracting signals from data sets, the system comprising:

• at least one hardware processor that is configured to:
• retrieve the data from the plurality of data sources;
• organize the retrieved data into chronological order
• detect a missing observation in the retrieved and chronologically organized data;
• estimate a value of the missing observation to obtain estimated data by performing at least one of:
• applying a Kalman Filter, applying an autoregressive integrated moving average, applying an arithmetic extrapolation, applying an arithmetic interpolation, applying a logarithmic extrapolation, applying a logarithmic interpolation, applying an arbitrary method, using a linear regression, and using a nonlinear regression;
• generate a correlation matrix or a covariance matrix based on the retrieved data and the estimated data;
• perform an eigenvalue decomposition on the correlation matrix or the covariance matrix to determine a set of eigenvector loadings;
• and
• convert a result of the eigenvalue decomposition into at least one indicator of a common signal among the retrieved data, wherein converting the result of the eigenvalue decomposition into the at least one indicator of the common signal comprises applying at least one loading from the set of eigenvector loadings as a weight to at least one corresponding observation in the retrieved data from the plurality of data sources and/or the estimated data.

A non-transitory computer-readable medium having thereon computer-executable instructions that, when executed by at least one processor, cause the at least one processor to rapidly perform a method for improving processing of data from a plurality of data sources by extracting signals from data sets, the method comprising:

• retrieving data, by the at least one processor, from the plurality of data sources;
• organizing the retrieved data into chronological order;
• detecting a missing observation in the retrieved and chronologically organized data;
• estimating a value of the missing observation to obtain estimated data by performing at least one of:
• applying a Kalman Filter, applying an autoregressive integrated moving average, applying an arithmetic extrapolation, applying an arithmetic interpolation, applying a logarithmic extrapolation, applying an logarithmic:
• interpolation, applying an arbitrary method, using a linear regression, and using a nonlinear regression;
• generating, by the at least one processor, a correlation matrix or a covariance matrix based on the retrieved data and the estimated data;
• performing, by the at least one processor, an eigenvalue decomposition on the correlation matrix or the covariance matrix to determine a set of eigenvector loadings;
• and
• converting, by the at least one processor, a result of the eigenvalue decomposition into at least one indicator of a common signal among the retrieved data, wherein converting the result of the eigenvalue decomposition into the at least one indicator of the common signal comprises applying at least one loading from the set of eigenvector loadings to at least one corresponding observation in the retrieved data from the plurality of data sources and/or the estimated data.

A method for improving processing of data from a plurality of data sources by extracting signals from data sets, the method comprising:

• retrieving, by at least one processor, the data from the plurality of data sources;
• generating, by the at least one processor, a correlation matrix or a covariance matrix based on the retrieved data;
• performing, by the at least one processor, an eigenvalue decomposition on the correlation matrix or the covariance matrix to determine a set of eigenvector loadings;
• converting, by the at least one processor, a result of the eigenvalue decomposition into at least one indicator of a common signal among the retrieved data, wherein converting the result of the eigenvalue decomposition into the at least one indicator of the common signal comprises applying at least one loading from the set of eigenvector loadings as a weight to at least one corresponding observation in the retrieved data from the plurality of data sources;
• and
• recursively repeating the retrieving, the generating, the performing, and the converting, wherein the recursively repeating further comprises recursively generating a data matrix of the retrieved data from the plurality of data sources, wherein the data matrix has n rows corresponding to n observation periods and k columns corresponding to k data sources, wherein recursively generating the data matrix comprises generating a first correlation matrix or covariance matrix based on the n-row and k-column data matrix to begin a recursive eigenvalue decomposition, successively expanding the data matrix to contain i additional rows in each recursion, and generating another correlation matrix or covariance matrix in each recursion based on the expanded data matrix, wherein n, i and k are integers.

A method for improving processing of data from a plurality of data sources by extracting signals from data sets, the method comprising:

• retrieving, by at least one processor, the data from the plurality of data sources;
• generating, by the at least one processor, a correlation matrix or a covariance matrix based on the retrieved data;
• performing, by the at least one processor, an eigenvalue decomposition on the correlation matrix or the covariance matrix to determine a set of eigenvector loadings;
• and
• converting, by the at least one processor, a result of the eigenvalue decomposition into at least one indicator of a common signal among the retrieved data, wherein converting the result of the eigenvalue decomposition into the at least one indicator of the common signal comprises applying at least one loading from the set of eigenvector loadings as a weight to at least one corresponding observation in the retrieved data from the plurality of data sources;
• the method further comprising recursively repeating the retrieving, the generating, the performing, and the converting, wherein the recursively repeating comprises iteratively generating a data matrix of the retrieved data from the plurality:
• of data sources, wherein the data matrix has n rows corresponding to n observation periods and k columns corresponding to k data. sources, wherein iteratively generating the data matrix comprises generating a first correlation matrix or covariance matrix based on the n-row and k-column data matrix to begin a rolling eigenvalue decomposition, iteratively determining the data matrix to contain n rows in each iteration, and generating another correlation matrix or covariance matrix in each iteration based on the generated n-row data matrix, wherein n is an integer.