Lag feature

A method of analyzing the effectiveness of a machine learning model, the method comprising:

• executing a plurality of test procedures of a machine learning model comprising a plurality of input features and a lag feature associated with each of the input features, wherein:
• executing each test procedure of the plurality of test procedures comprises:
• querying a plurality of data sources for historical data records associated with the input features;
• simulating effects of lag on a performance of the machine learning model when operated under real-world conditions by using each lag feature to restrict a query for historical data records from at least one of the data sources to only historical data records available at a predetermined time prior to a time of the test procedure of the machine learning model, wherein the predetermined time is based on a known amount of lag associated with the at least one of the data sources;
• receiving the historical data records from the plurality of data sources at the machine learning model;
• and
• generating outputs of the machine learning model;
• and
• analyzing the outputs of each of the plurality of test procedures to determine how impactful each of the different predetermined times of the plurality of lag features were to a result of the machine learning model, wherein:
• analyzing the outputs comprises generating one or more performance metrics of the machine learning model that are dependent on lag attributes based on the outputs of the machine learning model;
• and
• the one or more performance metrics comprise a performance decrement curve that illustrates how the performance of the machine learning model is affected with different amounts of lag built in for one or more of the plurality of input features.

A system, comprising:

• one or more computing devices;
• and
• memory storing instructions, the instructions being executable by the one or more computing devices, wherein the one or more computing devices are configured to:
• executing a test procedure of a machine learning model comprising a plurality of input features and a lag feature associated with at least one of the plurality of input features, wherein executing the test procedure comprises:
• querying one or more data sources for historical data records associated with the plurality of input features;
• simulating effects of lag on a performance of the machine learning model when operated under real-world conditions by using the lag feature to restrict a query for historical data records from at least one of the data sources to only historical data records available at a predetermined time prior to a time of the test procedure of the machine learning model, wherein the predetermined time is based on a known amount of lag associated with the at least one of the data sources;
• receiving the historical data records from the one or more data sources at the machine learning model;
• generating an output of the machine learning model;
• and
• generating one or more performance metrics of the machine learning model that are dependent on lag attributes based on the output of the machine learning model, wherein the one or more performance metrics comprise a performance decrement curve that illustrates how the performance of the machine learning model is affected with different amounts of lag built in for the at least one of the plurality of input features.