Optimal learning rate selection through step sampling

A system for training a model, the system comprising:

• a processor;
• and
• a memory storing instructions that, when executed, cause the processor to:
• select an initial learning rate for the model;
• train the model based on the initial learning rate;
• determine a derivative of a loss of the initial learning rate based on a result of the training;
• and
• based on the derivative of the loss being greater than a predetermined derivative threshold:
• determine at least one point of interest based on the result of the training;
• select a subsequent learning rate based on the at least one point of interest;
• and
• train the model based on the subsequent learning rate.

A non-transitory computer-readable storage medium comprising instructions that, when executed, cause at least one processor to:

• select an initial learning rate for the model;
• train a model based on the initial learning rate;
• determine a derivative of a loss of the initial learning rate based on a result of the training;
• and
• based on the derivative of the loss being greater than a predetermined derivative threshold:
• determine at least one point of interest based on the result of the training;
• select a subsequent learning rate based on the at least one point of interest;
• and
• train the model based on the subsequent learning rate.