Real-time automated cooking cycles system using computer vision and deep learning

A smart oven for controlling a heating process, comprising:

• one or more sensors;
• a heating system;
• and
• a processor, programmed to:
• recognize a food class of a food item,
• define a target doneness score for the food item based on the food class and a desired doneness level,
• capture sensor data of the food item over time during a cooking cycle, the sensor data including spatial sensor data and non-spatial data, the spatial sensor data including a sequence of images providing spatial and temporal information with respect to the food item, the non-spatial data specifying information over time without associated coordinates with respect to the food item,
• applying the sensor data including the sequence of images as an input to a recurrent model trained to learn spatio-temporal relationships to determine a current doneness score of the food item during the cooking cycle, wherein the recurrent model includes a plurality of recurrent units that utilize time-distributed layers to capture the spatio-temporal relationships, wherein a first of the plurality of recurrent units is a convolutional recurrent unit configured to receive spatial inputs derived at least from the sequence of images, and a second of the recurrent units is a non-convolutional recurrent unit configured to receive non-spatial inputs derived from the non-spatial data, wherein outputs of the convolutional recurrent unit and the non-convolutional recurrent unit are combined by the recurrent model to determine the current doneness score, and
• utilize the current doneness score and the target doneness score to control the heating system to cook the food item.

A non-transitory computer readable medium comprising instructions controlling a heating process of a smart oven having one or more sensors, a heating system, and a processor that, when executed by the processor of the smart oven, cause the smart oven to perform operations including to:

• recognize a food class of a food item;
• define a target doneness score for the food item based on the food class and a desired doneness level;
• capture sensor data of the food item over time during a cooking cycle, the sensor data including spatial sensor data and non-spatial data, the spatial sensor data including a sequence of images providing spatial and temporal information with respect to the food item, the non-spatial data specifying information over time without associated coordinates with respect to the food item;
• apply the sensor data including the sequence of images as an input to a recurrent model trained to learn spatio-temporal relationships to determine a current doneness score of the food item during the cooking cycle, wherein the recurrent model includes a plurality of recurrent units that utilize time-distributed layers to capture the spatio-temporal relationships, wherein a first of the plurality of recurrent units is a convolutional recurrent unit configured to receive spatial inputs derived at least from the sequence of images, and a second of the recurrent units is a non-convolutional recurrent unit configured to receive non-spatial inputs derived from the non-spatial data, wherein outputs of the convolutional recurrent unit and the non-convolutional recurrent unit are combined by the recurrent model to determine the current doneness score;
• and
• utilize the current doneness score and the target doneness score to control the heating system to cook the food item.