System and Method for Collecting and Annotating Cooking Images for Training Smart Cooking Appliances

A method of collecting and annotating training data, comprising:

• at a computing system that is communicably coupled with a cooking appliance and configured to control one or more functions of the cooking appliance, wherein the cooking appliance includes a food support platform configured to support food items, one or more first sensors for capturing structured data, including temperature data corresponding to the food items during operation of the cooking appliance, one or more second sensors for capturing unstructured data, including image data corresponding to the food items during the operation of the cooking appliance, and one or more heating units that are configured to the heat food items placed on the food support platform during the operation of the cooking appliance:
• activating the one or more first sensors and the one or more second sensors to capture first cooking data during a first cooking process in the cooking appliance, including capturing a sequence of temperature values during the first cooking process, and a sequence of images during the first cooking process, wherein each of the sequence of temperature values and each of the sequence of images are recorded with respective timestamps;
• displaying, via a display generation component coupled to the computing system, at least a subset of the sequence of images in a first annotation user interface, wherein the first annotation user interface includes:
• each of the subset of the sequence of images displayed in accordance with a timeline of the first cooking process;
• and
• a plurality of input regions for specifying respective cooking progress levels of food items in the subset of the sequence of images;
• receiving first user input specifying the respective cooking progress levels of food items in the subset of the sequence of images;
• grouping the subset of images with corresponding temperature data and cooking progress levels specified by the first user input;
• and
• transmitting the subset of images with the corresponding temperature data and cooking progress levels to a server of training data corpus for training smart cooking appliances.

A non-transitory computer-readable storage medium, including instructions, the instructions, when executed by one or more processors of a computing system that is communicably coupled with a cooking appliance and configured to control one or more functions of the cooking appliance, cause the processors to perform operations, wherein the cooking appliance includes a food support platform configured to support food items, one or more first sensors for capturing structured data, including temperature data corresponding to the food items during operation of the cooking appliance, one or more second sensors for capturing unstructured data, including image data corresponding to the food items during the operation of the cooking appliance, and one or more heating units that are configured to the heat food items placed on the food support platform during the operation of the cooking appliance, and the operations comprises:

• activating the one or more first sensors and the one or more second sensors to capture first cooking data during a first cooking process in the cooking appliance, including capturing a sequence of temperature values during the first cooking process, and a sequence of images during the first cooking process, wherein each of the sequence of temperature values and each of the sequence of images are recorded with respective timestamps;
• displaying, via a display generation component coupled to the computing system, at least a subset of the sequence of images in a first annotation user interface, wherein the first annotation user interface includes:
• each of the subset of the sequence of images displayed in accordance with a timeline of the first cooking process;
• and
• a plurality of input regions for specifying respective cooking progress levels of food items in the subset of the sequence of images;
• receiving first user input specifying the respective cooking progress levels of food items in the subset of the sequence of images;
• grouping the subset of images with corresponding temperature data and cooking progress levels specified by the first user input;
• and
• transmitting the subset of images with the corresponding temperature data and cooking progress levels to a server of training data corpus for training smart cooking appliances.