System and method for temperature sensing in cooking appliance with data fusion

An oven, comprising:

• a control unit comprising one or more processors and memory storing instructions;
• a display;
• and
• a cooking chamber, wherein the cooking chamber includes a first camera that includes a first array of thermal image sensors corresponding to a first resolution, a second camera that includes a second array of color image sensors corresponding to a second resolution higher than the first resolution, one or more heating elements, and a food support platform, and wherein the instructions, when executed by the one or more processors, cause the processors to perform operations comprising:
• in accordance with instructions provided by the control unit, concurrently capturing a first image using the first camera and a second image using the second camera, wherein each of the first image and the second image includes respective sets of pixels corresponding to one or more food items placed on the food support platform, wherein the first image is of the first resolution and captured from a first perspective relative to the one or more food items, and wherein the second image is of the second resolution and captured from a second perspective distinct from the first perspective relative to the one or more food items;
• rectifying the first image by using a difference between camera parameters for the first camera and camera parameters for the second camera, wherein the rectified first image is of the second resolution and from the second perspective relative to the one or more food items;
• identifying respective sets of pixels in the rectified first image that correspond to each of the one or more food items captured in the second image, including:
• performing instance segmentation on the rectified first image and the second image to identify one or more instances of the food items on the rectified first image and the second image, respectively;
• and
• performing edge detection on the identified instances of food items on the rectified first image and the second image to generate a first edge map and a second edge map, respectively;
• and
• in accordance with established calibration rules, establishing correlations between pixels of the second image to respective sets of pixels corresponding to the one or more food items identified in the first image, wherein the established calibration rules associate the first perspective to the second perspective and the first resolution to the second resolution;
• and
• generating a temperature map for the one or more food items with the second resolution in accordance with a correspondence between the pixels of the second image to the respective sets of pixels corresponding to the one or more food items identified in the first image, wherein the temperature map indicates temperature distribution on the one or more food items.

A non-transitory computer-readable storage medium storing instructions, the instructions, when executed by one or more processors of an oven, causes the processors to perform operations, wherein:

• the oven includes a display and a cooking chamber,
• the cooking chamber includes a first camera that includes a first array of thermal image sensors corresponding to a first resolution, a second camera that includes a second array of color image sensors corresponding to a second resolution higher than the first resolution, one or more heating elements, and a food support platform, and
• the operations include:
• in accordance with instructions provided by a control unit, concurrently capturing a first image using the first camera and a second image using the second camera, wherein each of the first image and the second image includes respective sets of pixels corresponding to one or more food items placed on the food support platform, wherein the first image is of the first resolution and captured from a first perspective relative to the one or more food items, and wherein the second image is of the second resolution and captured from a second perspective distinct from the first perspective relative to the one or more food items;
• rectifying the first image by using a difference between camera parameters for the first camera and camera parameters for the second camera, wherein the rectified first image is of the second resolution and from the second perspective relative to the one or more food items;
• identifying respective sets of pixels in the rectified first image that correspond to each of the one or more food items captured in the second image, including:
• performing instance segmentation on the rectified first image and the second image to identify one or more instances of the food items on the rectified first image and the second image, respectively;
• and
• performing edge detection on the identified instances of food items on the rectified first image and the second image to generate a first edge map and a second edge map, respectively;
• and
• in accordance with established calibration rules, establishing correlations between pixels of the second image to respective sets of pixels corresponding to the one or more food items identified in the first image, wherein the established calibration rules associate the first perspective to the second perspective and the first resolution to the second resolution;
• and
• generating a temperature map for the one or more food items with the second resolution in accordance with a correspondence between the pixels of the second image to the respective sets of pixels corresponding to the one or more food items identified in the first image, wherein the temperature map indicates temperature distribution on the one or more food items.