US 12,608,787 B2Granted

Machine learning-based assessment of food item quality

AssigneeApeel Technology, Inc.

InventorsOhad Michel, Richard Pattison, Tim Rappold, Sahana Venkatesh, Pooyan Abouzar, Saurabh Chatterjee, Jeffrey Fun-Shen Gau

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

Described herein are systems and methods for determining quality levels for food items using image data, such as time lapse RGB, hyperspectral, thermal, and/or multispectral images. The method can include receiving, from imaging devices, image data of food items, performing object detection on the image data to identify a bounding box around each food item, and identifying a quality level of each food item by applying trained models to the bounding boxes. The models were trained using image training data of other food items that was annotated based on previous identifications of a first portion of the other food items as having poor quality features and a second portion as having good quality features. The other food items and the food items are a same type. The method also includes determining, for each food item, a quality level score based on the identified quality level of the food item.

Claims24

Dependent claims

A method for determining quality levels for food items using image data, the method comprising:

receiving, by a computing system and from an imaging device, image data of food items;
performing, by the computing system, object detection on the image data to identify a bounding box around each of the food items in the image data;
determining, by the computing system and based on the bounding box around each of the food items in the image data, a grid structure for the image data, wherein each bounding box is assigned a grid index in the grid structure, wherein the grid index is used to identify the food item in a data store;
identifying, by the computing system and for each of the food items, a quality level of the food item by applying a plurality of trained models to bounding box portions of the image data that contains each of the food items, wherein each of the trained models were trained using image training data of other food items, the image training data being annotated based on previous identifications of a first portion of the other food items as having poor quality features and a second portion of the other food items as having good quality features, wherein the other food items are of a same food type as the food items, wherein the plurality of trained models include a first trained model trained to identify a first quality feature and a second trained model trained to identify a second quality feature that is different than the first quality feature, wherein identifying, by the computing system and for each of the food items, the quality level of the food item comprises:
calibrating colors of the image data to maintain color consistency in the image data;
determining, based on applying a color analyzer trained model to the calibrated bounding box portions of the image data, a median Euclidean distance between a color of the food item in the bounding box portions of the image data and a reference color for the food item, wherein the reference color is associated with a preferred quality level of the food item;
determining whether the median Euclidean distance exceeds a threshold value;
in response to determining that the median Euclidean distance exceeds the threshold value, identifying the quality level of the food item as poor quality;
and
in response to determining that the median Euclidean distance is less than the threshold value, identifying the quality level of the food item as good quality;
determining, by the computing system and for each of the food items, a quality level score for the food item based on the identified quality level of the food item;
and
storing, by the computing system and for each of the food items, (i) the bounding box portions of the image data that contains the food items, (ii) the grid index, (iii) the identified quality level of the food item, and (iv) the determined quality level score for the food item in the data store.

10 more elements

A method for determining quality levels for food items using image data, the method comprising, receiving, by a computing system and from an imaging device, image data of food items;

performing, by the computing system, object detection on the image data to identify a bounding box around each of the food items in the image data;
determining, by the computing system and based on the bounding box around each of the food items in the image data, a grid structure for the image data, wherein each bounding box is assigned a grid index in the grid structure, wherein the grid index is used to identify the food item in a data store;
identifying, by the computing system and for each of the food items, a quality level of the food item by applying a plurality of trained models to bounding box portions of the image data that contains each of the food items, wherein each of the trained models were trained using image training data of other food items, the image training data being annotated based on previous identifications of a first portion of the other food items as having poor quality features and a second portion of the other food items as having good quality features, wherein the other food items are of a same food type as the food items, wherein the plurality of trained models include a first trained model trained to identify a first quality feature and a second trained model trained to identify a second quality feature that is different than the first quality feature, wherein the quality level score for the food item is an overall quality metric for the food item and wherein the overall quality metric is determined by:
accessing, from a data store, rule-based mappings of (i) different ranges of values for scores corresponding to identification of the first and second quality features to (ii) enumerated categories of food item quality, wherein the enumerated categories of food item quality include at least one of salable, unsalable, edible, nonedible, good quality, bad quality, and okay quality;
iteratively determining, for each of the first and second quality features, whether respective rule-based mappings are met;
identifying, based on the respective rule-based mappings being met, an enumerated category of food item quality for the food item;
and
assigning the identified category to the food item as the overall quality metric for the food item;
determining, by the computing system and for each of the food items, a quality level score for the food item based on the identified quality level of the food item;
and
storing, by the computing system and for each of the food items, (i) the bounding box portions of the image data that contains the food items, (ii) the grid index, (iii) the identified quality level of the food item, and (iv) the determined quality level score for the food item in the data store.

8 more elements

A method for determining quality levels for food items using image data, the method comprising:

receiving, by a computing system and from an imaging device, image data of food items;
performing, by the computing system, object detection on the image data to identify a bounding box around each of the food items in the image data;
determining, by the computing system and based on the bounding box around each of the food items in the image data, a grid structure for the image data, wherein each bounding box is assigned a grid index in the grid structure, wherein the grid index is used to identify the food item in a data store;
identifying, by the computing system and for each of the food items, a quality level of the food item by applying a plurality of trained models to bounding box portions of the image data that contains each of the food items, wherein each of the trained models were trained using image training data of other food items, the image training data being annotated based on previous identifications of a first portion of the other food items as having poor quality features and a second portion of the other food items as having good quality features, wherein the other food items are of a same food type as the food items, wherein the plurality of trained models include a first trained model trained to identify a first quality feature and a second trained model trained to identify a second quality feature that is different than the first quality feature, wherein identifying, by the computing system and for each of the food items, the quality level of the food item comprises:
identifying, based on applying an internal quality trained model to the bounding box portions of the image data that contain each of the food items, internal quality defects in the food item, wherein the internal quality trained model was trained using image training data of other food items being previously annotated as having no internal defects, diffuse browning, vascular browning, flesh adhesion, internal rot, stem rot, and tissue breakdown;
and
determining, based on the internal quality defects, an overall quality level score of the food item, wherein a score of “good” indicates no internal quality defects in the food item, a score of “okay” indicates some internal quality defects in the food item, and a score of “bad” indicates most internal quality defects in the food item;
determining, by the computing system and for each of the food items, a quality level score for the food item based on the identified quality level of the food item;
and
storing, by the computing system and for each of the food items, (i) the bounding box portions of the image data that contains the food items, (ii) the grid index, (iii) the identified quality level of the food item, and (iv) the determined quality level score for the food item in the data store.

7 more elements

A method for determining quality levels for food items using image data, the method comprising

receiving, by a computing system and from an imaging device, image data of food items;
performing, by the computing system, object detection on the image data to identify a bounding box around each of the food items in the image data;
determining, by the computing system and based on the bounding box around each of the food items in the image data, a grid structure for the image data, wherein each bounding box is assigned a grid index in the grid structure, wherein the grid index is used to identify the food item in a data store;
identifying, by the computing system and for each of the food items, a quality level of the food item by applying a plurality of trained models to bounding box portions of the image data that contains each of the food items, wherein each of the trained models were trained using image training data of other food items, the image training data being annotated based on previous identifications of a first portion of the other food items as having poor quality features and a second portion of the other food items as having good quality features, wherein the other food items are of a same food type as the food items, wherein the plurality of trained models include a first trained model trained to identify a first quality feature and a second trained model trained to identify a second quality feature that is different than the first quality feature, wherein identifying, by the computing system and for each of the food items, the quality level of the food item comprises:
identifying, based on applying a product identifier trained model to the bounding box portions of the image data that contain each of the food items, a product label on the food item;
and
decoding, using optical character recognition (OCR), at least one of text and a product identifier on the product label,
wherein identifying, based on the decoded at least one of text and a product identifier on product label, the quality level score of the food item as a string value that includes at least one of the decoded text and the product identifier;
determining, by the computing system and for each of the food items, a quality level score for the food item based on the identified quality level of the food item;
and
storing, by the computing system and for each of the food items, (i) the bounding box portions of the image data that contains the food items, (ii) the grid index, (iii) the identified quality level of the food item, and (iv) the determined quality level score for the food item in the data store.

8 more elements