# US 2021/0279855 A1 · Neural network-based systems and computer-implemented methods for identifying and/or evaluating one or more food items present in a visual input > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0279855 A1 - Document ID: US20210279855A1 - Status: Published application - Kind: A1 - Application number: 17/179,771 - Filed: 2021-02-19 - Published: 2021-09-09 - Earliest claimed: 2018-08-22 - Assignees: Florent Technologies LLC - Applicants: Florent Technologies LLC - Inventors: Leonid Aleksandrovich Shangin; Ilya Evgenievich Ivanov; William Christopher Wynne - Web page: https://patentgrep.com/patent/US20210279855A1 ## CPC - G06Q 10/0639: Performance analysis of employees; Performance analysis of enterprise or organisation operations - G06F 18/241: relating to the classification model, e.g. parametric or non-parametric approaches - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06Q 10/06: Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling - G06Q 50/12: Hotels or restaurants - G06T 7/0004: {Industrial image inspection} - G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) - G06N 20/10: using kernel methods, e.g. support vector machines [SVM] - G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform - G06T 2207/30128: Food products - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract In some embodiments, the present invention provides for an exemplary inventive convolutional neural network-based and computer-implemented method for identifying and evaluating pizza, including: collecting video input representative of the food and received from at least one camera, applying a first CNN to select, from the video input, a set of best pizza containing video frames of a particular pizza from the plurality of pizza containing video frames; applying the first CNN to identify a best pizza containing image from the set, to localize at least one pizza portion of the particular pizza in the identified best pizza containing image, and to determine a type of the pizza of the particular pizza from the identified best pizza containing image; applying a second CNN to determine a map of pizza components of the particular pizza by automatically performing pizza image segmentation and to automatically score the particular pizza based on the determined map of pizza components. ## Claims ### Claim 1 A method, comprising: receiving, by a processor, a continuous video stream from at least one camera position over a table configured to receive prepared pizzas; collecting, by the processor, a plurality of pizza containing video frames of a particular pizza from the video stream; applying, by the processor, a first CNN to select a set of best pizza containing video frames of the particular pizza from the plurality of pizza containing video frames; applying, by the processor, a first CNN to identify a best pizza containing image of the particular pizza from the set of best pizza containing video frames; applying, by the processor, the first CNN to localize at least one pizza portion of the particular pizza in the identified best pizza containing image; applying, by the processor, the first CNN to determine a type of the pizza of the particular pizza from the identified best pizza containing image; applying, by the processor, a second CNN to determine a map of pizza components of the particular pizza by automatically performing pizza image segmentation of the pizza portion based on at least the type of the pizza; and applying, by the processor, the second CNN to automatically score the particular pizza based on the determined map of pizza components, comprising: dividing, by the processor, the pizza portion of the identified best image into a plurality of slices; grading, by the processor, one of the plurality of slices of the particular pizza; repeating, by the processor, the grading step to grade the remaining slices of the plurality of slices; and determining, by the processor, a final score of the particular pizza based on the grading of the plurality of slices. ### Claim 2 (depends on 1) The method of claim 1, wherein the video frames of the video stream are categorized into cases comprising: i) a first case for images that have no pizza present; ii) a second case for images in which a pizza is present and off-centered; iii) a third case for images in which the pizza is present and centered, and a pizza image has a resolution quality of X; iv) a fourth case for images in which the pizza is present and centered, and the pizza image has the resolution quality of Y, where Y is better than X; v) a fifth case for images in which the pizza is present, centered, and a first type, and the pizza image has a desired resolution quality; and v) a sixth case for images in which the pizza is present, centered, and a second type, and the pizza image has a desired resolution quality. ### Claim 3 (depends on 1) The method of claim 1, wherein selecting the set of best pizza containing video frames from the pizza containing video frames further comprises discounting each pizza containing video frame that has at least one of a motion blur or defocus blur. ### Claim 4 (depends on 1) The method of claim 1, wherein the number of the plurality of slices is 8. ### Claim 5 (depends on 1) The method of claim 1, wherein the second CNN has a contraction path and an expansion path. ### Claim 6 (depends on 5) The method of claim 5, wherein the contraction path comprises a plurality of convolution and activation layers. ### Claim 7 (depends on 6) The method of claim 6, wherein the contraction path further comprises a subsampling and batch normalization layer after a first convolution and activation layer. ### Claim 8 (depends on 6) The method of claim 6, wherein the contraction path further comprises a rectified linear unit (ReLU) layer and a pooling layer following each convolution and activation layer before proceeding to a subsequent convolution and activation layer. ### Claim 9 (depends on 5) The method of claim 5, wherein the expansive path comprises a sequence of up-convolutions and concatenations configured to combine feature spatial information with a predetermined resolution features from the contracting path. ### Claim 10 (depends on 1) The method of claim 1, wherein the applying the first CNN to localize the at least one pizza portion of the particular pizza in the identified best pizza containing image further comprises: defining a bounding box; and utilizing one or more pre-determined binary masks. ### Claim 11 A system, comprising: at least one image capturing device; a non-transitory storage memory; one or more processors; and computer program code stored on the non-transitory storage memory and, when executed by the one or more processors, causes the one or more processors to: receiving a continuous video stream from at least one camera position over a table configured to receive prepared pizzas; collecting a plurality of pizza containing video frames of a particular pizza from the video stream; applying a first CNN to select a set of best pizza containing video frames of the particular pizza from the plurality of pizza containing video frames; applying the first CNN to identify a best pizza containing image of the particular pizza from the set of best pizza containing video frames; applying the first CNN to localize at least one pizza portion of the particular pizza in the identified best pizza containing image; applying the first CNN to determine a type of the pizza of the particular pizza from the identified best pizza containing image; applying a second CNN to determine a map of pizza components of the particular pizza by automatically performing pizza image segmentation of the pizza portion based on at least the type of the pizza; and applying the second CNN to automatically score the particular pizza based on the determined map of pizza components, comprising: dividing the pizza portion of the identified best image into a plurality of slices; grading one of the plurality of slices of the particular pizza; repeating the grading step to grade the remaining slices of the plurality of slices; and determining a final score of the particular pizza based on the grading of the plurality of slices. ### Claim 12 A non-transitory computer-readable storage medium comprising processor-executable instructions for: receiving a continuous video stream from at least one camera position over a table configured to receive prepared pizzas; collecting a plurality of pizza containing video frames of a particular pizza from the video stream; applying a first CNN to select a set of best pizza containing video frames of the particular pizza from the plurality of pizza containing video frames; applying the first CNN to identify a best pizza containing image of the particular pizza from the set of best pizza containing video frames; applying the first CNN to localize at least one pizza portion of the particular pizza in the identified best pizza containing image; applying the first CNN to determine a type of the pizza of the particular pizza from the identified best pizza containing image; applying a second CNN to determine a map of pizza components of the particular pizza by automatically performing pizza image segmentation of the pizza portion based on at least the type of the pizza; and applying the second CNN to automatically score the particular pizza based on the determined map of pizza components, comprising: dividing the pizza portion of the identified best image into a plurality of slices; grading one of the plurality of slices of the particular pizza; repeating the grading step to grade the remaining slices of the plurality of slices; and determining a final score of the particular pizza based on the grading of the plurality of slices.