# US 12,182,982 B1 · Optical and other sensory processing of complex objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,182,982 B1
- Document ID: US12182982B1
- Status: Granted
- Kind: B1
- Application number: 18/644,261
- Filed: 2024-04-24
- Granted: 2024-12-31
- Earliest claimed: 2024-01-08
- Assignees: Calodar Ltd.
- Applicants: Calodar Ltd.
- Inventors: Joseph Swed
- Web page: https://patentgrep.com/patent/US12182982B1

## CPC

- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06T 7/11: Region-based segmentation
- G06T 7/62: of area, perimeter, diameter or volume
- G06V 10/7753: {Incorporation of unlabelled data, e.g. multiple instance learning [MIL]}
- G06V 10/82: using neural networks
- G06V 20/68: Food, e.g. fruit or vegetables
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30128: Food products
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods for optical and other sensory analysis of nutritional and other complex objects are disclosed. For example, techniques may include capturing an RGB-D image of a food using an integrated camera; inputting the RGB-D image into an instance detection network configured to detect food items; segmenting a plurality of food items from the RGB-D image into a plurality of masks, the plurality of masks representing individual food items; classifying a particular food item among the individual food items using a multimodal large language model; estimating a volume of the particular food item by overlaying an RGB image associated with the RGB-D image with a depth-map to create a point cloud; and estimating the calories of the particular food item using the estimated volume and a nutritional database.

## Claims

### Claim 1

A non-transitory computer-readable medium configured to use an integrated camera to scan food, the non-transitory computer-readable medium storing executable code configured to perform operations comprising:
capturing an RGB image of the food using the integrated camera;
inputting the RGB image into an instance detection network configured to detect food items, the instance detection network having been trained on a plurality of reference food items using a neural network;
segmenting a plurality of food items from the RGB image into a plurality of masks, the plurality of masks representing individual food items;
classifying a particular food item among the individual food items using a multimodal large language model;
estimating a volume of the particular food item by overlaying the RGB image with a depth-map to create a point cloud, wherein the depth map is created by monocular depth estimation; and
estimating the calories of the particular food item using the estimated volume and a nutritional database.

### Claim 2 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the operations further comprise comparing a first RGB image before intake to a second RGB image after intake, and generating an intake estimate associated with the particular food item.

### Claim 3 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the operations further comprise comparing a first RGB image after intake to a synthetic image.

### Claim 4 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the operations further comprise pairing an unlabeled image with a verified mask from the plurality of masks representing an individual food item to optimize a trained classifier.

### Claim 5 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein inputting the RGB image into the instance detection network comprises creating a square around the individual food item.

### Claim 6 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein inputting the RGB image into the instance detection network comprises creating a mask representing multiple food items.

### Claim 7 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the point cloud is a synthetic point cloud capable of being captured and stored for analysis.

### Claim 8 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the operations further comprise:
capturing a video segment using the integrated camera; and
extracting a plurality of image frames from the video segment.

### Claim 9 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein the extracted plurality of image frames includes the captured RGB image.

### Claim 10

A computer-implemented method for scanning food, comprising:
capturing an RGB image of the food using an integrated camera;
inputting the RGB image into an instance detection network configured to detect food items;
segmenting a plurality of food items from the RGB image into a plurality of masks, the plurality of masks representing individual food items;
classifying a particular food item among the individual food items using a multimodal large language model;
estimating a volume of the particular food item by overlaying the RGB image with a depth-map to create a point cloud, wherein the depth map is created by monocular depth estimation; and
estimating the calories of the particular food item using the estimated volume and a nutritional database.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, further comprising:
comparing a first RGB image before food intake to a second RGB image after food intake, and
generating, based on the comparison, a food intake estimate associated with the particular food item.

### Claim 12 (depends on 10)

The computer-implemented method of claim 10, further comprising:
comparing a first RGB image after food intake to a synthetic image.

### Claim 13 (depends on 10)

The computer-implemented method of claim 10, further comprising:
pairing an unlabeled image with a verified mask from the plurality of masks representing an individual food item to update a trained classifier.

### Claim 14 (depends on 10)

The computer-implemented method of claim 10, wherein inputting the RGB image into the instance detection network includes at least one of:
creating a square around the individual food item; or
creating a mask representing multiple food items.

### Claim 15 (depends on 10)

The computer-implemented method of claim 10, further comprising:
creating training data for synthetic data generation through automatic dataset generation via an assets database; and
generating segmented outputs from the synthetic data generation across various combinations of food items, lighting conditions, and other physical environments.

### Claim 16 (depends on 15)

The computer-implemented method of claim 15, wherein the segmented outputs are used for volume and calorie estimation.

### Claim 17 (depends on 10)

The computer-implemented method of claim 10, further comprising:
capturing a video segment using the integrated camera; and
extracting a plurality of image frames from the video segment.

### Claim 18 (depends on 17)

The computer-implemented method of claim 17, wherein the extracted plurality of image frames includes the captured RGB image.

### Claim 19 (depends on 10)

The computer-implemented method of claim 10, wherein the monocular depth estimation comprises estimating a distance of a plurality of pixels in the RGB image from the integrated camera.

### Claim 20 (depends on 10)

The computer-implemented method of claim 10, wherein the monocular depth estimation comprises predicting depth information of the particular food item based on the RGB image.