Optical and other sensory processing of complex objects

A non-transitory computer-readable medium configured to use an integrated camera to scan food, the non-transitory computer-readable medium storing executable code configured to perform operations comprising:

• capturing an RGB image of the food using the integrated camera;
• inputting the RGB image into an instance detection network configured to detect food items, the instance detection network having been trained on a plurality of reference food items using a neural network;
• segmenting a plurality of food items from the RGB image into a plurality of masks, the plurality of masks representing individual food items;
• classifying a particular food item among the individual food items using a multimodal large language model;
• estimating a volume of the particular food item by overlaying the RGB image with a depth-map to create a point cloud, wherein the depth map is created by monocular depth estimation;
• and
• estimating the calories of the particular food item using the estimated volume and a nutritional database.