# US 12,216,962 B2 · Automatic quantitative food intake tracking

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,216,962 B2
- Document ID: US12216962B2
- Status: Granted
- Kind: B2
- Application number: 17/867,863
- Filed: 2022-07-19
- Published: 2024-01-25
- Granted: 2025-02-04
- Earliest claimed: 2022-07-19
- Assignees: Meta Platforms Technologies, LLC
- Applicants: Meta Platforms Technologies, LLC
- Inventors: Wen Li; John Rumsfeld; Charles Liam Goudge; Freddy Abnousi
- Web page: https://patentgrep.com/patent/US12216962B2

## CPC

- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G02B 27/017: {Head mounted}
- G06V 10/82: using neural networks
- G06V 20/68: Food, e.g. fruit or vegetables
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G09B 5/02: with visual presentation of the material to be studied, e.g. using film strip
- G09B 19/0092: {Nutrition}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G16H 20/60: relating to nutrition control, e.g. diets
- G02B 2027/014: {comprising information/image processing systems}
- G10L 25/51: for comparison or discrimination
- G10L 2015/223: {Execution procedure of a spoken command}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Aspects of the present disclosure are directed to quantitatively tracking food intake using smart glasses and/or other wearable devices. In some implementations, the smart glasses can include an image capture device, such as a camera, that can seamlessly capture images of food being eaten by the user. A computing device in communication with the smart glasses (or the smart glasses themselves) can identify the type and volume of food being eaten by applying object recognition and volume estimation techniques to the images. Additionally or alternatively, the smart glasses and/or other wearable devices can track a user's eating patterns through the number of bites taken throughout the day by capturing and analyzing hand-to-mouth motions and chewing. The computing device can log the type of food, volume of food, and/or number of bites taken and compute statistics that can be displayed to the user on the smart glasses.

## Claims

### Claim 1

A method for quantitatively tracking food intake using smart glasses, the method comprising:
capturing at least one image of food, and motion data indicative of motion by a user of the smart glasses;
identifying a type of the food by performing object recognition on the at least one image of food;
determining a volume of the food by performing volume estimation on the at least one image of food;
obtaining nutritional data associated with the type of the food and the volume of the food;
generating nutritional performance data by comparing the nutritional data to a nutritional benchmark for the user;
identifying a plurality of hand-to-mouth motions by the user by analyzing the motion data;
identifying a plurality of chewing motions by the user using the smart glasses;
calculating a weighted average of a number of the plurality of hand-to-mouth motions and a number of the plurality of chewing motions, wherein calculating the weighted average includes weighing the number of the plurality of hand-to-mouth motions more heavily than the number of the plurality of chewing motions;
generating food intake frequency data by comparing the weighted average of the number of the plurality of hand-to-mouth motions and the number of the plurality of chewing motions to baseline metrics; and
displaying the nutritional performance data and the food intake frequency data to the user on the smart glasses.

### Claim 2 (depends on 1)

The method of claim 1, wherein identifying the plurality of hand-to-mouth motions includes determining that a gaze of the user is focused on food being brought to a mouth of the user.

### Claim 3 (depends on 1)

The method of claim 1, wherein the plurality of chewing motions are identified, at least in part, using an audio signal captured by a microphone on the smart glasses.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
receiving feedback from the user explicitly identifying the type of the food; and
updating, based on the feedback, a machine learning model trained to perform the object recognition for the type of the food.

### Claim 5 (depends on 1)

The method of claim 1, wherein the volume estimation is performed by analyzing a plurality of images of the food captured at different angles.

### Claim 6 (depends on 1)

The method of claim 1, wherein the volume estimation is performed by applying a machine learning model trained to predict depth of the food from one image of the food.

### Claim 7 (depends on 1)

The method of claim 1, wherein identifying the plurality of hand-to-mouth motions includes:
applying a machine learning model trained to receive the motion data and categorize the motion data as being or not being indicative of a hand-to-mouth motion.

### Claim 8 (depends on 1)

The method of claim 1, wherein the motion data is received from at least one of an inertial measurement unit (IMU) or an image capture device, or a combination thereof.

### Claim 9

A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for quantitatively tracking food intake using smart glasses, the process comprising:
capturing motion data indicative of motion by a user of the smart glasses;
identifying a plurality of hand-to-mouth motions by the user by analyzing the motion data;
identifying a plurality of chewing motions by the user using the smart glasses;
calculating a weighted average of a number of the plurality of hand-to-mouth motions and a number of the plurality of chewing motions, wherein calculating the weighted average includes weighing the number of the plurality of hand-to-mouth motions more heavily than the number of the plurality of chewing motions;
generating food intake frequency data by comparing the weighted average of the number of the plurality of hand-to-mouth motions and the number of the plurality of chewing motions to baseline metrics; and
displaying the food intake frequency data to the user on the smart glasses.

### Claim 10 (depends on 9)

The computer-readable storage medium of claim 9, wherein the process further comprises:
capturing at least one image of food;
identifying a type of the food by performing object recognition on the at least one image of food;
determining a volume of the food by performing volume estimation on the at least one image of food;
obtaining nutritional data associated with the type of the food and the volume of the food;
generating nutritional performance data by comparing the nutritional data to a nutritional benchmark for the user; and
displaying the nutritional performance data to the user on the smart glasses.

### Claim 11 (depends on 9)

The computer-readable storage medium of claim 9, wherein identifying the plurality of hand-to-mouth motions includes determining that a gaze of the user is focused on food being brought to a mouth of the user.

### Claim 12 (depends on 9)

The computer-readable storage medium of claim 9, wherein the plurality of chewing motions are identified, at least in part, using an audio signal captured by a microphone on the smart glasses.

### Claim 13 (depends on 9)

The computer-readable storage medium of claim 9, wherein identifying the plurality of hand-to-mouth motions includes:
applying a machine learning model trained to receive the motion data and categorize the motion data as being or not being indicative of a hand-to-mouth motion.

### Claim 14 (depends on 9)

The computer-readable storage medium of claim 9, wherein the motion data is received from at least one of an inertial measurement unit (IMU) or an image capture device, or a combination thereof.

### Claim 15

A computing system for quantitatively tracking food intake using smart glasses, the computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
capturing at least one image of food;
identifying a type of the food by performing object recognition on the at least one image of food;
determining a volume of the food by performing volume estimation on the at least one image of food;
obtaining nutritional data associated with the type of the food and the volume of the food;
generating nutritional performance data by comparing the nutritional data to a nutritional benchmark for a user of the smart glasses;
identifying a plurality of hand-to-mouth motions by the user by analyzing the motion data;
identifying a plurality of chewing motions by the user using the smart glasses;
calculating a weighted average of a number of the plurality of hand-to-mouth motions and a number of the plurality of chewing motions, wherein calculating the weighted average includes weighing the number of the plurality of hand-to-mouth motions more heavily than the number of the plurality of chewing motions;
generating food intake frequency data by comparing the weighted average of the number of the plurality of hand-to-mouth motions and the number of the plurality of chewing motions to baseline metrics; and
displaying the nutritional performance data and the food intake frequency data to the user on the smart glasses.

### Claim 16 (depends on 15)

The computing system of claim 15, wherein the volume estimation is performed by analyzing a plurality of images of the food captured at different angles.

### Claim 17 (depends on 15)

The computing system of claim 15, wherein the volume estimation is performed by applying a machine learning model trained to predict depth of the food from one image of the food.