# US 12,108,892 B2 · Information processing apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,108,892 B2
- Document ID: US12108892B2
- Status: Granted
- Kind: B2
- Application number: 17/769,927
- Filed: 2020-11-11
- Published: 2022-12-08
- Granted: 2024-10-08
- Earliest claimed: 2019-11-15
- Assignees: Ntt Docomo, Inc.
- Applicants: Ntt Docomo, Inc.
- Inventors: Kensuke Ueda; Nobutaka Matsushima
- Web page: https://patentgrep.com/patent/US12108892B2

## CPC

- A47G 23/10: Devices for counting or marking the number of consumptions (on beverage-dispensing apparatus B65D, B67D; counting in general G06M)
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 20/68: Food, e.g. fruit or vegetables
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G16H 20/60: relating to nutrition control, e.g. diets
- G16H 30/40: for processing medical images, e.g. editing
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30128: Food products
- G06T 2207/30196: Human being; Person
- G06T 2207/30232: Surveillance
- G06T 2207/30241: Trajectory
- G06T 2207/30242: Counting objects in image
- A47G: HOUSEHOLD OR TABLE EQUIPMENT (book-ends A47B5/00; knives B26B)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

An information processing apparatus includes, when a first determiner determines the amount of food or beverage in a platter being decreased, a second determiner to determine whether food or beverage on the platter was consumed by one person of multiple people, or the food or beverage was moved onto one small plate of small plates, a consumption information generator to generate, when the food or beverage on the platter was consumed by the one person, consumption information in which an amount of food or beverage consumed by the one person, the one person, and the type of the food or beverage are associated with each other, and an estimator to increase, when the food or beverage was moved to one small plate, the amount of food or beverage on the one small plate by an amount of decrease in the food or beverage in the platter.

## Claims

### Claim 1

An information processing apparatus comprising:
an acquirer configured to acquire video image information indicative of a video image obtained by photographically capturing a table, on which a vessel of a first type to serve food or beverage to be consumed by multiple people and a plurality of vessels of a second type in each of which a portion of the food or beverage is to be served are arranged, and the multiple people;
an identification information generator configured to generate, based on the video image information, first identification information for identifying each of the multiple people, second identification information for identifying each of the vessel of the first type and the plurality of vessels of the second type, and third identification information for identifying a type of the food or beverage;
an estimator configured to estimate, based on the video image information, an amount of food or beverage served in each of the vessel of the first type and the plurality of vessels of the second type;
a first determiner configured to determine, based on the video image information, whether an amount of food or beverage served in the vessel of the first type at a point in time of a determination is decreased compared to an amount of food or beverage served in the vessel of the first type at a point in time that is a predetermined period prior to the point in time of the determination;
a second determiner configured to, when a result of the determination by the first determiner is affirmative, determine, based on the video image information, whether food or beverage served in the vessel of the first type was consumed by one person of the multiple people, or the food or beverage was moved into one vessel of the plurality of vessels of the second type; and
a consumption information generator configured to, when a result of a determination by the second determiner indicates that food or beverage served in the vessel of the first type was consumed by the one person, generate consumption information in which an amount of food or beverage consumed by the one person, the one person, and the type of the food or beverage are associated with each other,
wherein the estimator is configured to, when the result of the determination by the second determiner indicates that the food or beverage was moved into the one vessel, increase the amount of the food or beverage served in the one vessel of the second type by an amount of decrease in food or beverage served in the vessel of the first type.

### Claim 2 (depends on 1)

The information processing apparatus according to claim 1, further comprising:
an identifier configured to identify, based on the video image information, which of the multiple people each of the plurality of vessels of the second type corresponds to; and
a third determiner configured to determine whether an amount of food or beverage served in the one vessel of the plurality of vessels of the second type at a point in time of a determination is decreased compared to an amount of food or beverage served in the one vessel at a point in time that is the predetermined period prior to the point in time of the determination,
wherein the consumption information generator is configured to, when a result of a determination by the third determiner is affirmative, generate consumption information in which a person corresponding to the one vessel of the second type, an amount of food or beverage decreased from the food or beverage served in the one vessel at the point in time that is the predetermined period prior to the point in time of the determination by the third determiner, and the type of the food or beverage are associated with each other.

### Claim 3 (depends on 2)

The information processing apparatus according to claim 2, wherein
the identifier is configured to:
calculate a distance between the one vessel of the plurality of vessels of the second type and each of the multiple people; and
identify a person positioned at a shortest distance among a plurality of distances calculated for the multiple people, as the person corresponding to the one vessel.

### Claim 4 (depends on 1)

The information processing apparatus according to claim 1, wherein:
the video image information includes image information indicative of a still image for each of a plurality of frames; and
the estimator is configured to estimate the amount of food or beverage served in each of the vessel of the first type and the plurality of vessels of the second type based on a piece of image information indicative of a still image, in which neither the vessel of the first type nor the plurality of vessels of the second type are blocked by an object, among a plurality of pieces of image information corresponding to the plurality of frames, respectively.

### Claim 5 (depends on 1)

The information processing apparatus according to claim 1, wherein:
the second determiner is configured to:
start to track, when the result of the determination by the first determiner is affirmative, a destination of food or beverage served in the vessel of the first type based on the video image information;
identify, based on the video image information, a person having a hand holding a utensil for moving food or beverage; and
determine, when the food or beverage disappears without being moved into the plurality of vessels of the second type based on the video image information, the person having the hand holding the utensil as the one person who consumed the food or beverage served in the vessel of the first type.

### Claim 6 (depends on 5)

The information processing apparatus according to claim 5, wherein
the second determiner is configured to:
calculate, based on the video image information, a distance between the hand holding the utensil and each of the multiple people; and
identify a person positioned at a shortest distance among a plurality of distances calculated for the multiple people, as the person having the hand holding the utensil.