# US 2022/0284684 A1 · Augmented reality feedback of inventory for an appliance

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0284684 A1
- Document ID: US20220284684A1
- Status: Published application
- Kind: A1
- Application number: 17/751,150
- Filed: 2022-05-23
- Published: 2022-09-08
- Earliest claimed: 2018-12-31
- Assignees: Whirlpool Corporation
- Applicants: Whirlpool Corporation
- Inventors: Joshua George Abdoo; Matthew P. Ebrom; Neomar Giacomini; Donald J. Gilmore; Brian N. Radford; Aaron Edward Showers; Collin Andrew Stipe
- Web page: https://patentgrep.com/patent/US20220284684A1

## CPC

- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- F25D 29/00: Arrangement or mounting of control or safety devices
- G06F 16/90335: {Query processing}
- G06F 16/9038: Presentation of query results
- G06V 20/20: in augmented reality scenes
- G06V 20/68: Food, e.g. fruit or vegetables
- F25D 2400/361: Interactive visual displays
- F25D 2700/06: Sensors detecting the presence of a product
- G06T 2207/30128: Food products
- G06T 2219/004: Annotating, labelling
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- F25D: REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR (refrigerated showcases A47F3/04; thermally-insulated vessels for domestic use A47J41/00; refrigerated vehicles, see the appropriate subclasses of classes B60 - B64; containers with thermal insulation in general B65D81/38; heat-transfer, heat-exchange or heat-storage materials, e.g. refrigerants, or materials for the production of heat or cold by chemical reactions other than by combustion C09K5/00; thermally-insulated vessels for liquefied or solidified gases F17C; air-conditioning or air-humidification F24F; refrigeration machines, plants, or systems F25B; cooling of instruments or comparable apparatus without refrigeration G12B)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A food storage system is disclosed. The system comprises an appliance having a cabinet defining an interior, the interior comprising at least one of a shelf, compartment, or drawer configured to store at least one food item, a vision system comprising at least one imager configured to record image data related to the at least one food item stored within the interior, and a controller in communication with the vision system. The controller can be operable to generate an augmented reality representation of the interior.

## Claims

### Claim 1

An inventory management system for an appliance, the inventory management system comprising:
a holder disposed in an interior of the appliance and configured to store a food item;
an imaging device configured to capture an image of the interior;
a display in communication with the imaging device configured to present image data corresponding to the image; and
a controller in communication with the imager and the display, the controller configured to:
determine an identity of the food item;
determine a target location for the food item based on the identity of the food item and an environmental condition of the interior; and
communicate an instruction to present the image data with an indication of the target location at the display.

### Claim 2 (depends on 1)

The inventory management system according to claim 1, wherein the image data includes an augmented reality representation including a photographic representation of the interior with a digital representation of the food item overlaying a digital representation of a position of the food item in the interior.

### Claim 3 (depends on 2)

The inventory management system according to claim 2, wherein the controller is further configured to generate the augmented reality representation based on the identity of the food item.

### Claim 4 (depends on 2)

The inventory management system according to claim 2, wherein the indication is a color indicator and the photographic representation of the interior includes the color indicator.

### Claim 5 (depends on 2)

The inventory management system according to claim 2, wherein the digital representation of the food item is presented as rendered graphics of the at least one food item.

### Claim 6 (depends on 1)

The inventory management system according to claim 1, wherein the controller is further configured to:
determine an expiration date of the food item based on a position of the food item in the interior and a heat distribution of the interior; and
communicate an instruction to present the image data with the expiration date at the display.

### Claim 7 (depends on 1)

The inventory management system according to claim 1, wherein the image data includes an overlay corresponding to a property of the food item.

### Claim 8 (depends on 7)

The inventory management system according to claim 7, wherein the property is a fill level and the overlay is a graphical image configured to indicate the fill level of the food item.

### Claim 9 (depends on 7)

The inventory management system according to claim 7, wherein the property is an expiration date and the overlay is a graphical image indicating the expiration date of the food item.

### Claim 10 (depends on 7)

The inventory management system according to claim 7, wherein the property is a favorite and the overlay is a graphical image indicating that the food item is a favorite.

### Claim at position 11

11: An inventory management system for an appliance, the inventory management system comprising:
at least one holder disposed in an interior of the appliance and configured to store a plurality of food items;
an imaging device configured to capture an image of the interior;
a display in communication with the imaging device configured to present image data corresponding to the image; and
a controller in communication with the imager and the display, the controller configured to:
determine an identity of each of one or more of the food items;
determine a target location for each of one or more of the food items based on the identity each of one or more of the food items and an environmental condition of the interior;
generate an augmented reality representation including a photographic representation of the interior with a digital representation of the food item overlaying a digital representation of a position of the food item in the interior, wherein the image data includes the augmented reality representation; and
communicate a first instruction to present the augmented reality representation with an indication of each target location at the display.

### Claim 12 (depends on 11)

The inventory management system according to claim 11, wherein presenting the augmented reality representation includes displaying food item overlays corresponding to the plurality of food items.

### Claim 13 (depends on 11)

The inventory management system according to claim 11, further comprising:
a lighting system in communication with the controller and configured to project a pattern of light in the interior, wherein the controller is further configured to control the lighting system to indicate the target location via the pattern of a light.

### Claim 14 (depends on 13)

The inventory management system according to claim 13, wherein the controller is further configured to control the lighting system to display a color spectrum in the interior indicating the environmental condition.

### Claim 15 (depends on 11)

The inventory management system according to claim 11, wherein the image data includes an overlay corresponding to a property of the food item.

### Claim 16 (depends on 15)

The inventory management system according to claim 15, wherein the property is a fill level and the overlay is a graphical image configured to indicate the fill level of the food item.

### Claim 17 (depends on 15)

The inventory management system according to claim 15, wherein the property is an expiration date and the overlay is a graphical image indicating the expiration date of the food item.

### Claim 18 (depends on 11)

The inventory management system according to claim 11, wherein the controller is further configured to:
receive recipe data corresponding to a recipe incorporating one or more of the plurality of food items;
communicate a second instruction to present a color indictor indicating positions of the one or more of the plurality of food items at the display.

### Claim 19

An inventory management system for an appliance, the inventory management system comprising:
a holder disposed in an interior of the appliance and configured to store a food item;
an imaging device configured to capture an image of the interior;
a display in communication with the imaging device configured to display image data corresponding to the image; and
a controller in communication with the imager and the display, the controller configured to:
determine a position of the food item based on the image;
determine a target location for the food item based on an environmental condition of the interior;
compare the position to the target location;
modify the image data to indicate the target location based on the comparison of the position to the target location; and
communicate an instruction to present the image data at the display.

### Claim 20 (depends on 19)

The inventory management system according to claim 19, wherein the image data includes an augmented reality representation including a photographic representation of the interior with a digital representation of the food item overlaying a digital representation of a position of the food item in the interior, wherein the photographic representation of the interior includes a color indicator indicating the comparison of the position to the target location, and wherein the environmental condition is a heat distribution of the interior.