# US 11,908,290 B2 · Visual recognition and sensor fusion weight detection system and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,290 B2
- Document ID: US11908290B2
- Status: Granted
- Kind: B2
- Application number: 17/323,031
- Filed: 2021-05-18
- Published: 2021-09-02
- Granted: 2024-02-20
- Earliest claimed: 2018-12-21
- Assignees: Maplebear Inc.
- Applicants: Maplebear Inc.
- Inventors: Lin Gao; Shiyuan Yang
- Web page: https://patentgrep.com/patent/US11908290B2

## CPC

- G07G 1/0036: {Checkout procedures}
- G07G 1/14: Systems including one or more distant stations co-operating with a central processing unit
- A47F 9/048: {automatically}
- G01G 19/415: combined with recording means
- G01G 21/22: Weigh pans or other weighing receptacles; Weighing platforms
- G06Q 20/18: involving self-service terminals [SST], vending machines, kiosks or multimedia terminals
- G06Q 20/208: {Input by product or record sensing, e.g. weighing or scanner processing}
- G06V 20/44: {Event detection}
- G07G 1/0054: {with control of supplementary check-parameters, e.g. weight or number of articles}
- G07G 1/0063: {with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration}
- G07G 1/0072: {with means for detecting the weight of the article of which the code is read, for the verification of the registration (weighing combined with recording means, e.g. in weighing scale, G01G19/415)}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 20/68: Food, e.g. fruit or vegetables
- G07G: REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
- A47F: SPECIAL FURNITURE, FITTINGS, OR ACCESSORIES FOR SHOPS, STOREHOUSES, BARS, RESTAURANTS OR THE LIKE; PAYING COUNTERS
- G01G: WEIGHING (sorting by weighing B07C5/16)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Disclosed are visual recognition and sensor fusion weight detection system and method. An example method includes: tracking, by a sensor system, objects and motions within a selected area of a store; activating, by the sensor system, a first computing device positioned in the selected area in response to detecting a presence of a customer within the selected area; identifying, by the sensor system, the customer and at least one item carried by the customer; transmitting, by the sensor system, identifying information of the customer and the at least one item to a computing server system via a communication network; measuring, by the first computing device, a weight of the at least one item; transmitting, by the first computing device, the weight to the computing server system via the communication network; and generating, by the computing server system, via the communication network, transaction information of the at least one item.

## Claims

### Claim 1

A system deployed within a communication network, the system comprising:
a processor coupled to a non-transitory computer-readable storage medium and configured to control a plurality of modules to execute instructions for:
tracking objects and motions within a selected area of a store based on one or more sensors or cameras within the store;
in response to detecting a presence of a customer carrying at least one item within the selected area, activating a computing device positioned within the selected area to measure a weight of the at least one item;
weighing the at least one item using the computing device;
obtaining identifying information of the customer and the at least one item carried by the customer; and
transmitting the weight of the at least one item, the identifying information of the customer and the at least one item to the computing server system via the communication network.

### Claim 2 (depends on 1)

The system of claim 1, further comprising a plurality of cameras and sensors configured to track the objects and motions within the selected area of the store.

### Claim 3 (depends on 1)

The system of claim 1, wherein the non-transitory computer-readable storage medium is configured to store information relating to one or more characteristics of each item for sale in the store.

### Claim 4 (depends on 1)

The system of claim 1, wherein the processor is further configured to:
detect the presence of the customer in relation to a location of the computing device within the selected area; and
capture one or more images of the customer in response to detecting that the customer is within a selected distance of the computing device.

### Claim 5 (depends on 4)

The system of claim 4, wherein the processor is further configured to:
generate a unique identifier for the customer;
associate the one and more images of the customer with the unique identifier; and
continuously accumulate and store information relating to the customer on the non-transitory computer-readable storage medium at different locations and times in the store in connection with the unique identifier prior to a final check out by the customer.

### Claim 6 (depends on 1)

The system of claim 1, wherein the processor is further configured to:
detect characteristics of the customer and surrounding objects obtained inside and outside the selected area for identifying the customer.

### Claim 7 (depends on 6)

The system of claim 6, wherein the processor is further configured to:
apply a weight to each detected characteristic of the customer, wherein facial features of the customer are configured to carry a greater weight than that of a color and texture details of an upper body clothes of the customer; and
determine a threshold for the customer presented in the selected area in connection with a total weight of all detected characteristics of the customer.

### Claim 8

A method, comprising:
tracking, by a system, objects and motions within a selected area of a store based on one or more sensors or cameras within the store;
in response to detecting a presence of a customer carrying at least one item within the selected area, activating, by the system, a computing device positioned within the selected area to measure a weight of the at least one item;
weighing the at least one item using the computing device
obtaining, by the system, identifying information of the customer and the at least one item carried by the customer; and
transmitting, by the system, the weight of the at least one item, the identifying information of the customer and the at least one item to the computing server system via the communication network.

### Claim 9 (depends on 8)

The method of claim 8, further comprising storing, by the system, information relating to one or more characteristics of each item for sale in the store.

### Claim 10 (depends on 8)

The method of claim 8, further comprising:
detecting, by the system, the presence of the customer in relation to a location of the computing device within the selected area; and
capturing, by a plurality of cameras and sensors of the system, one or more images of the customer in response to detecting that the customer is within a selected distance of the computing device.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
generating, by the system, a unique identifier for the customer;
associating, by the system, the one and more images of the customer with the unique identifier; and
continuously accumulating and storing information relating to the customer, by the system, at different locations and times in the store in connection with the unique identifier prior to a final check out by the customer.

### Claim 12 (depends on 8)

The method of claim 8, further comprising:
detecting, by the system, characteristics of the customer and surrounding objects obtained inside and outside the selected area for identifying the customer.

### Claim 13 (depends on 12)

The method of claim 12, further comprising:
applying, by the system, a weight to each detected characteristic of the customer, wherein facial features of the customer are configured to carry a greater weight than that of a color and texture details of an upper body clothes of the customer; and
determining, by the system, a threshold for the customer presented in the selected area in connection with a total weight of all detected characteristics of the customer.

### Claim 14

A non-transitory computer readable medium storing computer executable instructions for a system deployed in a communication network, the instructions being configured for:
tracking objects and motions within a selected area of a store based on one or more sensors or cameras within the store;
in response to detecting a presence of a customer carrying at least one item within the selected area, activating a computing device positioned within the selected area to measure a weight of the at least one item;
weighing the at least one item using the computing device;
obtaining identifying information of the customer and the at least one item carried by the customer; and
transmitting the weight of the at least one item, the identifying information of the customer and the at least one item to the computing server system via the communication network.

### Claim 15 (depends on 14)

The non-transitory computer readable medium of claim 14, further comprising instructions for storing information relating to one or more characteristics of each item for sale in the store.

### Claim 16 (depends on 15)

The non-transitory computer readable medium of claim 15, further comprising instructions for:
detecting the presence of the customer in relation to a location of the computing device within the selected area; and
capturing, by a plurality of cameras and sensors of the system, one or more images of the customer in response to detecting that the customer is within a selected distance of the computing device.

### Claim 17 (depends on 16)

The non-transitory computer readable medium of claim 16, further comprising instructions for:
generating a unique identifier for the customer;
associating the one and more images of the customer with the unique identifier; and
continuously accumulating and storing information relating to the customer at different locations and times in the store in connection with the unique identifier prior to a final check out by the customer.

### Claim 18 (depends on 14)

The non-transitory computer readable medium of claim 14, further comprising instructions for:
detecting characteristics of the customer and surrounding objects obtained inside and outside the selected area for identifying the customer.

### Claim 19 (depends on 18)

The non-transitory computer readable medium of claim 18, further comprising instructions for:
applying a weight to each detected characteristic of the customer, wherein facial features of the customer are configured to carry a greater weight than that of a color and texture details of an upper body clothes of the customer; and
determining a threshold for the customer presented in the selected area in connection with a total weight of all detected characteristics of the customer.