# US 2018/0025412 A1 · Determining in-store location based on images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0025412 A1
- Document ID: US20180025412A1
- Status: Published application
- Kind: A1
- Application number: 15/656,922
- Filed: 2017-07-21
- Published: 2018-01-25
- Earliest claimed: 2016-07-22
- Assignees: Focal Systems, Inc.
- Applicants: Focal Systems, Inc.
- Inventors: Francois Chaubard; Adriano Quiroga Garafulic
- Web page: https://patentgrep.com/patent/US20180025412A1

## CPC

- G06Q 30/0639: {Locating goods or services, e.g. based on physical position of the goods or services within a shopping facility}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06N 20/00: Machine learning
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An in-store location system determines the location of a shopper within a store based on images received from a shopper client device. The shopper client device can be attached to a shopping cart and may be connected to one or more cameras that capture images of products on shelves. The in-store location system can detect products in images received from the shopper client device using a machine-learned product detection model to detect the products in the received images. The in-store location system can then determine the location of the shopper within the store based on the received images. The in-store location system may compare the detected products to a store map or a planogram describing the store. The in-store location system also may apply a machine-learned location-determination model to the received images.

## Claims

### Claim 1

A method comprising:
receiving an image from a shopper client device associated with a shopper, the image capturing one or more products offered for sale by a store;
detecting the one or more products described in the image;
determining a location of the shopper within the store based on the detected one or more products; and
storing the location of the shopper at an in-store location system.

### Claim 2 (depends on 1)

The method of claim 1, wherein the image is captured via a camera of or connected to the shopper client device.

### Claim 3 (depends on 1)

The method of claim 1, wherein the shopper client device is attached to a shopping unit being used by the shopper.

### Claim 4 (depends on 1)

The method of claim 1, wherein determining the location of the shopper comprises:
comparing the received image to one or more references images received from a store client device operated by a store associate.

### Claim 5 (depends on 1)

The method of claim 1, wherein determining the location of the shopper comprises:
applying a location-determination model to the received image.

### Claim 6 (depends on 5)

The method of claim 5, wherein the location-determination model is trained based on reference images captured by a store client device.

### Claim 7 (depends on 6)

The method of claim 6, wherein the location-determination model is trained based on location information associated with each reference image of the reference images.

### Claim 8 (depends on 1)

The method of claim 1, wherein detecting the one or more products comprises:
applying a product-detection model to the received image.

### Claim 9 (depends on 8)

The method of claim 8, wherein the product-detection model is trained based on reference images received from a store client device.

### Claim 10 (depends on 9)

The method of claim 9, wherein training the product-detection model comprises:
receiving boundary boxes from a store client device, each boundary box indicating a portion of the received image that represents a product.

### Claim 11 (depends on 1)

The method of claim 1, wherein detecting the one or more products comprises:
applying an optical character recognition algorithm to the received image.

### Claim 12 (depends on 1)

The method of claim 1, further comprises:
receiving an identifier identifying a product within the store; and
transmitting, to the shopper client device for presentation to the shopper, a route from the location of the shopper to a location of the product within the store.

### Claim 13 (depends on 1)

The method of claim 1, further comprising: determining that a product is out of stock based on whether the product is detected in the image.

### Claim 14 (depends on 1)

The method of claim 1, further comprising: determining that the store is out of compliance with a planogram associated with the store based on the detected one or more products.

### Claim 15 (depends on 1)

The method of claim 1, further comprising: determining shopper behavior of the shopper based on the location of the shopper.

### Claim 16 (depends on 1)

The method of claim 1, further comprising: transmitting product information to the shopper client device, the product information being associated with a product of the detected one or more products.

### Claim 17 (depends on 1)

The method of claim 1, further comprising: transmitting the location of the shopper to the shopper client device for presentation to the shopper.

### Claim 18

A non-transitory, computer-readable medium comprising instructions that, when executed by a processor, cause the processor to:
receive an image from a shopper client device associated with a shopper, the image capturing one or more products offered for sale by a store;
detect the one or more products described in the image;
determine a location of the shopper within the store based on the detected one or more products; and
store the location of the shopper at an in-store location system.

### Claim 19 (depends on 18)

The computer-readable medium of claim 18, wherein the instructions for determining the location of the shopper comprise instructions that cause the processor to: apply a location-determination model to the received image.

### Claim 20 (depends on 18)

The computer-readable medium of claim 18, wherein the instructions for detecting the one or more products comprise instructions that cause the processor to: apply an optical character recognition algorithm to the received image.
