# US 2025/0095304 A1 · Object counting on ar wearable devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0095304 A1
- Document ID: US20250095304A1
- Status: Published application
- Kind: A1
- Application number: 18/959,194
- Filed: 2024-11-25
- Published: 2025-03-20
- Earliest claimed: 2022-08-23
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Piotr Gurgul; Sharon Moll; Tomasz Zakrzewski
- Web page: https://patentgrep.com/patent/US20250095304A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/20: in augmented reality scenes
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 7/73: using feature-based methods
- G06V 20/53: {Recognition of crowd images, e.g. recognition of crowd congestion}
- G06V 20/68: Food, e.g. fruit or vegetables
- G06T 2200/16: involving adaptation to the client's capabilities
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/30242: Counting objects in image
- G06T 2210/12: Bounding box
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems, methods, and computer readable media for object counting on augmented reality (AR) wearable devices are disclosed. Embodiments are disclosed that enable display of a count of objects as part of a user view. Upon receipt of a request to count objects, the AR wearable device captures an image of the user view. The AR wearable device transmits the image to a backend for processing to determine the objects in the image. The AR wearable device selects a group of objects of the determined objects to count and overlays boundary boxes over counted objects within the user view. The position of the boundary boxes is adjusted to account for movement of the AR wearable device. A hierarchy of objects is used to group together objects that are related but have different labels or names.

## Claims

### Claim 1

An augmented reality (AR) wearable device comprising:
at least one processor; and
one or more memories storing instructions that, when executed by the at least one processor, configure the AR wearable device to perform operations comprising:
capturing, by an image capturing device of the AR wearable device, an image corresponding to a user view of a real-world scene;
processing the image to determine locations of a plurality of objects within the image, and labels for the plurality of objects;
selecting a group of objects of the plurality of objects based at least on the labels for the plurality of objects indicating a same label;
counting a number of objects in the group of objects to generate a count for the group of objects;
identifying an area of the image where additional objects are obscured, wherein the additional objects are excluded from the count of the group of objects; and
displaying, on a display of the AR wearable device, indications that the number of objects in the group of objects have been counted, an indication that the additional objects are obscured, and an indication of the count.

### Claim 2 (depends on 1)

The AR wearable device of claim 1, wherein the identifying further includes identifying three-dimensional coordinates within a world coordinate system for the plurality of objects.

### Claim 3 (depends on 1)

The AR wearable device of claim 1, wherein locations of the indications on the display are based on the locations of the objects in the group of objects and a second user view of the real-world scene.

### Claim 4 (depends on 1)

The AR wearable device of claim 1, wherein the processing further comprises:
sending, via a wireless communication link, an instruction for a computing device to process the image; and
receiving input indicating the locations of the plurality of objects and labels for the plurality of objects.

### Claim 5 (depends on 1)

The AR wearable device of claim 1 wherein the selecting is further based on at least one of:
the group of objects being located in a central portion of the image, the group of objects being located within a threshold distance of one another, the group of objects being within a threshold distance of a user of the AR wearable device, the group of objects having a same label, or an area selected by the user.

### Claim 6 (depends on 1)

The AR wearable device of claim 1 wherein the indications that the number of objects in the group of objects have been counted are boundary boxes around each of the number of objects of the group of objects.

### Claim 7 (depends on 1)

The AR wearable device of claim 1 wherein probabilities are associated with the labels, the probabilities indicating a likelihood a corresponding object of the group of objects is correctly labeled, and wherein boundary boxes indicate the probabilities associated with the labels of the group of objects.

### Claim 8 (depends on 1)

The AR wearable device of claim 1, wherein the operations further comprise:
receiving an indication from a user to count objects having a label of the labels.

### Claim 9 (depends on 1)

The AR wearable device of claim 1, wherein the operations further comprise:
before the identifying, receiving a selection of an area of the image, and wherein the identifying is within the area of the image.

### Claim 10 (depends on 1)

The AR wearable device of claim 1, wherein the labels identify names of the plurality of objects.

### Claim 11 (depends on 1)

The AR wearable device of claim 1, wherein the identifying further includes identifying three-dimensional coordinates within a world coordinate system for the plurality of objects, and wherein the counting the number of objects further comprises:
determining that a subset of objects of the group of objects were previously counted; and
generating a count of objects of the group of objects which were not previously counted.

### Claim 12 (depends on 1)

The AR wearable device of claim 1, wherein the operations further comprise:
determining values for objects of the group of objects;
determining a total value for the group of objects based on the values; and
displaying, on the display, an indication of the total value.

### Claim 13 (depends on 12)

The AR wearable device of claim 12, wherein the values for the objects are currency values and the objects are currency or the values for the objects are calorie estimates and the objects are food.

### Claim 14 (depends on 1)

The AR wearable device of claim 1, wherein the number of objects are people, and wherein the operations further comprise:
determining, by processing the image, a venue where the people are located; and
in response to the count exceeding or transgressing a threshold of a number of people for the venue, displaying, on the display, an indication that the number of people has exceeded a capacity for the venue.

### Claim 15 (depends on 1)

The AR wearable device of claim 1, wherein the image is a first image, the count is a first count, the group of objects is a first group of objects, the plurality of objects is a first plurality of objects, the labels are first labels, the indications are first indications, wherein the processing further comprises processing the image to determine first three-dimensional coordinates within a world coordinate system for the first plurality of objects, and wherein the operations further comprise:
capturing, by the image capturing device, a second image corresponding to a third user view of the real-world scene;
processing the second image to determine indications of a second plurality of objects within the second image, second three-dimensional coordinates within the world coordinate system for the second plurality of objects, and second labels for the second plurality of objects;
selecting a second group of objects of the second plurality of objects based on objects of the second group of objects being in a same group of objects as the first group of objects;
counting objects of the second group of objects which were not previously counted in the first group of objects to generate a second count; and
displaying, on the display of the AR wearable device, second indications that the second group of objects which were not previously counted have been counted and an indication of the second count, wherein locations of the second indications are based on the second three-dimensional coordinates of the second group of objects and a third user view of the real-world scene.

### Claim 16 (depends on 15)

The AR wearable device of claim 15, further comprising:
displaying, on the display of the AR wearable device, third indications that the second group of objects which were previously counted have not been counted in the second count and an indication of the second count plus the first count, wherein locations of the third indications are based on the second three-dimensional coordinates of the second group of objects which were previously counted and the third user view of the real-world scene.

### Claim 17

A method performed by an apparatus of an augmented reality (AR) wearable device comprising:
capturing, by an image capturing device of the AR wearable device, an image corresponding to a user view of a real-world scene;
processing the image to determine locations of a plurality of objects within the image, and labels for the plurality of objects;
selecting a group of objects of the plurality of objects based at least on the labels for the plurality of objects indicating a same label;
counting a number of objects in the group of objects to generate a count for the group of objects;
identifying an area of the image where additional objects are obscured, wherein the additional objects are excluded from the count of the group of objects; and
displaying, on a display of the AR wearable device, indications that the number of objects in the group of objects have been counted, an indication that the additional objects are obscured, and an indication of the count.

### Claim 18 (depends on 17)

The method of claim 17, wherein the selecting is further based on at least one of: the group of objects being located in a central portion of the image, the group of objects being located within a threshold distance of one another, the group of objects being within a threshold distance of a user of the AR wearable device, the group of objects having a same label, or an area selected by the user.

### Claim 19

A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by at least one processor of an apparatus of an augmented reality (AR) wearable device, cause the at least one processor to perform operations comprising:
capturing, by an image capturing device of the AR wearable device, an image corresponding to a user view of a real-world scene;
processing the image to determine locations of a plurality of objects within the image, and labels for the plurality of objects;
selecting a group of objects of the plurality of objects based at least on the labels for the plurality of objects indicating a same label;
counting a number of objects in the group of objects to generate a count for the group of objects;
identifying an area of the image where additional objects are obscured, wherein the additional objects are excluded from the count of the group of objects; and
displaying, on a display of the AR wearable device, indications that the number of objects in the group of objects have been counted, an indication that the additional objects are obscured, and an indication of the count.

### Claim 20 (depends on 19)

The non-transitory computer-readable storage medium of claim 19, wherein the selecting is further based on at least one of: the group of objects being located in a central portion of the image, the group of objects being located within a threshold distance of one another, the group of objects being within a threshold distance of a user of the AR wearable device, the group of objects having a same label, or an area selected by the user.
