# US 2022/0044488 A1 · Computerized system and method for providing a mobile augmented reality item display and selection experience

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044488 A1
- Document ID: US20220044488A1
- Status: Published application
- Kind: A1
- Application number: 17/509,521
- Filed: 2021-10-25
- Published: 2022-02-10
- Earliest claimed: 2019-04-03
- Assignees: Posnap, Inc.
- Applicants: Posnap, Inc.
- Inventors: Mao Shen YEH
- Web page: https://patentgrep.com/patent/US20220044488A1

## CPC

- G06T 7/73: using feature-based methods
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06T 7/74: {involving reference images or patches}
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/20: in augmented reality scenes
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06T 2207/20081: Training; Learning
- G06T 2207/30201: Face
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Disclosed are systems and methods for detecting the specific positioning, orientation, plane(s) and scale of a real-world object, and providing an augmented reality (AR) experience of the object based therefrom. The disclosed systems and method provide a novel, efficient and accurate mechanism for launching an AR application that provides an AR view of a captured or currently being viewed image. The instant disclosure's AR positioning and tracking systems and methods provide a streamlined system that maximizes the device's computational resources in order to accurately determine and track the viewed object's and the capturing device's positioning and orientation, as well as the object's physical dimensions, thereby ensuring an efficiently produced AR experience.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

A method comprising:
capturing, by a device, an image of a user, the image comprising a digital representation of the user;
displaying, by the device, the image within a user interface (UI) associated with an augmented reality (AR) application executing on the device;
analyzing, by the device, the image, and determining a set of features of the user represented within the displayed image;
determining, by the device, visual characteristics of each feature in the set of features, the visual characteristics, for each feature, comprising information indicating a location of a feature and color information of each feature;
receiving, by the device, information related to an item, the item information indicating a specific type of item which corresponds to a particular feature of a person, the item information further indicating a color of the item;
identifying, by the device, a feature from the set of features based on the particular feature associated with the item;
displaying, by the device, the item as an overlay of the identified feature of the user displayed within the UI, the display of the item is based on the color information of the identified feature and the color of the item.

### Claim 22 (depends on 21)

The method of claim 21, further comprising:
determining a contrast of the color information of the identified feature and the color of the item, wherein the display of the item is based on the determined contrast such that accuracy of item's display is improved.

### Claim 23 (depends on 21)

The method of claim 21, further comprising:
analyzing the visual characteristics, and determining a predetermined set of locations on the user within the image;
determining the color information for each location based on features and visual characteristics associated with location.

### Claim 24 (depends on 23)

The method of claim 23, wherein the color information corresponds to a skin tone of the user associated with a particular feature.

### Claim 25 (depends on 23)

The method of claim 23, wherein the determination of the color information is based on execution of a computer vision algorithm.

### Claim 26 (depends on 21)

The method of claim 21, further comprising:
determining a pose and rotation of the user based on the determined features of the user.

### Claim 27 (depends on 26)

The method of claim 26, further comprising:
determining a three dimensional (3D) spatial orientation and alignment of the user, wherein the pose of the user is based on the 3D spatial orientation and alignment.

### Claim 28 (depends on 26)

The method of claim 26, further comprising:
analyzing the visual characteristics, and determining a predetermined set of locations on the user within the image; and
determining the rotation of the user based on the predetermined set of locations.

### Claim 29 (depends on 26)

The method of claim 26, wherein the item information further indicates a shape of the item.

### Claim 30 (depends on 29)

The method of claim 29, further comprising:
displaying the item based on the shape of the item and the pose and rotation of the user, in addition to the color information of the identified feature and the color of the item.

### Claim 31 (depends on 21)

The method of claim 21, further comprising:
causing communication, over the network, of information related to the item to a third party platform to obtain a digital content item comprising third party provided digital content associated with the item information;
receiving, over the network, the digital content item; and
communicating the digital content item to the user for display in association with the UI.

### Claim 32

A non-transitory computer-readable storage medium tangibly encoded with computer-executable instructions, that when executed by a device, perform a method comprising:
capturing, by the device, an image of a user, the image comprising a digital representation of the user;
displaying, by the device, the image within a user interface (UI) associated with an augmented reality (AR) application executing on the device;
analyzing, by the device, the image, and determining a set of features of the user represented within the displayed image;
determining, by the device, visual characteristics of each feature in the set of features, the visual characteristics, for each feature, comprising information indicating a location of a feature and color information of each feature;
receiving, by the device, information related to an item, the item information indicating a specific type of item which corresponds to a particular feature of a person, the item information further indicating a color of the item;
identifying, by the device, a feature from the set of features based on the particular feature associated with the item;
displaying, by the device, the item as an overlay of the identified feature of the user displayed within the UI, the display of the item is based on the color information of the identified feature and the color of the item.

### Claim 33 (depends on 32)

The non-transitory computer-readable storage medium of claim 32, further comprising:
determining a contrast of the color information of the identified feature and the color of the item, wherein the display of the item is based on the determined contrast such that accuracy of item's display is improved.

### Claim 34 (depends on 32)

The non-transitory computer-readable storage medium of claim 32, further comprising:
analyzing the visual characteristics, and determining a predetermined set of locations on the user within the image;
determining the color information for each location based on features and visual characteristics associated with location.

### Claim 35 (depends on 34)

The non-transitory computer-readable storage medium of claim 34, wherein the color information corresponds to a skin tone of the user associated with a particular feature.

### Claim 36 (depends on 32)

The non-transitory computer-readable storage medium of claim 32, further comprising:
determining a pose and rotation of the user based on the determined features of the user.

### Claim 37 (depends on 36)

The non-transitory computer-readable storage medium of claim 36, further comprising:
determining a three dimensional (3D) spatial orientation and alignment of the user, wherein the pose of the user is based on the 3D spatial orientation and alignment.

### Claim 38 (depends on 36)

The non-transitory computer-readable storage medium of claim 36, further comprising:
analyzing the visual characteristics, and determining a predetermined set of locations on the user within the image; and
determining the rotation of the user based on the predetermined set of locations.

### Claim 39 (depends on 36)

The non-transitory computer-readable storage medium of claim 36, further comprising:
displaying the item based on a shape of the item and the pose and rotation of the user, in addition to the color information of the identified feature and the color of the item, wherein the item information further indicates the shape of the item.

### Claim 40

A device comprising:
a processor configured to:
capture an image of a user, the image comprising a digital representation of the user;
display the image within a user interface (UI) associated with an augmented reality (AR) application;
analyze, the image, and determine a set of features of the user represented within the displayed image;
determine visual characteristics of each feature in the set of features, the visual characteristics, for each feature, comprising information indicating a location of a feature and color information of each feature;
receive information related to an item, the item information indicating a specific type of item which corresponds to a particular feature of a person, the item information further indicating a color of the item;
identify a feature from the set of features based on the particular feature associated with the item;
display the item as an overlay of the identified feature of the user displayed within the UI, the display of the item is based on the color information of the identified feature and the color of the item.
