# US 2023/0419537 A1 · Computerized system and method for providing a mobile augmented reality item display and selection experience

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0419537 A1
- Document ID: US20230419537A1
- Status: Published application
- Kind: A1
- Application number: 18/341,418
- Filed: 2023-06-26
- Published: 2023-12-28
- Earliest claimed: 2019-04-03
- Assignees: Posnap, Inc.
- Applicants: Posnap, Inc.
- Inventors: Mao Shen YEH
- Web page: https://patentgrep.com/patent/US20230419537A1

## CPC

- G06T 7/73: using feature-based methods
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06T 7/74: {involving reference images or patches}
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/20: in augmented reality scenes
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06T 2207/20081: Training; Learning
- G06T 2207/30201: Face
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Disclosed are systems and methods for detecting the specific positioning, orientation, plane(s) and scale of a real-world object, and providing an augmented reality (AR) experience of the object based therefrom. The disclosed systems and method provide a novel, efficient and accurate mechanism for launching an AR application that provides an AR view of a captured or currently being viewed image. The instant disclosure's AR positioning and tracking systems and methods provide a streamlined system that maximizes the device's computational resources in order to accurately determine and track the viewed object's and the capturing device's positioning and orientation, as well as the object's physical dimensions, thereby ensuring an efficiently produced AR experience.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

A method comprising:
acquiring, by a hand held device, an image of a user, the image comprising a digital representation of the user;
representing, by the hand held device, the image within a user interface (UI) associated with an augmented reality (AR) application executing on the hand held device;
evaluating, by the hand held device, the image, and determining a set of physical characteristics of the user represented within the represented image;
determining, by the hand held device, visual characteristics of each characteristic in the set of physical characteristics, the visual characteristics, for each characteristic, comprising information indicating a location of a characteristic and color information of each characteristic, the color information being based on an average color value;
obtaining, by the hand held device, information related to an item, the item information indicating a specific type of item which corresponds to a particular characteristic of a person, the item information further indicating a color of the item, the color being computed based on an average color value;
identifying, by the hand held device, a characteristic from the set of physical characteristics based on the particular characteristic associated with the item;
representing, by the hand held device, the item as an overlay of the identified characteristic of the user represented within the UI, the represent of the item is based the color information of the identified characteristic and the color of the item.

### Claim 22 (depends on 21)

The method of claim 21, further comprising:
determining a contrast of the color information of the identified characteristic and the color of the item, wherein the represent of the item is based on the determined contrast such that accuracy of item's represent is improved.

### Claim 23 (depends on 21)

The method of claim 21, further comprising:
evaluating the visual characteristics, and determining a predetermined set of locations on the user within the image;
determining the color information for each location based on physical characteristics and visual characteristics associated with location.

### Claim 24 (depends on 23)

The method of claim 23, wherein the color information corresponds to a skin tone of the user associated with a particular characteristic.

### Claim 25 (depends on 23)

The method of claim 23, wherein the determination of the color information is based on execution of a computer vision algorithm.

### Claim 26 (depends on 21)

The method of claim 21, further comprising:
determining a pose and rotation of the user based on the determined physical characteristics of the user.

### Claim 27 (depends on 26)

The method of claim 26, further comprising:
determining a three dimensional (3D) spatial orientation and alignment of the user, wherein the pose of the user is based on the 3D spatial orientation and alignment.

### Claim 28 (depends on 26)

The method of claim 26, further comprising:
evaluating the visual characteristics, and determining a predetermined set of locations on the user within the image; and
determining the rotation of the user based on the predetermined set of locations.

### Claim 29 (depends on 26)

The method of claim 26, wherein the item information further indicates a shape of the item.

### Claim 30 (depends on 29)

The method of claim 29, further comprising:
representing the item based on the shape of the item and the pose and rotation of the user, in addition to the color information of the identified characteristic and the color of the item.

### Claim 31 (depends on 21)

The method of claim 21, further comprising:
causing communication, over the network, of information related to the item to a third party platform to obtain a digital content item comprising third party provided digital content associated with the item information;
obtaining, over the network, the digital content item; and
communicating the digital content item to the user for represent in association with the UI.

### Claim 32

A non-transitory computer-readable storage medium tangibly encoded with computer-executable instructions, that when executed by a hand held device, perform a method comprising:
acquiring, by the hand held device, an image of a user, the image comprising a digital representation of the user;
representing, by the hand held device, the image within a user interface (UI) associated with an augmented reality (AR) application executing on the hand held device;
evaluating, by the hand held device, the image, and determining a set of physical characteristics of the user represented within the represented image;
determining, by the hand held device, visual characteristics of each characteristic in the set of physical characteristics, the visual characteristics, for each characteristic, comprising information indicating a location of a characteristic and color information of each characteristic, the color information being based on an average value;
obtaining, by the hand held device, information related to an item, the item information indicating a specific type of item which corresponds to a particular characteristic of a person, the item information further indicating a color of the item, the color being based on an average value;
identifying, by the hand held device, a characteristic from the set of physical characteristics based on the particular characteristic associated with the item;
representing, by the hand held device, the item as an overlay of the identified characteristic of the user represented within the UI, the represent of the item is based on the color information of the identified characteristic and the color of the item.

### Claim 33 (depends on 32)

The non-transitory computer-readable storage medium of claim 32, further comprising:
determining a contrast of the color information of the identified characteristic and the color of the item, wherein the represent of the item is based on the determined contrast such that accuracy of item's represent is improved.

### Claim 34 (depends on 32)

The non-transitory computer-readable storage medium of claim 32, further comprising:
evaluating the visual characteristics, and determining a predetermined set of locations on the user within the image;
determining the color information for each location based on physical characteristics and visual characteristics associated with location.

### Claim 35 (depends on 34)

The non-transitory computer-readable storage medium of claim 34, wherein the color information corresponds to a skin tone of the user associated with a particular characteristic.

### Claim 36 (depends on 32)

The non-transitory computer-readable storage medium of claim 32, further comprising:
determining a pose and rotation of the user based on the determined physical characteristics of the user.

### Claim 37 (depends on 36)

The non-transitory computer-readable storage medium of claim 36, further comprising:
determining a three dimensional (3D) spatial orientation and alignment of the user, wherein the pose of the user is based on the 3D spatial orientation and alignment.

### Claim 38 (depends on 36)

The non-transitory computer-readable storage medium of claim 36, further comprising:
evaluating the visual characteristics, and determining a predetermined set of locations on the user within the image; and
determining the rotation of the user based on the predetermined set of locations.

### Claim 39 (depends on 36)

The non-transitory computer-readable storage medium of claim 36, further comprising:
representing the item based on a shape of the item and the pose and rotation of the user, in addition to the color information of the identified characteristic and the color of the item, wherein the item information further indicates the shape of the item.

### Claim 40

A hand held device comprising:
a processor configured to:
acquire an image of a user, the image comprising a digital representation of the user;
represent the image within a user interface (UI) associated with an augmented reality (AR) application;
evaluate, the image, and determine a set of physical characteristics of the user represented within the represented image;
determine visual characteristics of each characteristic in the set of physical characteristics, the visual characteristics, for each characteristic, comprising information indicating a location of a characteristic and color information of each characteristic, the color information being based on the average value;
obtain information related to an item, the item information indicating a specific type of item which corresponds to a particular characteristic of a person, the item information further indicating a color of the item, the color being based on an average value;
identify a characteristic from the set of physical characteristics based on the particular characteristic associated with the item;
represent the item as an overlay of the identified characteristic of the user represented within the UI, the represent of the item is based on the color information of the identified characteristic and the color of the item.
