# US 2023/0055013 A1 · Accessory Detection and Determination for Avatar Enrollment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0055013 A1
- Document ID: US20230055013A1
- Status: Published application
- Kind: A1
- Application number: 17/819,746
- Filed: 2022-08-15
- Published: 2023-02-23
- Earliest claimed: 2021-08-23
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Gabriela N. Venturini
- Web page: https://patentgrep.com/patent/US20230055013A1

## CPC

- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/469: {Contour-based spatial representations, e.g. vector-coding}
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Devices, methods, and non-transitory program storage devices (NPSDs) are disclosed herein to allow individual users an opportunity to create customized instances of three-dimensional (3D) avatars, wherein each instance of the avatar may be customized to have particular visual characteristics and/or accessories that may reflect an individual user's appearance. When images are captured, e.g., during an individual user's avatar enrollment or customization process, novel shape matching techniques may be employed between two-dimensional (2D) objects of interest (e.g., eyeglasses frames) identified in the enrollment image and 3D models stored in one or more 3D model object libraries. A ranked listing of 3D models from an object library that provide the best shape matches to the identified 2D objects of interest in the captured image may automatically be determined and/or presented to the user for selection, e.g., via a user interface, for use in the creation and/or customization of the user's 3D avatar.

## Claims

### Claim 1

A device, comprising:
a memory;
an image capture device; and
one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions causing the one or more processors to:
obtain a first image captured by the image capture device, wherein the first image comprises a first subject wearing a first pair of eyeglasses;
determine a first outline of the first pair of eyeglasses;
obtain an object library, wherein the object library comprises a first plurality of three-dimensional (3D) models of pairs of eyeglasses;
determine a second outline for each of a first set of the first plurality of 3D models of pairs of eyeglasses in the object library; and
determine a similarity score for each of the 3D models of pairs of eyeglasses in the first set based, at least in part, on a comparison of the first outline to the respective second outline for each of the 3D models of pairs of eyeglasses in the first set.

### Claim 2 (depends on 1)

The device of claim 1, wherein the one or more processors are further configured to execute instructions causing the one or more processors to:
determine a ranked listing of the 3D models of pairs of eyeglasses in the first set based, at least in part, on their respective similarity scores.

### Claim 3 (depends on 2)

The device of claim 2, wherein the device further comprises a display and the one or more processors are further configured to execute instructions causing the one or more processors to:
present, on the display, at least a part of the ranked listing of the 3D models of pairs of eyeglasses in the first set.

### Claim 4 (depends on 3)

The device of claim 3, wherein the device further comprises a user interface and the one or more processors are further configured to execute instructions causing the one or more processors to:
receive, via the user interface, a selection of a first one of the 3D models of pairs of eyeglasses in the first set; and
present, on the display of the device, a view of an avatar representing the first subject, wherein the avatar is wearing the selected first one of the 3D models of pairs of eyeglasses.

### Claim 5 (depends on 1)

The device of claim 1, wherein the first outline comprises an outline of a frontal view of the first pair of eyeglasses.

### Claim 6 (depends on 1)

The device of claim 1, wherein the similarity score for each of the 3D models of pairs of eyeglasses in the first set comprises:
an estimated area between the first outline and the second outline for the respective 3D model of pairs of eyeglasses in the first set.

### Claim 7 (depends on 6)

The device of claim 6, wherein the instructions to determine a similarity score for each of the 3D models of pairs of eyeglasses in the first set further comprise instructions configured to cause the one or more processors to:
align the first outline and the second outline for the respective 3D model of pairs of eyeglasses in the first set before estimating the area between the first outline and the second outline for the respective 3D model.

### Claim 8

A non-transitory program storage device (NPSD) comprising computer readable instructions executable by one or more processors to:
obtain a first image captured by an image capture device, wherein the first image comprises a first subject wearing a first pair of eyeglasses;
determine a first outline of the first pair of eyeglasses;
obtain an object library, wherein the object library comprises a first plurality of three-dimensional (3D) models of pairs of eyeglasses;
determine a second outline for each of a first set of the first plurality of 3D models of pairs of eyeglasses in the object library; and
determine a similarity score for each of the 3D models of pairs of eyeglasses in the first set based, at least in part, on a comparison of the first outline to the respective second outline for each of the 3D models of pairs of eyeglasses in the first set.

### Claim 9 (depends on 8)

The NPSD of claim 8, further comprising computer readable instructions executable by the one or more processors to:
determine a ranked listing of the 3D models of pairs of eyeglasses in the first set based, at least in part, on their respective similarity scores.

### Claim 10 (depends on 9)

The NPSD of claim 9, further comprising computer readable instructions executable by the one or more processors to:
present, on a display, at least a part of the ranked listing of the 3D models of pairs of eyeglasses in the first set.

### Claim 11 (depends on 10)

The NPSD of claim 10, further comprising computer readable instructions executable by the one or more processors to:
receive a selection of a first one of the 3D models of pairs of eyeglasses in the first set; and
present, on the display, a view of an avatar representing the first subject, wherein the avatar is wearing the selected first one of the 3D models of pairs of eyeglasses.

### Claim 12

An image processing method, comprising:
obtaining a first image captured by an image capture device, wherein the first image comprises a first object of interest;
determining a first outline of the first object of interest;
obtaining an object library, wherein the object library comprises a first plurality of three-dimensional (3D) models of variants of the first object of interest;
determining a second outline for each of a first set of the first plurality of 3D models of variants of the first object of interest; and
determining a similarity score for each of the 3D models of variants of the first object of interest in the first set based, at least in part, on a comparison of the first outline to the respective second outline for each of the 3D models of variants of the first object of interest in the first set.

### Claim 13 (depends on 12)

The method of claim 12, further comprising:
determining a ranked listing of the 3D models of variants of the first object of interest in the first set based, at least in part, on their respective similarity scores.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
presenting, on a display, at least a part of the ranked listing of the 3D models of variants of the first object of interest in the first set.

### Claim 15 (depends on 12)

The method of claim 12, wherein the first outline and each of the second outlines for the 3D models of variants of the first object of interest in the first set comprise: an outline of a specified view of the first object of interest; and outlines of the specified view of the variants of the first object of interest, respectively.

### Claim 16 (depends on 12)

The method of claim 12, wherein the similarity score for each of the 3D models of variants of the first object of interest in the first set comprises:
an estimated area between the first outline and the second outline for the respective 3D model of the variants of the first object of interest in the first set.

### Claim 17 (depends on 16)

The method of claim 16, wherein determining a similarity score for each of the 3D models of variants of the first object of interest in the first set further comprises:
aligning the first outline and the second outline for the respective 3D model of the variant of the first object of interest in the first set before estimating the area between the first outline and the second outline for the respective 3D model.

### Claim 18 (depends on 12)

The method of claim 12, wherein determining a first outline of the first object of interest further comprises:
obtaining a first masked region from the first image, wherein the first masked region comprises an indication of pixels from the first image determined to be part of the first object of interest.

### Claim 19 (depends on 18)

The method of claim 18, wherein determining a first outline of the first object of interest further comprises:
extracting a set of edge pixels from the first masked region;
placing the set of edge pixels over a grid comprising a plurality of cells;
determining a center of mass of edge pixels located within each cell of the grid; and
determining the first outline of the first object of interest as an ordered list of key points, wherein the key points comprise the centers of mass of cells of the grid containing edge pixels representing an outer edge of the first object of interest.

### Claim 20 (depends on 12)

The method of claim 12, wherein determining a second outline for each of a first set of the first plurality of 3D models of variants of the first object of interest in the object library further comprises:
for each 3D model of variants of the first object of interest in the first set:
extracting a set of edge pixels from the respective 3D model, as oriented in a specified view;
placing the set of edge pixels over a grid comprising a plurality of cells;
determining a center of mass of edge pixels located within each cell of the grid; and
determining the second outline of the respective 3D model as an ordered list of key points, wherein the key points comprise the centers of mass of cells of the grid containing edge pixels representing an outer edge of the respective 3D model of a variant of the first object of interest.
