# US 2022/0092857 A1 · Systems and methods for generating augmented reality content based on distorted three-dimensional models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0092857 A1
- Document ID: US20220092857A1
- Status: Published application
- Kind: A1
- Application number: 17/029,463
- Filed: 2020-09-23
- Published: 2022-03-24
- Earliest claimed: 2020-09-23
- Assignees: Shopify Inc.
- Applicants: Shopify Inc.
- Inventors: JUHO MIKKO HAAPOJA; BYRON LEONEL DELGADO; STEPHAN LEROUX; DANIEL BEAUCHAMP; MAAS MANSOOR ALI LALANI
- Web page: https://patentgrep.com/patent/US20220092857A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06N 20/00: Machine learning
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06T 15/20: Perspective computation
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 2207/20081: Training; Learning
- G06T 2207/30204: Marker
- G06T 2210/16: Cloth
- G06T 2219/2021: Shape modification
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Systems and methods for generating AR content to virtually alter a real-world item are provided. The real-world item may be a deformable item including one or more flexible surfaces. According to an embodiment, a method includes determining the relative positions of a plurality of points on the real-world item. Based on the relative positions of the plurality of points, a 3D model of a virtual item is distorted to produce a distorted 3D model of the virtual item. The distorted 3D model may have substantially the same shape as the real-world item. AR content is then generated based on the distorted three-dimensional model.

## Claims

### Claim 1

A computer-implemented method comprising:
determining relative positions of a plurality of points on a real-world item;
distorting a three-dimensional model of a virtual item comprising a visual appearance for the real-world item, wherein the distorting is based on the relative positions of the plurality of points on the real-world item, and the distorting produces a distorted three-dimensional model of the virtual item; and
generating augmented reality (AR) content for presentation, the AR content being based on the distorted three-dimensional model.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein generating the AR content includes rendering the distorted three-dimensional model.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein:
the distorted three-dimensional model is rendered from a viewpoint corresponding to a captured image of the real-world item; and
generating the AR content further includes compositing the rendering with the image to overlay the real-world item.

### Claim 4 (depends on 2)

The computer-implemented method of claim 2, further comprising:
determining a position of a user relative to the real-world item, wherein the distorted three-dimensional model is rendered based on the position of the user.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the distorting of the three-dimensional model of the virtual item based on the relative positions of the plurality of points includes modifying the three-dimensional model to reposition points in the three-dimensional model to be in relative positions corresponding to the relative positions of the plurality of points.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the real-world item includes a flexible surface.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein the real-world item is or includes a garment.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein:
the real-world item comprises a plurality of markers, each of the plurality of markers being at a respective different one of the plurality of points on the real-world item; and
determining the relative positions of the plurality of points on the real-world item comprises determining relative positions of the plurality of markers.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein the plurality of markers comprises optical markers, the method further comprising:
obtaining at least one image of the real-world item, wherein determining the relative positions of the plurality of markers includes performing image analysis on the at least one image to detect the optical markers.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein determining the relative positions of the plurality of points on the real-world item comprises:
obtaining measurements of the real-world item;
inputting the measurements into a machine learning (ML) model; and
obtaining an output of the ML model, wherein the relative positions of the plurality of points on the real-world item are based on the output of the ML model.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein:
the relative positions of the plurality of points are first relative positions of the plurality of points, the distorted model of the virtual item is a first distorted model of the virtual item, and the AR content is first AR content;
the first relative positions of the plurality of points correspond to the real-world item in a first shape; and
the method further comprises:
determining second relative positions of the plurality of points on the real-world item, the second relative positions corresponding to the real-world item in a second shape;
distorting, based on the second relative positions of the plurality of points, the three-dimensional model of the virtual item to produce a second distorted three-dimensional model of the virtual item; and
generating second AR content for presentation, the second AR content being based on the second distorted three-dimensional model.

### Claim 12

(canceled)

### Claim 13 (depends on 1)

The computer-implemented method of claim 1, wherein:
the relative positions of the plurality of points on the real-world item are based on measurements obtained from a device; and
the AR content is for presentation at that device.

### Claim 14 (depends on 1)

The computer-implemented method of claim 1, wherein the three-dimensional model of the virtual item is a first three-dimensional model of the virtual item, the relative positions of the plurality of points are first relative positions of the plurality of points, and the AR content is first AR content, the method further comprising:
obtaining an instruction to replace the first three-dimensional model of the virtual item with a second three-dimensional model of the virtual item, the instruction originating from user input at a device;
determining second relative positions of the plurality of points on the real-world item;
distorting, based on the second relative positions of the plurality of points, the second three-dimensional model of the virtual item to produce a distorted second three-dimensional model of the virtual item; and
generating second AR content for presentation, the second AR content being based on the distorted second three-dimensional model.

### Claim 15

A system comprising:
memory to store relative positions of a plurality of points on a real-world item; and
at least one processor to:
distort a three-dimensional model of a virtual item comprising a visual appearance for the real-world item, wherein the distorting is based on the relative positions of the plurality of points on the real-world item, and the distorting produces a distorted three-dimensional model of the virtual item; and
generate augmented reality (AR) content for presentation, the AR content being based on the distorted three-dimensional model.

### Claim 16 (depends on 15)

The system of claim 15, wherein:
the at least one processor is further to render the distorted three-dimensional model; and
the AR content includes the render.

### Claim 17 (depends on 16)

The system of claim 16, wherein:
the distorted three-dimensional model is rendered from a viewpoint corresponding to a captured image of the real-world item; and
the AR content includes the render com posited with the image to overlay the real-world item.

### Claim 18 (depends on 15)

The system of claim 15, wherein the distorted three-dimensional model comprises points repositioned to be in relative positions corresponding to the relative positions of the plurality of points.

### Claim 19 (depends on 15)

The system of claim 15, wherein:
the real-world item comprises a plurality of markers, each of the plurality of markers being at a respective different one of the plurality of points on the real-world item; and
the at least one processor is further to determine relative positions of the plurality of markers.

### Claim 20 (depends on 19)

The system of claim 19, wherein:
the plurality of markers comprises optical markers; and
the at least one processor is further to perform image analysis on at least one image of the real-world item to detect the optical markers.

### Claim 21 (depends on 15)

The system of claim 15, wherein the at least one processor is further to:
obtain measurements of the real-world item;
input the measurements into a machine learning (ML) model; and
obtain an output of the ML model, wherein the relative positions of the plurality of points on the real-world item are based on the output of the ML model.

### Claim 22 (depends on 15)

The system of claim 15, wherein:
the relative positions of the plurality of points are first relative positions of the plurality of points, the distorted model of the virtual item is a first distorted model of the virtual item, and the AR content is first AR content;
the first relative positions of the plurality of points correspond to the real-world item in a first shape;
the memory is further to store second relative positions of the plurality of points on the real-world item, the second relative positions corresponding to the real-world item in a second shape; and
the at least one processor is further to:
distort, based on the second relative positions of the plurality of points, the three-dimensional model of the virtual item to produce a second distorted three-dimensional model of the virtual item; and
generate second AR content for presentation, the second AR content being based on the second distorted three-dimensional model.

### Claim 23 (depends on 15)

The system of claim 15, wherein:
the relative positions of the plurality of points on the real-world item are based on measurements obtained from a device; and
the AR content is for presentation at that device.

### Claim 24 (depends on 15)

The system of claim 15, wherein the three-dimensional model of the virtual item is a first three-dimensional model of the virtual item, the relative positions of the plurality of points are first relative positions of the plurality of points, the AR content is first AR content, and the at least one processor is further to:
obtain an instruction to replace the first three-dimensional model of the virtual item with a second three-dimensional model of the virtual item, the instruction originating from user input at a device;
the memory is further to store second relative positions of the plurality of points on the real-world item; and
the at least one processor is further to:
distort, based on the second relative positions of the plurality of points, the second three-dimensional model of the virtual item to produce a distorted second three-dimensional model of the virtual item; and
generate second AR content for presentation, the second AR content being based on the distorted second three-dimensional model.

### Claim 25

A non-transitory computer readable medium storing computer executable instructions which, when executed by a computer, cause the computer to perform a method comprising:
determining relative positions of a plurality of points on a real-world item;
distorting a three-dimensional model of a virtual item comprising a visual appearance for the real-world item, wherein the distorting is based on the relative positions of the plurality of points on the real-world item, and the distorting produces a distorted three-dimensional model of the virtual item; and
generating augmented reality (AR) content for presentation, the AR content being based on the distorted three-dimensional model.