# US 11,308,706 B2 · Local augmented reality persistent sticker objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,706 B2
- Document ID: US11308706B2
- Status: Granted
- Kind: B2
- Application number: 16/927,273
- Filed: 2020-07-13
- Published: 2020-12-31
- Granted: 2022-04-19
- Earliest claimed: 2016-01-29
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Jia Li; Linjie Luo; Rahul Bhupendra Sheth; Ning Xu; Jianchao Yang
- Web page: https://patentgrep.com/patent/US11308706B2

## CPC

- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 7/248: {involving reference images or patches}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04L 51/10: Multimedia information
- H04L 51/52: for supporting social networking services
- G06T 2207/20104: Interactive definition of region of interest [ROI]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Systems and methods for local augmented reality (AR) tracking of an AR object are disclosed. In one example embodiment a device captures a series of video image frames. A user input is received at the device associating a first portion of a first image of the video image frames with an AR sticker object and a target. A first target template is generated to track the target across frames of the video image frames. In some embodiments, global tracking based on a determination that the target is outside a boundary area is used. The global tracking comprises using a global tracking template for tracking movement in the video image frames captured following the determination that the target is outside the boundary area. When the global tracking determines that the target is within the boundary area, local tracking is resumed along with presentation of the AR sticker object on an output display of the device.

## Claims

### Claim 1

A method comprising:
processing a user input associating a first portion of a first image of video image frames captured by an image sensor with an AR sticker object and a target;
generating, based on the user input and the first portion of the first image, a first target template associated with the target;
performing local tracking of the target across frames of the video image frames following the first image by calculating changes in the first portion of the first image using the first target template;
determining that the target is outside a boundary area with respect to the local tracking;
initiating, in response to the determining, a global tracking comprising using a global tracking template for tracking movement in the video image frames captured following the determining; and
resuming the local tracking of the target when the global tracking determines that the target is within the boundary area, and displaying the AR sticker object based on the local tracking of the target.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
generating a local AR model of a scene corresponding to the video image frames, wherein performing the local tracking is based on the local AR model.

### Claim 3 (depends on 2)

The method of claim 2, wherein displaying the AR sticker object is based on adding the AR sticker object to the local AR model.

### Claim 4 (depends on 2)

The method of claim 2, wherein the local AR model is generated in response to the user input, which places the AR sticker object relative to the first image.

### Claim 5 (depends on 4)

The method of claim 4, wherein generating the local AR model of the scene comprises:
identifying, based on the placement of the AR sticker object, the target within the first image.

### Claim 6 (depends on 1)

The method of claim 1, wherein tracking the target across the frames of the video image frames following the first image by calculating the changes in the first portion of the first image using the first target template comprises:
identifying the first target template associated with the AR sticker object described by Ji=I0(si);
wherein Ji is the first target template associated with the AR sticker object, the first target template comprising a set of color values sampled at a plurality of sample points Si associated with the target and a target area in an initial AR model image I0 of one or more images of the video image frames.

### Claim 7 (depends on 6)

The method of claim 6, further comprising tracking the AR sticker object by:
processing a second plurality of images received after the initial AR model image such that:
Ji(l)=In(T(si))
wherein T(n) is a transform describing movement of the target between sequential images of the second plurality of images; and
calculating a local tracking energy to determine an updated target template for each image of the second plurality of images according to an iterative nonlinear optimization:
T
n
+
1
=
arg
⁢
⁢
min
⁢
T
n
+
1
⁢
∑
i
⁢
⁢

I
n
+
1
⁡
(
T
n
+
1
⁡
(
s
i
)
)
-
J
i

2
;
such
⁢
⁢
that
T
n
+
1
=
arg
⁢
⁢
min
⁢
T
n
+
1
⁢
1
=
4
,
3
⁢
∑
i
⁢
⁢

I
n
+
1
(
l
)
⁡
(
T
n
+
1
⁡
(
s
i
)
)
-
J
i
(
l
)

2
;
and
T
n
+
1
=
arg
⁢
⁢
min
⁢
T
n
⁢
1
=
2
,
1
,
0
⁢
∑
i
⁢
⁢

I
n
+
1
(
R
)
⁡
(
T
n
+
1
⁡
(
s
i
)
)
-
J
i
(
l
)

2
are used to determine the updated target template associated with each image of the second plurality of images.

### Claim 8

A system comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the processor to perform operations comprising:
processing a user input associating a first portion of a first image of video image frames captured by an image sensor with an AR sticker object and a target;
generating, based on the user input and the first portion of the first image, a first target template associated with the target;
performing local tracking of the target across frames of the video image frames following the first image by calculating changes in the first portion of the first image using the first target template;
determining that the target is outside a boundary area with respect to the local tracking;
initiating, in response to the determining, a global tracking comprising using a global tracking template for tracking movement in the video image frames captured following the determining; and
resuming the local tracking of the target when the global tracking determines that the target is within the boundary area, and displaying the AR sticker object based on the local tracking of the target.

### Claim 9 (depends on 8)

The system of claim 8, the operations further comprising:
generating a local AR model of a scene corresponding to the video image frames, wherein performing the local tracking is based on the local AR model.

### Claim 10 (depends on 9)

The system of claim 9, wherein displaying the AR sticker object is based on adding the AR sticker object to the local AR model.

### Claim 11 (depends on 9)

The system of claim 9, wherein the local AR model is generated in response to the user input, which places the AR sticker object relative to the first image.

### Claim 12 (depends on 11)

The system of claim 11, wherein generating the local AR model of the scene comprises:
identifying, based on the placement of the AR sticker object, the target within the first image.

### Claim 13 (depends on 8)

The system of claim 8, wherein tracking the target across the frames of the video image frames following the first image by calculating the changes in the first portion of the first image using the first target template comprises:
identifying the first target template associated with the AR sticker object described by Ji=I0(si);
wherein Ji is the first target template associated with the AR sticker object, the first target template comprising a set of color values sampled at a plurality of sample points Si associated with the target and a target area in an initial AR model image I0 of one or more images of the video image frames.

### Claim 14 (depends on 13)

The system of claim 13, the operations further comprising tracking the AR sticker object by:
processing a second plurality of images received after the initial AR model image such that:
Ji(l)=In(T(si))
wherein T(n) is a transform describing movement of the target between sequential images of the second plurality of images; and
calculating a local tracking energy to determine an updated target template for each image of the second plurality of images according to an iterative nonlinear optimization:
T
n
+
1
=
arg
⁢
⁢
min
⁢
T
n
+
1
⁢
∑
i
⁢
⁢

I
n
+
1
⁡
(
T
n
+
1
⁡
(
s
i
)
)
-
J
i

2
;
such
⁢
⁢
that
T
n
+
1
=
arg
⁢
⁢
min
⁢
T
n
⁢
1
=
4
,
3
⁢
∑
i
⁢
⁢

I
n
+
1
(
l
)
⁡
(
T
n
+
1
⁡
(
s
i
)
)
-
J
i
(
l
)

2
;
and
T
n
+
1
=
arg
⁢
⁢
min
⁢
T
n
⁢
1
=
2
,
1
,
0
⁢
∑
i
⁢
⁢

I
n
+
1
(
R
)
⁡
(
T
n
+
1
⁡
(
s
i
)
)
-
J
i
(
l
)

2
are used to determine the updated target template associated with each image of the second plurality of images.

### Claim 15

A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising:
processing a user input associating a first portion of a first image of video image frames captured by an image sensor with an AR sticker object and a target;
generating, based on the user input and the first portion of the first image, a first target template associated with the target;
performing local tracking of the target across frames of the video image frames following the first image by calculating changes in the first portion of the first image using the first target template;
determining that the target is outside a boundary area with respect to the local tracking;
initiating, in response to the determining, a global tracking comprising using a global tracking template for tracking movement in the video image frames captured following the determining; and
resuming the local tracking of the target when the global tracking determines that the target is within the boundary area, and displaying the AR sticker object based on the local tracking of the target.

### Claim 16 (depends on 15)

The computer-readable storage medium of claim 15, the operations further comprising:
generating a local AR model of a scene corresponding to the video image frames, wherein performing the local tracking is based on the local AR model.

### Claim 17 (depends on 16)

The computer-readable storage medium of claim 16, wherein displaying the AR sticker object is based on adding the AR sticker object to the local AR model.

### Claim 18 (depends on 16)

The computer-readable storage medium of claim 16, wherein the local AR model is generated in response to the user input, which places the AR sticker object relative to the first image.

### Claim 19 (depends on 18)

The computer-readable storage medium of claim 18, wherein generating the local AR model of the scene comprises:
identifying, based on the placement of the AR sticker object, the target within the first image.

### Claim 20 (depends on 15)

The computer-readable storage medium of claim 15, wherein tracking the target across the frames of the video image frames following the first image by calculating the changes in the first portion of the first image using the first target template comprises:
identifying the first target template associated with the AR sticker object described by
Ji=I0(si);
wherein Ji is the first target template associated with the AR sticker object, the first target template comprising a set of color values sampled at a plurality of sample points Si associated with the target and a target area.