# US 2024/0202940 A1 · Feature Detection for Image-Based Augmented Reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202940 A1
- Document ID: US20240202940A1
- Status: Published application
- Kind: A1
- Application number: 18/084,606
- Filed: 2022-12-20
- Published: 2024-06-20
- Earliest claimed: 2022-12-20
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Chang Xiao; Ryan Rossi; Enyu Cai
- Web page: https://patentgrep.com/patent/US20240202940A1

## CPC

- G06T 7/248: {involving reference images or patches}
- G06T 7/215: Motion-based segmentation
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/74: {involving reference images or patches}
- G06T 15/205: {Image-based rendering}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30244: Camera pose
- G06T 2210/56: Particle system, point based geometry or rendering
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Certain aspects and features of this disclosure relate to providing a hybrid approach for camera pose estimation using a deep learning-based image matcher and a match refinement procedure. The image matcher takes an image pair as an input and estimates coarse point-to-point feature matches between the two images. The coarse point-to-point feature matches can be filtered based on a stability threshold to produce high-stability point-to-point matches. A perspective-n-point (PnP) camera pose for each frame of video, including one or more added digital visual elements can be computed using the high-stability matches and video frames can be rendered, each using its computed camera pose.

## Claims

### Claim 1

A method comprising:
accessing a template image and a content image for a current frame of video including at least one digital visual element;
estimating coarse point-to-point feature matches between the template image and the content image for the current frame using a trained, convolutional, graph neural network;
filtering the coarse point-to-point feature matches based on a stability threshold to produce high-stability point-to-point matches;
computing, based on the high-stability point-to-point matches, a perspective-n-point (PnP) camera pose for the current frame including the at least one digital visual element; and
rendering the current frame of video with the at least one digital visual element using the PnP camera pose.

### Claim 2 (depends on 1)

The method of claim 1, wherein the stability threshold for a point-to-point match of the coarse point-to-point feature matches comprises a number of frames or a duration of video over which the point-to-point match persists.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
using matching information from a prior frame of video including the at least one digital visual element to produce a mask for screen segmentation of the current frame of video; and
padding the mask with added pixels to compensate for movement between the current frame of video and the prior frame of video.

### Claim 4 (depends on 3)

The method of claim 3, wherein computing the PNP camera pose further comprises:
producing a three-dimensional point cloud based on the current frame and the prior frame of video; and
determining the PnP camera pose relative to the three-dimensional point cloud.

### Claim 5 (depends on 1)

The method of claim 1, wherein estimating the coarse point-to-point feature matches further comprises determining a feature point location and a descriptor for each coarse point-to-point feature match using the trained, convolutional, graph neural network.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
accessing a dataset including a plurality of target images;
applying a random homography to each of the plurality of target images in the dataset to produce a ground truth for each of the plurality of target images; and
training a convolutional neural network based on the ground truth for each of the plurality of target images to produce the trained, convolutional, graph neural network.

### Claim 7 (depends on 6)

The method of claim 6, further comprising deploying the trained, convolutional, graph neural network and the template image to an end-user computing device.

### Claim 8

A system comprising:
a memory component; and
a processing device coupled to the memory component, the processing device to perform operations comprising:
estimating, using a trained, convolutional, graph neural network, coarse point-to-point feature matches between a template image and a content image for each frame of video in a video stream including a digital visual element;
filtering the coarse point-to-point feature matches based on a stability threshold to produce high-stability point-to-point matches;
computing, based on the high-stability point-to-point matches, a perspective-n-point (PnP) camera pose for each frame of video including the digital visual element; and
rendering the video stream with the digital visual element using the PnP camera pose as computed for each frame of video.

### Claim 9 (depends on 8)

The system of claim 8, wherein the stability threshold for a point-to-point match of the coarse point-to-point feature matches comprises a number of frames or a duration of video over which the point-to-point match persists.

### Claim 10 (depends on 9)

The system of claim 9, wherein the operations further comprise:
using matching information from a prior frame of video including the digital visual element to produce a mask for screen segmentation of a current frame of video; and
padding the mask with added pixels to compensate for movement between the current frame of video and the prior frame of video.

### Claim 11 (depends on 9)

The system of claim 9, wherein computing the PNP camera pose further comprises:
producing a three-dimensional point cloud based on a current frame video and a prior frame of video; and
determining the PnP camera pose relative to the three-dimensional point cloud.

### Claim 12 (depends on 8)

The system of claim 8, wherein estimating the coarse point-to-point feature matches further comprises determining a feature point location and a descriptor for each coarse point-to-point feature match using the trained, convolutional, graph neural network.

### Claim 13 (depends on 8)

The system of claim 8, wherein the operations further comprise:
accessing a dataset including a plurality of target images;
applying a random homography to each of the plurality of target images in the dataset to produce a ground truth for each of the plurality of target images; and
training a convolutional neural network based on the ground truth for each of the plurality of target images to produce the trained, convolutional, graph neural network.

### Claim 14 (depends on 13)

The system of claim 13, wherein the operations further comprise deploying the trained, convolutional, graph neural network and the template image to an end-user computing device.

### Claim 15

A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
accessing a template image and a content image for a current frame of video including at least one digital visual element;
a step for producing, using a trained, convolutional, graph neural network, high-stability point-to-point matches between the template image and the content image for the current frame of video;
computing, based on the high-stability point-to-point matches, a perspective-n-point (PnP) camera pose for the current frame of video including the at least one digital visual element; and
rendering the current frame of video with the at least one digital visual element using the PnP camera pose.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the high-stability point-to-point matches are defined in part by a stability threshold comprising a number of frames or a duration of video over which a point-to-point match persists.

### Claim 17 (depends on 16)

The non-transitory computer-readable medium of claim 16, wherein the operations further comprise:
using matching information from a prior frame of video including the at least one digital visual element to produce a mask for screen segmentation of the current frame of video; and
padding the mask with added pixels to compensate for movement between the current frame of video and the prior frame of video.

### Claim 18 (depends on 16)

The non-transitory computer-readable medium of claim 16, wherein computing the PNP camera pose further comprises:
producing a three-dimensional point cloud based on the current frame and a prior frame of video; and
determining the PnP camera pose relative to the three-dimensional point cloud.

### Claim 19 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the operations further comprise:
accessing a dataset including a plurality of target images;
applying a random homography to each of the plurality of target images in the dataset to produce a ground truth for each of the plurality of target images; and
training a convolutional neural network based on the ground truth for each of the plurality of target images to produce the trained, convolutional, graph neural network.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein the operations further comprise deploying the trained, convolutional, graph neural network and the template image to an end-user computing device.
