# US 2024/0202935 A1 · Method and system for providing augmented reality object tracking service based on deep learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202935 A1
- Document ID: US20240202935A1
- Status: Published application
- Kind: A1
- Application number: 18/539,841
- Filed: 2023-12-14
- Published: 2024-06-20
- Earliest claimed: 2022-12-14
- Assignees: Virnect Inc.
- Applicants: Virnect Co.,Ltd.
- Inventors: Ki Young KIM; Noh Young Park
- Web page: https://patentgrep.com/patent/US20240202935A1

## CPC

- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/251: {involving models}
- G06T 7/344: {involving models}
- G06T 7/55: from multiple images
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G06T 2219/2021: Shape modification
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method for providing an AR object tracking service based on deep learning according to an embodiment of the present disclosure, by which a tracking application executed by at least one processor of a terminal provides an AR object tracking service based on deep learning, comprises obtaining first image data; inputting the first image data to a first deep learning neural network; obtaining 3D depth data including each descriptor of a target object within the first image data and a distance value corresponding to the descriptor from the first deep learning neural network; and performing AR object tracking based on the 3D depth data, wherein the first deep learning neural network is a deep learning algorithm performing monocular depth estimation.

## Claims

### Claim 1

A method for providing an AR object tracking service based on deep learning, by which a tracking application executed by at least one processor of a terminal provides an AR object tracking service based on deep learning, the method comprising:
obtaining first image data;
inputting the first image data to a first deep learning neural network;
obtaining 3D depth data including each descriptor of a target object within the first image data and a distance value corresponding to the descriptor from the first deep learning neural network; and
performing AR object tracking based on the 3D depth data,
wherein the first deep learning neural network is a deep learning algorithm performing monocular depth estimation.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
inputting the first image data to a second deep learning neural network and obtaining object area information representing the area occupied by each object within the first image data from the second deep learning neural network,
wherein the second deep learning neural network is a deep learning algorithm performing semantic segmentation.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
determining the area of the target object based on the object area information.

### Claim 4 (depends on 3)

The method of claim 3, wherein the determining of the target object area includes:
detecting at least one target object candidate area based on the object area information and
determining one of the target object candidate areas as a target object area.

### Claim 5 (depends on 3)

The method of claim 3, further comprising:
obtaining the 3D depth data based on the area of the target object.

### Claim 6 (depends on 2)

The method of claim 2, further comprising:
operating the first deep learning neural network and the second deep learning neural network in parallel.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
obtaining the 3D depth data based on a primitive model with a preconfigured 2D or 3D shape.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
generating 3D integrated depth data combining the 3D depth data obtained based on the primitive model and the 3D depth data obtained based on the first deep learning neural network and
performing AR object tracking based on the 3D integrated depth data.

### Claim 9

A system for providing an AR object tracking service based on deep learning comprising:
at least one memory storing a tracking application; and
at least one processor providing an AR object tracking service based on deep learning by reading the tracking application stored in the memory,
wherein commands of the tracking application include commands for performing:
obtaining first image data,
inputting the first image data to a first deep learning neural network,
obtaining 3D depth data including each descriptor of a target object within the first image data and a distance value corresponding to the descriptor from the first deep learning neural network, and
performing AR object tracking based on the 3D depth data.

### Claim 10 (depends on 9)

The system of claim 9, wherein the commands of the tracking application further comprise commands for performing:
generating a 3D definition model based on the 3D depth data,
determining a target virtual object to be augmented and displayed based on the target object,
generating an AR environment model by anchoring the target virtual object and the 3D definition model, and
performing the AR object tracking based on the AR environment model.
