# US 11,307,731 B2 · Computer implemented platform, software, and method for drawing using augmented reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,307,731 B2
- Document ID: US11307731B2
- Status: Granted
- Kind: B2
- Application number: 16/888,760
- Filed: 2020-05-31
- Published: 2020-11-19
- Granted: 2022-04-19
- Earliest claimed: 2019-05-15
- Assignees: Sketchar, Uav
- Applicants: Sketchar, Uab
- Inventors: Andrey Drobitko; Aleksandr Danilin; Mikhail Kopeliovich; Mikhail Petrushan
- Web page: https://patentgrep.com/patent/US11307731B2

## CPC

- G06T 7/74: {involving reference images or patches}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06T 7/248: {involving reference images or patches}
- G06T 7/35: using statistical methods
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/20: in augmented reality scenes
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A computer-implemented method of the present invention allows the user to draw images using augmented reality. The method is implemented by a processor executing a marker-less tracking algorithm stored in a memory of a mobile computing device. The method comprises detecting, by a page detector, an image of a drawing area; initializing a marker-less tracker, wherein said initializing comprises (a) capturing, via the page detector, a frame of the drawing area, (b) displaying, via a graphical user interface (GUI) of the mobile computing device, the frame of the drawing area, and (c) uniformly distributing on the GUI a template patch over the frame of the drawing area, and wherein the template patch is a fragment of texture of a template image used for surface tracking. The method further comprises executing a template patch tracking, wherein a perspective transformation of the template image to the frame as a current image is evaluated in video streaming on the GUI of the mobile computing device.

## Claims

### Claim 1

A non-transitory computer-readable medium storing a code with computer-readable instructions for drawing using augmented reality that when executed by a processor, causes the processor to execute a marker-less tracking algorithm stored in a memory of a mobile computing device to perform the steps of:
detecting, by a page detector, an image of a drawing area;
initializing a marker-less tracker, wherein said initializing comprises (a) capturing, via the page detector, a frame of the drawing area, (b) displaying, via a graphical user interface (GUI) of the mobile computing device, the frame of the drawing area, and (c) uniformly distributing on the GUI a template patch over the frame of the drawing area, wherein the template patch is a fragment of texture of a template image used for surface tracking; and
executing a template patch tracking, wherein a perspective transformation of the template image to the frame as a current image is evaluated in video streaming on the GUI of the mobile computing device;
wherein the processor is configured to detect a correspondence between one or more coordinates of the template image and one or more coordinates of the current image; and wherein the processor is configured to evaluate the perspective transformation based on the correspondence by random sample consensus (RANSAC);
estimate a confidence level of the perspective transformation as a percentage based on the one or more coordinates of the current image having an under-threshold difference from one or more coordinates predicted by said evaluating the perspective transformation by RANSAC;
switch, utilizing the percentage confidence level as a criterion to a re-initialization mode;
update an initial template patch with an initial descriptor to an updated template patch with a new descriptor;
monitor the updated template patch to validate it and responsive to a failure to validate the updated patch, switching back to the initial template patch with the initial descriptor;
enhance the texture of the template image through template patches accumulation based on temporal averaging of binarized patches; and
prepare a rough estimation of the perspective transformation by computing, via an inverse matrix, the perspective transformation from the template image to the current image.

### Claim 2 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the processor is configured to: estimate a confidence level of the perspective transformation after said template patch tracking; and in response to the confidence level being low, switch from said template patch tracking to a re-initialization mode.

### Claim 3 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the processor is configured to use an inverse matrix of the perspective transformation.

### Claim 4 (depends on 3)

The non-transitory computer-readable medium of claim 3, wherein the processor is configured to use inverse matrix to transform, via a rough estimation, the current image of the perspective transformation to the template image.

### Claim 5 (depends on 3)

The non-transitory computer-readable medium of claim 3, wherein the processor is configured to enhance a local texture contrast by performing image binarization with an adaptive local binarization threshold, wherein the local binarization threshold is calculated for each pixel as a function of intensities in a local surrounding region.

### Claim 6 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the processor is configured to detect an out-of-frame or an obscured template patch.

### Claim 7 (depends on 6)

The non-transitory computer-readable medium of claim 6, wherein the processor is configured to filter the out-of-the-frame or obscured template patch; and update a contour of the out-of-the-frame or the obscured template patch, wherein said template patch tracking is based on searching of one or more template patches as the current image transformed by a roughly estimated perspective transformation.