# US 2018/0322703 A1 · Method of controlling a device for generating an augmented reality environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0322703 A1
- Document ID: US20180322703A1
- Status: Published application
- Kind: A1
- Application number: 15/773,274
- Filed: 2016-11-27
- Published: 2018-11-08
- Earliest claimed: 2015-12-02
- Assignees: Devar Entertainment Limited
- Applicants: Devar Entertainment Limited
- Inventors: Vitalii Vitalievich AVERIANOV; Andrei Valerievich KOMISSAROV
- Web page: https://patentgrep.com/patent/US20180322703A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06K 7/1417: {2D bar codes}
- G06K 19/06037: {multi-dimensional coding}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 20/20: in augmented reality scenes
- G06V 30/144: using a slot moved over the image; using discrete sensing elements at predetermined points; using automatic curve following means
- G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The invention relates to processing and generating image data and analysis and displaying a 3D image. A method of controlling a device for generating an augmented reality consists in receiving object-related data from sensors, recognizing the object and generating a virtual object, the authenticity of an activation code that identifies a set of pictures containing objects is pre-verified, data related to an object obtained in the form of photo or video frame(s), the object is identified on said frame(s) by comparison with images stored in the memory of the user device, and a virtual object is generated in the form of a 3D model, reproduced in real time, on a display of the user device, on top of the acquired frames. Wherein the object is a two-dimensional image of at least one item depicted on a picture contained in a set to which an activation code is assigned.

## Claims

### Claim at position 1

-10. (canceled)

### Claim 11

A method for generating an augmented reality environment using a user device, comprising:
acquiring an activation code for identifying a set of pictures containing objects,
communicating said activation code to a server for authentication,
if the activation code is authenticated on the server, generating an activation command for forming the augmented reality environment and transmitting said activation code to the user device,
in response to the activation code, acquiring data associated with an object from user device sensors, said data associated with the object being acquired in a form of at least one photo or video image frame of the object,
recognizing the object, wherein object recognition is performed on the photo or video image frame by comparing and matching with images of the object stored in a memory of the user device,
generating a virtual object as a 3D model,
reproducing an image of said 3D model, in real time by display means of the user device, over the at least one acquired photo or video image frame of the object,
wherein the object, in relation to the 3D model, is a two-dimensional image of at least one object represented in the set of pictures associated with the activation code.

### Claim 12 (depends on 11)

The method of claim 11, wherein the activation code is a code word containing a sequence of letters and/or digits.

### Claim 13 (depends on 11)

The method of claim 11, wherein the activation code is a two-dimensional image that can be scanned using a photo or video camera.

### Claim 14 (depends on 11)

The method of claim 11, wherein in the presence of more than one acquired photo or video image frame, a selection is made based upon acceptable image quality.

### Claim 15 (depends on 11)

The method of claim 11, wherein the 3D model generated in response to the object recognition is context related to said object.

### Claim 16 (depends on 11)

The method of claim 11, wherein the 3D model is formed in color, the method further comprising:
forming a transformation matrix of image coordinates into its own, namely Cartesian coordinates, characterized by an orthogonality of axes,
coloring polygons of the 3D model such that colors of the 3D model polygons correspond with colors of corresponding photo elements by forming a texture of a scanned area by using the transformation matrix and interpolation of data with a subsequent assignment texture to the 3D model, such that the corresponding polygons are covered by the corresponding textures based upon coordinates preformed at a texturizing stage.

### Claim 17 (depends on 11)

The method of claim 11, wherein the 3D model is formed in color, further comprising:
forming a transformation matrix of image coordinates into its own, namely Cartesian coordinates, characterized by an orthogonality of axes,
coloring polygons of the 3D model in such a way that colors of the 3D model polygons correspond with colors of corresponding photo elements by determining colors of materials of the 3D model based on a color reading at predetermined points of the acquired photo or video image frame using the coordinate transformation matrix, and
assigning colors to corresponding materials of the 3D model.

### Claim 18 (depends on 11)

The method of claim 11, wherein the 3D model is formed in color, further comprising:
forming a transformation matrix of image coordinates into its own, namely Cartesian coordinates, characterized by an orthogonality of axes,
coloring polygons of the 3D model such that colors of the 3D model polygons correspond with colors of corresponding photo elements by creating a texture, said texture comprising areas of the acquired photo or video image frame, said texture also comprising areas filled with solid colors taken at predefined points of the acquired photo or video image frame, using the coordinate transformation matrix.

### Claim 19 (depends on 18)

The method of claim 18, wherein, while the texture is created for the 3D model, predetermined regions are painted by filling adjacent pixels with colors to remove contours of an object being painted in a resulting painting.

### Claim 20 (depends on 18)

The method of claim 18, wherein a map of shadows is applied to the texture created for the 3D model.
