# US 10,482,674 B1 · System and method for mobile augmented reality > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,674 B1 - Document ID: US10482674B1 - Status: Granted - Kind: B1 - Application number: 16/020,920 - Filed: 2018-06-27 - Granted: 2019-11-19 - Earliest claimed: 2018-06-27 - Assignees: Beijing Jingdong Shangke Information Technology Co., Ltd.; JD.com American Technologies Corporation - Applicants: Beijing Jingdong Shangke Information Technology Co., Ltd.; JD.com American Technologies Corporation - Inventors: Yuzhang Wu; Yi Xu; Dan Miao; Hui Zhou - Web page: https://patentgrep.com/patent/US10482674B1 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06N 20/00: Machine learning - G06Q 30/0276: {Advertisement creation} - G06Q 30/0282: Rating or review of business operators or products - G06Q 30/0643: {graphically representing goods, e.g. 3D product representation} - G06T 17/05: Geographic models - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G09B 25/00: Models for purposes not provided for in G09B23/00, e.g. full-sized devices for demonstration purposes (model vehicles, tracks therefor, models in the nature of toys A63H) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) ## Abstract A system and method for realizing augmented reality. The system includes a visual sensor configured to capture images of an environment and a computing device in communication with the visual sensor. The computing device has a processor and a storage device storing computer executable code. The computer executable code, when executed at the processor, is configured to: identify two-dimensional (2D) objects in the captured images; construct a three-dimensional (3D) map of the environment using the captured images; define 3D objects in the 3D map by mapping the 2D objects identified in the captured images to the corresponding points in the 3D map; and render a 3D model on the 3D map based on the 3D objects defined in the 3D map. ## Claims ### Claim 1 A system for realizing augmented reality, comprising: a visual sensor configured to capture images of an environment; and a computing device in communication with the visual sensor, wherein the computing device comprises a processor and a storage device storing computer executable code, and the computer executable code, when executed at the processor, is configured to: identify two-dimensional (2D) objects in the captured images; construct a three-dimensional (3D) map of the environment using the captured images; define 3D objects in the 3D map by mapping the 2D objects identified in the captured images to corresponding feature points in the 3D map; measure dimension and recognize function and style of the 2D objects; project the 2D objects to the 3D map to define dimension, function and style of the 3D objects in the 3D map corresponding to the 2D objects; recommend a product that is not present in the environment based on the dimension, function and style of the 3D objects; and place and render a 3D model of the product to the 3D map based on the 3D objects defined in the 3D map, wherein the computer executable code is further configured to recommend a product by: determining a plurality of products based on a category of the environment; selecting a first subset of products from the plurality of products based on the style of the 3D objects; selecting a second subset of products from the first subset of products with a function complementary to the function of the 3D objects; and determining at least one product from the second subset of products that has a size fitting an empty space around the 3D objects. ### Claim 2 (depends on 1) The system of claim 1, wherein the computer executable code is further configured to define 3D objects by determining planes of the 3D objects. ### Claim 3 (depends on 2) The system of claim 2, wherein the planes comprising horizontal planes, vertical planes, and planes with different orientations. ### Claim 4 (depends on 2) The system of claim 2, wherein the computer executable code is further configured to: detect collision between the 3D model and the 3D objects defined in the 3D map based on the planes of the 3D objects. ### Claim 5 (depends on 1) The system of claim 1, wherein the computer executable code is further configured to: place the 3D model in a position in the 3D map based on the dimension, function and style of the 3D objects. ### Claim 6 (depends on 1) The system of claim 1, wherein the computer executable code is configured to: identify the 2D objects using an image/point processing module; and render the 3D model on the 3D map using a rendering module, wherein both the image/point processing module and the rendering module are coded using C++. ### Claim 7 (depends on 1) The system of claim 1, wherein the computer executable code is configured to define 3D objects in the 3D map by: determining a plurality of 2D feature points in the captured images that correspond to one of the 2D objects; determining a plurality of 3D feature points in the 3D map that correspond to the determined 2D feature points; and adding an object label to the determined 3D feature points to define 3D objects corresponding to the 2D objects. ### Claim 8 (depends on 1) The system of claim 1, wherein the computer executable code is further configured to: operate the 3D model on the 3D map in response to a gesture or speech of a user. ### Claim 9 (depends on 1) The system of claim 1, wherein the computer executable code is further configured to: measure dimension; and recognize function and style of the 2D objects using machine learning. ### Claim 10 A method for realizing augmented reality using a visual sensor and a computing device in communication with the visual sensor, the method comprising: capturing images of an environment by the visual sensor; identifying, by the computing device, two-dimensional (2D) objects in the captured images; constructing, by the computing device, a three-dimensional (3D) map of the environment using the captured images; defining, by the computing device, 3D objects in the 3D map by mapping the 2D objects identified in the captured images to the corresponding feature points in the 3D map; measuring dimension and recognizing function and style of the 2D objects; projecting the 2D objects to the 3D map to define dimension, function and style of the 3D objects in the 3D map corresponding to the 2D objects; recommending a product that is not present in the environment based on the dimension, function and style of the 3D objects; and placing and rendering, by the computing device, a 3D model of the product to the 3D map based on the 3D objects defined in the 3D map, wherein the step of recommending a product comprises: determining a plurality of products based on a category of the environment; selecting a first subset of products from the plurality of products based on the style of the 3D objects; selecting a second subset of products from the first subset of products with a function complementary to the function of the 3D objects; and determining at least one product from the second subset of products that has a size fitting an empty space around the 3D objects. ### Claim 11 (depends on 10) The method of claim 10, further comprising defining the 3D objects in the 3D map by determining planes of the 3D objects. ### Claim 12 (depends on 11) The method of claim 11, wherein the planes comprising horizontal planes, vertical planes, and planes with different orientations. ### Claim 13 (depends on 11) The method of claim 11, further comprising detecting collision based on the planes. ### Claim 14 (depends on 10) The method of claim 10, further comprising: placing the 3D model in a position in the 3D map based on the dimension, function and style of the 3D objects. ### Claim 15 (depends on 10) The method of claim 10, further comprising measuring dimension; and recognizing function and style of the 2D objects using machine learning. ### Claim 16 A non-transitory computer readable medium storing computer executable code, wherein the computer executable code, when executed at a processor of a computing device, is configured to: receive images captured by a visual sensor; identify two-dimensional (2D) objects in the captured images; construct a three-dimensional (3D) map of the environment using the captured images; define 3D objects in the 3D map by mapping the 2D objects identified in the captured images to the corresponding feature points in the 3D map; measure dimension and recognize function and style of the 2D objects; project the 2D objects to the 3D map to define dimension, function and style of the 3D objects in the 3D map corresponding to the 2D objects; recommend a product that is not present in the environment based on the dimension, function and style of the 3D objects; and place and render a 3D model of the product to the 3D map based on the 3D objects defined in the 3D map, wherein the computer executable code is further configured to recommend a product by: determining a plurality of products based on a category of the environment; selecting a first subset of products from the plurality of products based on the style of the 3D objects; selecting a second subset of products from the first subset of products with a function complementary to the function of the 3D objects; and determining at least one product from the second subset of products that has a size fitting an empty space around the 3D objects. ### Claim 17 (depends on 16) The non-transitory computer readable medium of claim 16, wherein the computer executable code is further configured to: define 3D objects by determining horizontal planes and vertical planes of the 3D objects; and detect collision base on the planes of the 3D objects.