# US 10,319,150 B1 · Object preview in a mixed reality environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,319,150 B1
- Document ID: US10319150B1
- Status: Granted
- Kind: B1
- Application number: 15/595,670
- Filed: 2017-05-15
- Granted: 2019-06-11
- Earliest claimed: 2017-05-15
- Assignees: A9.com, Inc.
- Applicants: A9.com, Inc.
- Inventors: Jason Canada; Rupa Chaturvedi; Jared Corso; Michael Patrick Cutter; Sean Niu; Shaun Michael Post; Peiqi Tang; Stefan Vant; Mark Scott Waldo; Andrea Zehr
- Web page: https://patentgrep.com/patent/US10319150B1

## CPC

- G06T 11/00: Two-dimensional [2D] image generation
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06T 11/60: Creating or editing images; Combining images with text
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/1454: {involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay}
- G06F 3/147: using display panels
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 2210/04: Architectural design, interior design
- G09G 2340/12: Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
- G09G 2350/00: Solving problems of bandwidth in display systems
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)

## Abstract

Users can view images or renderings of items placed (virtually) within a physical space. For example, a rendering of an item can be placed within a live camera view of the physical space. A snapshot of the physical space can be captured and the snapshot can be customized, shared, etc. The renderings can be represented as two-dimensional images, e.g., virtual stickers or three-dimensional models of the items. Users can have the ability to view different renderings, move those items around, and develop views of the physical space that may be desirable. The renderings can link to products offered through an electronic marketplace and those products can be consumed. Further, collaborative design is enabled through modeling the physical space and enabling users to view and move around the renderings in a virtual view of the physical space.

## Claims

### Claim 1

A computing device, comprising:
a camera;
a display screen;
at least one processor;
a memory device including instructions that, when executed by the at least one processor, enable the computing device to:
receive a selection of a user-selectable element configured to activate an augmented reality environment, the user-selectable element associated with a product from a product catalog of an electronic marketplace;
receive first image data that includes a representation of the product;
analyze the first image data to determine a set of features associated with the representation of the product;
recognize edges of the representation of the product based at least in part on the set of features;
generate second image data that includes a modified view of the representation of the product, the modified view including a border of a predetermined width and a drop shadow, the second image data corresponding to a virtual sticker;
capture image data of a first view of a physical space, the physical space corresponding to a field of view of the camera;
display a camera view of the first view of the physical space on the display screen, the camera view including the virtual sticker of the product appearing to be in a first location in the physical space, the product being part of a category associated with the physical space; and
enable purchase of the product from the electronic marketplace.

### Claim 2 (depends on 1)

The computing device of claim 1, wherein the instructions, when executed further enable the computing device to:
capture second image data that includes a second view of the physical space; and
display at least a portion of the first view and the second view, the virtual sticker appearing to be in the first location of the physical space.

### Claim 3

A computing device, comprising:
an image capture element;
a display element;
at least one processor;
a memory device including instructions that, when executed by the at least one processor, enable the computing device to:
capture image data of a physical space using the image capture element, the physical space within a field of view of the image capture element;
analyze the image data to recognize a category associated with the physical space;
determine a set of items of interest based at least in part on the category, the set of items of interest associated with respective renderings;
display the respective renderings of the set of items of interest;
enable interaction with the respective renderings;
display a camera view of the physical space on the display element, the camera view including a rendering of an item of interest of the set of items of interest appearing to be positioned in the physical space, the item of interest being part of the category associated with the physical space; and
enable purchase of the item of interest from an electronic marketplace.

### Claim 4 (depends on 3)

The computing device of claim 3, wherein the instructions, when executed further enable the computing device to:
display a product page that includes the item of interest, the item of interest being a product from a product catalog of an electronic marketplace;
receive a selection of a user-selectable element configured to activate the camera view of an augmented reality environment; and
enable the camera view.

### Claim 5 (depends on 3)

The computing device of claim 3, wherein the instructions, when executed further enable the computing device to:
enable modification of at least one property associated with the rendering, the at least one property including a size of the rendering or a position of the rendering, and wherein the rendering includes a virtual sticker representation of the item of interest.

### Claim 6 (depends on 3)

The computing device of claim 3, wherein the instructions, when executed further enable the computing device to:
receive a selection of a user-selectable element configured to activate a three-dimensional viewing mode;
obtain space dimension data for the physical space;
determine object dimension data and point of view data for an image overlay that includes a three-dimensional representation of the item of interest; and
determine scale and point of view appropriate image data for the physical space, the scale and point of view appropriate image data determined using at least the image data, the point of view data, the space dimension data, and the object dimension data,
wherein the camera view includes the image overlay for the item of interest, with the scale and point of view appropriate image data determined using at least the image data, the point of view data, the space dimension data, and the object dimension data.

### Claim 7 (depends on 3)

The computing device of claim 3, wherein the instructions, when executed further enable the computing device to:
obtain a representation of the item of interest;
analyze the representation of the item of interest to determine an outline of the representation of the item of interest;
add a border of a predetermined width to the representation of the item of interest;
add a drop shadow to the border; and
store an updated representation of the item of interest as the rendering.

### Claim 8 (depends on 3)

The computing device of claim 3, wherein the instructions, when executed further enable the computing device to:
receive a request to view a set of items of interest, the set of items of interest related to the item of interest;
display respective renderings of the set of items of interest; and
enable interaction with the respective renderings.

### Claim 9 (depends on 3)

The computing device of claim 3, wherein the instructions, when executed further enable the computing device to:
receive a search request to view a set of items of interest, the search request including a search term;
determine a set of items of interest based at least in part on the search term, the set of items of interest associated with respective renderings;
display the respective renderings of the set of items of interest; and
enable interaction with the respective renderings.

### Claim 10 (depends on 3)

The computing device of claim 3, wherein the rendering of the item of interest can be at least one of moved within, or removed from, the camera view, or resized.

### Claim 11 (depends on 3)

The computing device of claim 3, wherein the instructions, when executed to provide information further enable the computing device to:
receive user input to change a location of the rendering in the camera view from a first location to a second location; and
display the rendering in the second location in the camera view.

### Claim 12 (depends on 11)

The computing device of claim 11, wherein the instructions, when executed to provide information further enable the computing device to:
receive user input to associate the rendering with a representation of a first area of the physical space corresponding to a first image capture position;
move the image capture element to a second image capture position to capture a second area of the physical space; and
store information associating the rendering with the representation of the first area of the physical space.

### Claim 13 (depends on 12)

The computing device of claim 12, wherein the instructions, when executed to provide information further enable the computing device to:
move the image capture element to the first image capture position; and
display the rendering to appear to be positioned in the representation of the first area of the physical space.

### Claim 14

A method, comprising:
capturing image data of a physical space using an image capture element of a computing device, the physical space within a field of view of the image capture element;
analyzing the image data to recognize a category associated with the physical space;
determining a set of items of interest based at least in part on the category, the set of items of interest associated with respective renderings;
displaying the respective renderings of the set of items of interest;
enabling interaction with the respective renderings;
displaying a camera view of the physical space on a display element of the computing device, the camera view including a rendering of an item of interest of the set of items of interest appearing to be positioned in the physical space, the item of interest being part of the category associated with the physical space; and
enabling purchase of the item of interest from an electronic marketplace.

### Claim 15 (depends on 14)

The method of claim 14, further including:
receiving a request to model the physical space;
causing a set of mapping data to automatically be obtained for a set of views within the physical space, the set of mapping data including at least image data and dimension data, at least one view of the set of views comprising a top view of at least a portion of the physical space;
generating, using the set of mapping data for the set of views, a three-dimensional virtual model of the physical space;
receiving a request for a view of the three-dimensional virtual model, the view corresponding to a specific viewpoint; and
rendering the view of the three-dimensional virtual model from the specific viewpoint and providing the view for display, the view having associated dimension data for the physical space such that the rendering of the item of interest to be placed within the view of the three-dimensional virtual model is able to be performed with a scale corresponding to the associated dimension data.

### Claim 16 (depends on 15)

The method of claim 15, further including:
determining a set of items corresponding to data for at least one of the physical space or a customer of the physical space;
providing data for at least a subset of the set of items for display with the three-dimensional virtual model;
receiving a request relating to a selected item of the set of items;
rendering a view of the selected item using model data for the selected item, the view of the selected item being scale-appropriate for the view of the three-dimensional virtual model; and
displaying the view of the selected item in the view of the three-dimensional virtual model.

### Claim 17 (depends on 14)

The method of claim 14, wherein the rendering includes a virtual sticker, the virtual sticker associated with a border and a drop shadow.

### Claim 18 (depends on 14)

The method of claim 14, further comprising:
displaying a product page that includes the item of interest, the item of interest being a product from a product catalog of an electronic marketplace;
receiving a selection of a user-selectable element configured to activate the camera view of an augmented reality environment; and
enabling the camera view.