# US 2021/0174087 A1 · System and method for hybrid visual searches and augmented reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174087 A1
- Document ID: US20210174087A1
- Status: Published application
- Kind: A1
- Application number: 17/113,377
- Filed: 2020-12-07
- Published: 2021-06-10
- Earliest claimed: 2019-12-05
- Assignees: W.W. Grainger, Inc.
- Applicants: W.W. Grainger, Inc.
- Inventors: Ojasve Srikanth; Fouad Bousetouane; Nirav Saraiya
- Web page: https://patentgrep.com/patent/US20210174087A1

## CPC

- G06V 20/20: in augmented reality scenes
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 20/647: {by matching two-dimensional images to three-dimensional objects}
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06T 2219/2016: Rotation, translation, scaling
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A search processing system receives a search query having information indicative of an object of interest, such as an image. The search processing system uses the received search query to locate data for use in creating a computer-generated model of a product determined to be a match for the object of interest. A display generating device uses the data to create the computer-generated model of the product and causes the computer-generated model of the product to be displayed a display associated with the display generating device and/or to be projected onto a surface whereby the computer-generated model of the product can be accurately compared with an image of the object of interest or the object of interest itself

## Claims

### Claim 1

A computer-implemented method, comprising:
receiving by a search processing system a search query having information indicative of an object of interest;
using by the search processing system the received search query to locate data for use in creating a computer-generated model of a product determined by the search processing system to be a match for the object of interest;
using by a display generating device in communication with the search processing system the data to create the computer-generated model of the product; and
causing the computer-generated model of the product to be displayed by the display generating device in a display associated with the display generating device and/or to be projected onto a surface whereby the computer-generated model of the product can be accurately compared with the object of interest.

### Claim 2 (depends on 1)

The method as recited in claim 1, wherein an image of the object of interest is displayed in the display with the computer-generated model.

### Claim 3 (depends on 2)

The method as recited in claim 2, wherein the image of the object of interest comprises the information indictive of the object of interest received by the search processing system.

### Claim 4 (depends on 1)

The method as recited in claim 1, wherein the information indictive of the object of interest comprises an image of the object of interest captured by an image capturing element.

### Claim 5 (depends on 1)

The method as recited in claim 1, wherein the display generating device comprises the image capturing element.

### Claim 6 (depends on 5)

The method as recited in claim 5, wherein the display generating device comprises a smart phone or tablet computing device.

### Claim 7 (depends on 1)

The method as recited in claim 1, wherein display generating device uses information associated with a capturing of an image of the object of interest to scale the computer-generated model of the product to the object of interest so that the computer-generated model of the product can be accurately compared with the object of interest.

### Claim 8

A non-transitory, computer-readable media having stored thereon instructions, the instructions, when executing by a display generating device, causing the display generating device to perform steps comprising:
capturing an image of an object of interest;
providing at least a portion of the image of the object of interest to a search processing system to allow the search processing system to locate data for use in creating a computer-generated model of a product determined by the search processing system to be a match for the object of interest;
receiving from the search processing system the data;
using the data to create the computer-generated model of the product determined by the search processing system to be a match for the object of interest;
causing the computer-generated model of the product to be displayed in a display associated with the display generating device whereby the computer-generated model of the product can be accurately compared with the object of interest.

### Claim 9 (depends on 8)

The non-transitory, computer-readable media method as recited in claim 8, wherein the computer-generated model of the product comprises a three-dimensional image that is capable of being repositioned and reoriented when displayed in the display associated with the display generating device.

### Claim 10 (depends on 8)

The non-transitory, computer-readable media method as recited in claim 8, wherein the captured image of the object of interest is displayed in the display with the computer-generated model.

### Claim 11 (depends on 8)

The non-transitory, computer-readable media method as recited in claim 8, wherein the display generating device comprises a smart phone or tablet computing device.

### Claim 12 (depends on 8)

The non-transitory, computer-readable media method as recited in claim 8, wherein the instructions, when executing by the display generating device, cause the display generating device to pre-process the at least the portion of the image of the object of interest prior to providing the at least the portion of the image of the object of interest to the search processing system.

### Claim 13 (depends on 8)

The non-transitory, computer-readable media method as recited in claim 8, wherein the instructions, when executing by the display generating device, cause the display generating device to use information associated with a capturing of the image of the object of interest to scale the computer-generated model of the product to the object of interest so that the computer-generated model of the product can be accurately compared with the object of interest.

### Claim 14

A non-transitory, computer-readable media having stored thereon instructions, the instructions, when executing by a display generating device, causing the display generating device to perform steps comprising:
capturing an image of an object of interest;
providing at least a portion of the image of the object of interest to a search processing system to allow the search processing system to locate data for use in creating a computer-generated model of a product determined by the search processing system to be a match for the object of interest;
receiving from the search processing system the data;
using the data to create the computer-generated model of the product determined by the search processing system to be a match for the object of interest;
causing the computer-generated model of the product to be projected onto a surface whereby the computer-generated model of the product can be accurately compared with the object of interest.

### Claim 15 (depends on 14)

The non-transitory, computer-readable media method as recited in claim 14, wherein the computer-generated model of the product comprises a three-dimensional image that is capable of being repositioned and reoriented when projected onto the surface.

### Claim 16 (depends on 14)

The non-transitory, computer-readable media method as recited in claim 14, wherein the display generating device comprises a smart phone or tablet computing device.

### Claim 17 (depends on 14)

The non-transitory, computer-readable media method as recited in claim 14, wherein the instructions, when executing by the display generating device, cause the display generating device to pre-process the at least the portion of the image of the object of interest prior to providing the at least the portion of the image of the object of interest to the search processing system.

### Claim 18 (depends on 14)

The non-transitory, computer-readable media method as recited in claim 14, wherein the instructions, when executing by the display generating device, cause the display generating device to use information associated with projecting the model of the product onto the surface to scale the computer-generated model of the product to the object of interest so that the computer-generated model of the product can be accurately compared with the object of interest.
