# US 2013/0258117 A1 · User-guided object identification

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2013/0258117 A1
- Document ID: US20130258117A1
- Status: Published application
- Kind: A1
- Application number: 13/431,079
- Filed: 2012-03-27
- Published: 2013-10-03
- Earliest claimed: 2012-03-27
- Assignees: Amazon Technologies, Inc.
- Inventors: Francislav P. Penov; Aaron Michael Donsbach; Geoffrey Scott Heller; Kenneth M. Karakotsios; Daniel Bibireata; Kah Kuen Fu; Richard Howard Suplee, III; Timothy Youngjin Sohn
- Web page: https://patentgrep.com/patent/US20130258117A1

## CPC

- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06V 10/235: {based on user input or interaction}
- G06V 10/24: Aligning, centring, orientation detection or correction of the image
- G06V 10/245: {by locating a pattern; Special marks for positioning}
- G06V 20/647: {by matching two-dimensional images to three-dimensional objects}
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A user attempting to obtain information about an object can capture image information including a view of that object, and the image information can be used with a matching or identification process to provide information about that type of object to the user. In order to narrow the search space to a specific category, and thus improve the accuracy of the results and the speed at which results can be obtained, the user can be guided to capture image information with an appropriate orientation. An outline or other graphical guide can be displayed over image information captured by a computing device, in order to guide the user in capturing the object from an appropriate direction and with an appropriate scale for the type of matching and/or information used for the matching. Such an approach enables three-dimensional objects to be analyzed using conventional two-dimensional identification algorithms, among other such processes.

## Claims

### Claim 1

A computer-implemented method of identifying an object, comprising:
receiving, from a user of a computing device, an indication of a category of object to be identified;
causing image information, captured by a camera of the computing device, to be displayed on a display screen of the computing device, the image information including a current view of the object, the current view capable of changing depending at least in part upon a position of the computing device;
causing an orientation guide for the category of object to be displayed with the image information on the display screen;
enabling the user to adjust the position of the computing device with respect to the object of interest until the current view of the object in the captured image information has a similar orientation and size to the orientation guide;
analyzing the captured image information to attempt to match at least a portion of the captured image information to information for a type of object in the category;
prompting the user for additional information for use in further analyzing the captured information when matching object information is unable to be determined with at least a minimum level of confidence; and
providing matching object information for display to the user when the matching object information is able to be determined with at least the minimum level of confidence.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein receiving indication of a category of object to be identified includes at least one of receiving an interface navigation request, a search request, an outline selection, voice category input, audio input, gesture input, location input, or a category selection.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein analyzing the captured image information includes processing the image information to generate matching data and comparing the matching data against a data store of matching data for a plurality of identified objects.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein the matching data includes at least one of optical character recognition data, text, logo information, barcode data, contextual information, and spatial information from a pair of stereo cameras or a depth sensing camera.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the orientation guide is a graphical representation of an outline of an object from the category of object to be identified.

### Claim 6 (depends on 4)

The computer-implemented method of claim 4, wherein a set of orientation guides is provided for the category, and wherein the user is enabled to select an appropriate orientation guide from the set of orientation guides.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, further comprising:
providing haptic feedback to provide guidance to the user in scrolling to one or more orientation guides of the set.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, further comprising:
enabling the user to one or more of: flip, resize, skew, stretch, distort, or rotate the displayed orientation guide.

### Claim 9

A computer-implemented method, comprising:
receiving, from a computing device, a category of object to be identified;
selecting, using at least one processor, at least one orientation guide for the category, the at least one orientation guide having at least one of an orientation or a scale that, when displayed to a user of a computing device, assists the user of the computing device in capturing an image of the object having at least one of a size and orientation useful for a matching process;
receiving image information captured by a camera of the computing device;
attempting, using the at least one processor, to identify the object of interest by comparing the image information against data for a plurality of object images using the matching process; and
sending information about at least one matching object, for display on the computing device, when the at least one matching object is able to be located for the image information.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, further comprising:
causing the computing device to generate a prompt for additional information from the user when at least one matching object is unable to be located for the image information,

### Claim 11 (depends on 9)

The computer-implemented method of claim 9, further comprising:
locating at least one related item for the at least one matching object; and
providing information about the at least one related item for display to the user.

### Claim 12 (depends on 9)

The computer-implemented method of claim 9, further comprising:
receiving an indication from the computing device that one of the at least one matching objects was selected; and
storing information regarding the selection.

### Claim 13 (depends on 9)

The computer-implemented method of claim 9, wherein the image information is compared against data for a plurality of object images using a planar-surface matching algorithm.

### Claim 14 (depends on 9)

The computer-implemented method of claim 9, further comprising:
receiving selection information indicating the orientation guide selected by the user; and
determining a category in which to compare the image information against data for a plurality of object images based at least in part upon the selection information.

### Claim 15

A method of identifying an object, comprising:
receiving an indication of a type of object to be identified;
capturing, using a camera of a computing device, image information including a current view of the object to be identified, the current view capable of changing depending at least in part upon a position of the computing device;
displaying the captured image information on a display screen of the computing device;
determining, using a processor of the computing device, an orientation guide for the type of object to be identified, the orientation guide having an orientation and size for the category of object that is useful for an image matching process;
displaying the orientation guide on the display screen concurrently with the captured image information, wherein the computing device is configured to provide a user with guidance in adjusting the position of the computing device to the object in order to approximate the orientation and size of the orientation guide in the captured image information; and
causing the captured image information to be analyzed to attempt to identify the object.

### Claim 16 (depends on 15)

The method of claim 15, wherein causing the image information to be analyzed includes transferring at least a portion of the captured image information to a remote matching service for analysis or analyzing the image information on the computing device to generate matching data and sending the matching data to the remote matching service.

### Claim 17 (depends on 15)

The method of claim 15, wherein at least a portion of a matching process is capable of being performed on the computing device.

### Claim 18 (depends on 16)

The method of claim 16, further comprising:
receiving information for at least one matching object; and
displaying the received information on the display screen of the computing device.

### Claim 19 (depends on 16)

The method of claim 16, further comprising:
prompting the user for additional information when at least one matching object is unable to be located or a number of potential matching objects exceeds an allowable threshold.

### Claim 20 (depends on 16)

The method of claim 16, further comprising:
prompting the user to pan the computing device around at least a portion of the object to be identified;
capturing image information during the panning, the image information including multiple views of the object to be identified; and
causing at least one of the multiple views to be analyzed to attempt to identify the object.

### Claim 21 (depends on 16)

The method of claim 16, further comprising:
enabling the user to select a different orientation guide to be displayed on the display screen; and
storing information about the different orientation guide with data stored for the type of object.

### Claim 22 (depends on 17)

The method of claim 17, wherein the captured image information is analyzed using at least one of a stereo matching process or a multi-frame matching process.

### Claim 23

A computing device, comprising:
a processor;
at least one camera;
a display screen; and
a memory device including instructions that, when executed by the processor, cause the computing device to:
determine a category of an object to be identified;
capture, using the at least one camera, video information including a current view of the object to be identified, the current view capable of changing depending at least in part upon a position of the computing device;
displaying the video information on a display screen of the computing device during the capture of video information;
determine a set of orientation guides for the type of object to be identified, each of the set of orientation guides having an orientation and size for the category of object that is useful for an image matching process;
display the orientation guide on the display screen concurrently with the video information, wherein the computing device is capable of providing a user with guidance in adjusting a relative position of the computing device to the object to be identified in order to cause the view of the object in the video information to approximate the orientation and size of the displayed orientation guide;
capture an image of the object to be identified when the orientation and size of the object to be identified in the video information is within an allowable amount of variation from the displayed orientation guide; and
cause the captured image to be analyzed to attempt to identify the object.

### Claim 24 (depends on 23)

The computing device of claim 23, wherein the category of object to be identified is determined using contextual information available to the computing device.

### Claim 25 (depends on 23)

The computing device of claim 23, wherein capturing video information of the object to be identified includes capturing three-dimensional data for the object for use in determining the set of orientation guides.

### Claim 26 (depends on 25)

The computing device of claim 25, wherein the three-dimensional data includes at least one of structured light data, intensity data, shadow data, and reflection data.

### Claim 27 (depends on 23)

The computing device of claim 23, wherein the instructions when executed further cause the computing device to:
display information for an identified object when an image of the identified object matches the captured image with at least a determined level of confidence.

### Claim 28

A non-transitory computer-readable storage medium including instructions that, when executed by a processor of a computing device, cause the computing device to:
determine a type of object to be identified;
capture, using a camera of the computing device, image information including a current view of the object to be identified, the current view capable of changing depending at least in part upon a position of the computing device;
display the captured image information on a display screen of the computing device;
determine, using a processor of the computing device, an object outline guide for the type of object to be identified, the object outline indicating a determined orientation and size for the category of object;
display the object outline on the display screen over the object in the captured image information, the computing device providing guidance in adjusting the current view of the object to be identified in the image information such that an orientation and size of the object in the current view matches the orientation and size of the object outline; and
cause the captured image information to be analyzed to attempt to identify the object.

### Claim 29 (depends on 28)

The non-transitory computer-readable storage medium of claim 28, wherein causing the image information to be analyzed includes transferring at least a portion of the captured image information to a remote matching service for analysis or analyzing the image information on the computing device to generate matching data and sending the matching data to the remote matching service.

### Claim 30 (depends on 28)

The non-transitory computer-readable storage medium of claim 28, wherein the instructions when executed further cause the computing device to:
receive information for at least one matching object; and
display the received information on the display screen of the computing device.
