# US 10,311,593 B2 · Object instance identification using three-dimensional spatial configuration

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,311,593 B2
- Document ID: US10311593B2
- Status: Granted
- Kind: B2
- Application number: 15/352,586
- Filed: 2016-11-16
- Published: 2018-05-17
- Granted: 2019-06-04
- Earliest claimed: 2016-11-16
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Benjamin M Cohen; Asaf Tzadok; Yochay Tzur
- Web page: https://patentgrep.com/patent/US10311593B2

## CPC

- H04N 7/185: {from a mobile camera, e.g. for remote control}
- G02B 27/017: {Head mounted}
- G06F 18/2321: using statistics or function optimisation, e.g. modelling of probability density functions
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/763: {Non-hierarchical techniques, e.g. based on statistics of modelling distributions}
- G06V 20/64: Three-dimensional [3D] objects
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0141: {characterised by the informative content of the display}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system for identifying specific instances of objects in a three-dimensional (3D) scene, comprising: a camera for capturing an image of multiple objects at a site; at least one processor executable to: use a location and orientation of the camera to create a 3D model of the site including multiple instances of objects expected to be in proximity to the camera, and generate multiple candidate clusters each representing a different projection of the 3D model, detect at least two objects in the image, and determine a spatial configuration for each detected object; and match the detected image objects to one of the multiple candidate cluster using the spatial configurations, associate the detected objects with the expected object instances of the matched cluster, and retrieve information of one of the detected objects that is stored with the associated expected object instance; and a head-wearable display configured to display the information.

## Claims

### Claim 1

A system for identifying object instances in a three-dimensional (3D) scene, comprising:
a camera configured to capture an image of multiple objects at a site;
at least one hardware processor; and
a non-transitory memory device having embodied thereon program code executable by said at least one hardware processor to:
receive, from said camera, a captured image that depicts multiple objects that are physically present at the site,
detect at least two objects in the image,
retrieve 3D information of the site, wherein the 3D information comprises location and orientation of objects that have been previously determined to be located at the site,
generate, based on the 3D information of the site, multiple candidate clusters of objects that have been previously determined to be located at the site and are of the same type as the detected objects, wherein each of the candidate clusters represents a different relative spatial configuration between the objects in the respective candidate cluster,
determine a spatial configuration of the objects detected in the image, with respect to each other and to said camera,
match the objects detected in the image to one of the multiple candidate clusters, by:
(a) calculating a 3D transform error between the spatial configuration of (i) the objects detected in the image and (ii) the objects on the respective candidate cluster, and
(b) selecting a candidate cluster with a minimal 3D transform error as a most probable cluster,
associate the objects detected in the image with the objects of the most probable cluster, and
retrieve information of at least one of the objects of the most probable cluster.

### Claim 2 (depends on 1)

The system of claim 1, wherein the program code is further executable by said at least one hardware processor to display the retrieved information.

### Claim 3 (depends on 2)

The system of claim 2, further comprising a head-wearable display selected from the group consisting of: an augmented reality display, and a virtual reality display,
wherein the displaying of the retrieved information is on the head-wearable display.

### Claim 4 (depends on 3)

The system of claim 3, wherein one or more of the at least one hardware processors are disposed with the head-wearable display.

### Claim 5 (depends on 3)

The system of claim 3, wherein the camera is disposed with the head-wearable display.

### Claim 6 (depends on 2)

The system of claim 2, further comprising a user interface configured to receive a selection of one of the objects detected in the image, wherein the displayed retrieved information is associated with one of the objects of the most probable cluster which corresponds to the selected object.

### Claim 7

A method for identifying multiple object instances, comprising:
capturing, by a camera, an image depicting multiple objects that are physically present at a site;
detecting at least two objects in the image;
retrieving three-dimensional (3D) information of the site, wherein the 3D information comprises location and orientation of objects that have been previously determined to be located at the site;
generating, based on the 3D information of the site, multiple candidate clusters of objects that have been previously determined to be located at the site and are of the same type as the detected objects, wherein each of the candidate clusters represents a different relative spatial configuration between the objects in the respective candidate cluster;
determining a spatial configuration of the objects detected in the image, with respect to each other and to said camera;
matching the objects detected in the image to one of the multiple candidate clusters, by:
(a) calculating a 3D transform error between the spatial configuration of (i) the objects detected in the image and (ii) the objects in the respective candidate cluster, and
(b) selecting a candidate cluster with a minimal 3D transform error as a most probable cluster;
associating the objects detected in the image with the objects of the most probable cluster; and
retrieving information of at least one of the objects of the most probable cluster.

### Claim 8 (depends on 7)

The method of claim 7, further comprising displaying the retrieved information.

### Claim 9 (depends on 8)

The method of claim 8, further comprising receiving a user selection of one of the objects detected in the image, wherein the displayed retrieved information is associated with one of the objects of the most probable cluster which corresponds to the selected object.

### Claim 10

A computer program product comprising a non-transitory computer-readable storage medium having program code embodied thereon, the program code executable by at least one hardware processor to:
receive, from a camera, a captured image that depicts multiple objects that are physically present at a site;
detect at least two objects in the image;
retrieve 3D information of the site, wherein the 3D information comprises location and orientation of objects that have been previously determined to be located at the site;
generate, based on the 3D information of the site, multiple candidate clusters of objects that have been previously determined to be located at the site and are of the same type as the detected objects, wherein each of the candidate clusters represents a different relative spatial configuration between the objects in the respective candidate cluster;
determine a spatial configuration of the objects detected in the image, with respect to each other and to the camera;
match the objects detected in the image to one of the multiple candidate clusters, by:
(a) calculating a 3D transform error between the spatial configuration of (i) the objects detected in the image and (ii) the objects in the respective candidate cluster, and
(b) selecting a candidate cluster with a minimal 3D transform error as a most probable cluster;
associate the objects detected in the image with the objects of the most probable cluster; and
retrieve information of at least one of the objects of the most probable cluster.

### Claim 11 (depends on 10)

The computer program product of claim 10, wherein the program code is further executable to display the retrieved information.

### Claim 12 (depends on 11)

The computer program product of claim 11, the program code is further executable to receive a user selection of one of the objects detected in the image, wherein the displayed retrieved information is associated with one of the objects of the most probable cluster which corresponds to the selected object.

### Claim 13 (depends on 8)

The method of claim 8, wherein the displaying of the retrieved information is on a head-wearable display selected from the group consisting of: an augmented reality display, and a virtual reality display.

### Claim 14 (depends on 13)

The method of claim 13, wherein the camera is disposed with the head-wearable display.

### Claim 15 (depends on 11)

The computer program product of claim 11, wherein the displaying of the retrieved information is on a head-wearable display selected from the group consisting of: an augmented reality display, and a virtual reality display.

### Claim 16 (depends on 15)

The computer program product of claim 15, wherein the camera is disposed with the head-wearable display.