# US 2020/0394848 A1 · Scalable three-dimensional object recognition in a cross reality system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0394848 A1
- Document ID: US20200394848A1
- Status: Published application
- Kind: A1
- Application number: 16/899,878
- Filed: 2020-06-12
- Published: 2020-12-17
- Earliest claimed: 2019-06-14
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Siddharth Choudhary; Divya Ramnath; Shiyu Dong; Siddarth Mahendran; Arumugam Kalai Kannan; Prateek Singhal; Khushi Gupta; Nitesh Sekhar; Manushree Gangwar
- Web page: https://patentgrep.com/patent/US20200394848A1

## CPC

- G06T 7/11: Region-based segmentation
- G06T 7/50: Depth or shape recovery
- G06T 7/73: using feature-based methods
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 10/764: using classification, e.g. of video objects
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for scalable three-dimensional (3-D) object recognition in a cross reality system. One of the methods includes maintaining object data specifying objects that have been recognized in a scene. A stream of input images of the scene is received, including a stream of color images and a stream of depth images. A color image is provided as input to an object recognition system. A recognition output that identifies a respective object mask for each object in the color image is received. A synchronization system determines a corresponding depth image for the color image. A 3-D bounding box generation system determines a respective 3-D bounding box for each object that has been recognized in the color image. Data specifying one or more 3-D bounding boxes is received as output from the 3-D bounding box generation system.

## Claims

### Claim 1

A computer-implemented method, the method comprising:
maintaining object data specifying objects that have been recognized in a scene in an environment;
receiving a stream of input images of the scene, wherein the stream of input images comprises a stream of color images and a stream of depth images;
for each of a plurality of color images in the stream of color images:
providing the color image as input to an object recognition system;
receiving, as output from the object recognition system, a recognition output that identifies a respective object mask in the color image for each of one or more objects that have been recognized in the color image;
providing the color image and a plurality of depth images in the stream of depth images as input to a synchronization system that determines a corresponding depth image for the color image based on a timestamp of the corresponding depth image and a timestamp of the color image;
providing the object data, the recognition output identifying the object masks, and the corresponding depth image as input to a three-dimensional (3-D) bounding box generation system that determines, from the object data, the object masks, and the corresponding depth image, a respective 3-D bounding box for each of one or more of the objects that have been recognized in the color image; and
receiving, as output from the 3-D bounding box generation system, data specifying one or more 3-D bounding boxes for one or more of the objects recognized in the color image; and
providing, as output, data specifying the one or more 3-D bounding boxes.

### Claim 2 (depends on 1)

The method of claim 1, wherein the 3-D bounding box generation system comprises:
a multi-view fusion system that generates an initial set of 3-D object masks.

### Claim 3 (depends on 2)

The method of claim 2, wherein the object recognition system, the synchronization system, the multi-view fusion system operate in a stateless manner and independently from one another.

### Claim 4 (depends on 2)

The method of claim 2, wherein the multi-view fusion system comprises:
an association system that identifies, from the maintained object data, matched object data specifying a corresponding object with the respective object mask of each recognized object in the color image; and
a fusion system that generates, for each recognized object in the color image, an initial 3-D object mask by combining the object mask in the color image with the matched object data.

### Claim 5 (depends on 2)

The method of claim 2, wherein the 3-D bounding box generation system further comprises an object refinement system that refines the initial set of 3-D object masks to generate an initial set of 3-D bounding boxes.

### Claim 6 (depends on 2)

The method of claim 2, wherein the 3-D bounding box generation system further comprises a bounding box refinement system that refines the initial set of 3-D bounding boxes to generate the one or more 3-D bounding boxes.

### Claim 7 (depends on 1)

The method of claim 1, wherein the object recognition system comprises a trained deep neural network (DNN) model that takes the color image as input and generates a respective two-dimensional (2-D) object mask for each of the one or more objects that have been recognized in the color image.

### Claim 8 (depends on 1)

The method of claim 1, wherein determining, by the synchronization system, a corresponding depth image for the color image based on timestamps of the corresponding depth images and timestamp of the color image comprises:
identifies a candidate depth image which has a closest timestamp to the timestamp of the color image;
determining that a time difference between the candidate depth image and the color image is less than a threshold; and
in response, determining the candidate depth image as the corresponding depth image for the color image.

### Claim 9 (depends on 1)

The method of claim 1, wherein the 3-D bounding box generation system determines, from the object masks and the corresponding depth image, a respective 3-D object mask for each of the one or more of the objects that have been recognized in the color image, and wherein the method further comprises:
receiving, as output from the 3-D bounding box generation system, data specifying one or more 3-D object masks for the one or more of the objects recognized in the color image; and
providing, as output, data specifying the one or more 3-D object masks.

### Claim 10

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
maintaining object data specifying objects that have been recognized in a scene in an environment;
receiving a stream of input images of the scene, wherein the stream of input images comprises a stream of color images and a stream of depth images;
for each of a plurality of color images in the stream of color images:
providing the color image as input to an object recognition system;
receiving, as output from the object recognition system, a recognition output that identifies a respective object mask in the color image for each of one or more objects that have been recognized in the color image;
providing the color image and a plurality of depth images in the stream of depth images as input to a synchronization system that determines a corresponding depth image for the color image based on a timestamp of the corresponding depth image and a timestamp of the color image;
providing the object data, the recognition output identifying the object masks, and the corresponding depth image as input to a three-dimensional (3-D) bounding box generation system that determines, from the object data, the object masks, and the corresponding depth image, a respective 3-D bounding box for each of one or more of the objects that have been recognized in the color image; and
receiving, as output from the 3-D bounding box generation system, data specifying one or more 3-D bounding boxes for one or more of the objects recognized in the color image; and
providing, as output, data specifying the one or more 3-D bounding boxes.

### Claim 11 (depends on 10)

The system of claim 10, wherein the 3-D bounding box generation system comprises a multi-view fusion system that generates an initial set of 3-D object masks, wherein the object recognition system, the synchronization system, the multi-view fusion system operate in a stateless manner and independently from one another.

### Claim 12

A computer program product encoded on one or more non-transitory computer readable media, the computer program product comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
maintaining object data specifying objects that have been recognized in a scene in an environment;
receiving a stream of input images of the scene, wherein the stream of input images comprises a stream of color images and a stream of depth images;
for each of a plurality of color images in the stream of color images:
providing the color image as input to an object recognition system;
receiving, as output from the object recognition system, a recognition output that identifies a respective object mask in the color image for each of one or more objects that have been recognized in the color image;
providing the color image and a plurality of depth images in the stream of depth images as input to a synchronization system that determines a corresponding depth image for the color image based on a timestamp of the corresponding depth image and a timestamp of the color image;
providing the object data, the recognition output identifying the object masks, and the corresponding depth image as input to a three-dimensional (3-D) bounding box generation system that determines, from the object data, the object masks, and the corresponding depth image, a respective 3-D bounding box for each of one or more of the objects that have been recognized in the color image; and
receiving, as output from the 3-D bounding box generation system, data specifying one or more 3-D bounding boxes for one or more of the objects recognized in the color image; and
providing, as output, data specifying the one or more 3-D bounding boxes.

### Claim 13

A computer-implemented method, the method comprising:
maintaining object data specifying objects that have been recognized in a scene in an environment;
receiving a stream of input images of the scene;
for each of a plurality of input images in the stream of input images:
providing the input image as input to an object recognition system;
receiving, as output from the object recognition system, a recognition output that identifies a respective bounding box in the input image for each of one or more objects that have been recognized in the input image;
providing data identifying the bounding boxes as input to a three-dimensional (3-D) bounding box generation system that determines, from the object data and the bounding boxes, a respective 3-D bounding box for each of one or more of the objects that have been recognized in the input image; and
receiving, as output from the 3-D bounding box generation system, data specifying one or more 3-D bounding boxes for one or more of the objects recognized in the input image; and
providing, as output, data specifying the one or more 3-D bounding boxes.

### Claim 14 (depends on 13)

The method of claim 13, wherein the 3-D bounding box generation system comprises:
a multi-view fusion system that generates an initial set of 3-D bounding boxes; and
a bounding box refinement system that refines the initial set of 3-D bounding boxes to generate the one or more 3-D bounding boxes.

### Claim 15 (depends on 14)

The method of claim 14, wherein the object recognition system, the multi-view fusion system, and the bounding box refinement system operate in a stateless manner and independently from one another.

### Claim 16 (depends on 14)

The method of claim 14, wherein the maintained object data comprises an ellipsoid that is generated from a plurality of two-dimensional (2-D) bounding boxes of each object that have been recognized in the scene, wherein the multi-view fusion system generates the initial set of 3-D bounding boxes by performing at least the following steps:
for each 2-D bounding box identified in the input image,
determining whether the 2-D bounding box identified in the input image is associated with one or more 2-D bounding boxes of an object that has been recognized in the maintained object data;
in response to determining that the 2-D bounding box identified in the input image is associated with one or more 2-D bounding boxes of an object that has been recognized, updating the maintained object data by calculating an updated ellipsoid of the object using the 2-D bounding box identified in the input image;
in response to determining that the 2-D bounding box identified in the input image is not associated with any objects that have been recognized, creating a new object by generating an ellipsoid from at least the 2-D bounding box identified in the input image; and
generating the initial set of 3-D bounding boxes using the ellipsoids of the objects that have been recognized in the input image.

### Claim 17 (depends on 13)

The method of claim 13, wherein the object recognition system comprises a trained deep neural network (DNN) model that takes the input image and generates a respective two-dimensional (2-D) object bounding box for each of the one or more objects that have been recognized in the input image.

### Claim 18 (depends on 13)

The method of claim 13, wherein the stream of input images of the scene are captured from two or more user devices.

### Claim 19

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
maintaining object data specifying objects that have been recognized in a scene in an environment;
receiving a stream of input images of the scene;
for each of a plurality of input images in the stream of input images:
providing the input image as input to an object recognition system;
receiving, as output from the object recognition system, a recognition output that identifies a respective bounding box in the input image for each of one or more objects that have been recognized in the input image;
providing data identifying the bounding boxes as input to a three-dimensional (3-D) bounding box generation system that determines, from the object data and the bounding boxes, a respective 3-D bounding box for each of one or more of the objects that have been recognized in the input image; and
receiving, as output from the 3-D bounding box generation system, data specifying one or more 3-D bounding boxes for one or more of the objects recognized in the input image; and
providing, as output, data specifying the one or more 3-D bounding boxes.

### Claim 20

A computer program product encoded on one or more non-transitory computer readable media, the computer program product comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
maintaining object data specifying objects that have been recognized in a scene in an environment;
receiving a stream of input images of the scene;
for each of a plurality of input images in the stream of input images:
providing the input image as input to an object recognition system;
receiving, as output from the object recognition system, a recognition output that identifies a respective bounding box in the input image for each of one or more objects that have been recognized in the input image;
providing data identifying the bounding boxes as input to a three-dimensional (3-D) bounding box generation system that determines, from the object data and the bounding boxes, a respective 3-D bounding box for each of one or more of the objects that have been recognized in the input image; and
receiving, as output from the 3-D bounding box generation system, data specifying one or more 3-D bounding boxes for one or more of the objects recognized in the input image; and
providing, as output, data specifying the one or more 3-D bounding boxes.
