# US 11,715,236 B2 · Method and system for re-projecting and combining sensor data for visualization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,715,236 B2
- Document ID: US11715236B2
- Status: Granted
- Kind: B2
- Application number: 17/680,802
- Filed: 2022-02-25
- Published: 2022-06-09
- Granted: 2023-08-01
- Earliest claimed: 2019-02-15
- Assignees: Interaptix Inc.
- Applicants: Interaptix Inc.
- Inventors: Dae Hyun Lee; Tyler James Doyle
- Web page: https://patentgrep.com/patent/US11715236B2

## CPC

- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- H04N 5/265: Mixing
- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- G06T 15/205: {Image-based rendering}
- H04N 17/002: {for television cameras}
- G06T 2210/56: Particle system, point based geometry or rendering
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

There is provided a system and method of re-projecting and combining sensor data of a scene from a plurality of sensors for visualization. The method including: receiving the sensor data from the plurality of sensors; re-projecting the sensor data from each of the sensors into a new viewpoint; localizing each of the re-projected sensor data; combining the localized re-projected sensor data into a combined image; and outputting the combined image. In a particular case, the receiving and re-projecting can be performed locally at each of the sensors.

## Claims

### Claim 1

A non-transitory machine-readable medium storing instructions, which when executed cause a processor to:
receive sensor data representing a scene from a plurality of sensors, comprising red-green-blue-depth (RGB-D) channel values for each of a plurality of pixels;
re-project the sensor data from each of the sensors into a new viewpoint by:
calibrating each of the sensors to determine one or more calibration values for the respective sensor;
generating a point cloud for the respective sensor by applying the calibration values to the depth D channel values for each pixel; and
applying a matrix representative of the new viewpoint to the point cloud for each sensor;
localize each of the re-projected sensor data;
combine the localized re-projected sensor data into a combined image; and
output the combined image.

### Claim 2 (depends on 1)

The non-transitory machine-readable medium of claim 1, wherein calibrating each of the sensors comprises determining intrinsic parameters for each of the sensors, the intrinsic parameters comprising distortions and camera matrix.

### Claim 3 (depends on 1)

The non-transitory machine-readable medium of claim 1, wherein the localization comprises performing simultaneous localization and mapping (SLAM) to position and orient the re-projected sensor data relative to each other.

### Claim 4 (depends on 3)

The non-transitory machine-readable medium of claim 3, wherein simultaneous localization and mapping comprises performing point-to-plane iterative closest point.

### Claim 5 (depends on 1)

The non-transitory machine-readable medium of claim 1, wherein the instructions further configure the processor to add supporting elements to the combined image.

### Claim 6 (depends on 5)

The non-transitory machine-readable medium of claim 5, wherein the supporting elements comprise graphical elements associated with one or more objects located in the scene.

### Claim 7 (depends on 1)

The non-transitory machine-readable medium of claim 1, wherein the combined image comprises a top-down view.