# US 9,521,397 B2 · System and method for selecting a two-dimensional region of interest using a range sensor

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,521,397 B2
- Document ID: US9521397B2
- Status: Granted
- Kind: B2
- Application number: 14/204,119
- Filed: 2014-03-11
- Published: 2015-09-17
- Granted: 2016-12-13
- Earliest claimed: 2014-03-11
- Assignees: GM Global Technology Operations LLC
- Applicants: GM Global Technology Operations LLC
- Inventors: Neil David McKay; James W. Wells; Roland J. Menassa
- Web page: https://patentgrep.com/patent/US9521397B2

## CPC

- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 7/4808: {Evaluating distance, position or velocity data}
- G01S 17/894: Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A system includes a first camera defining a first camera coordinate system (C1) and configured to acquire a first image of a scene. A range sensor is spaced a first distance from the first camera and defines a range sensor coordinate system (R). A controller is operatively connected to the first camera and range sensor. The controller has a processor and a tangible, non-transitory memory device on which is recorded instructions for executing a method for obtaining a two-dimensional region of interest (u1*, v1*) in the first image, which is a two-dimensional intensity image. The first image is represented by a plurality of first points (u1, v1) in a first image plane. The controller is configured to acquire a range image of the scene with the range sensor. The range image is represented by a plurality of second points (u2, v2, d) in a second image plane.

## Claims

### Claim 1

A system comprising:
a first camera defining a first camera coordinate system (C1) and configured to acquire a first image of a scene;
a range sensor spaced a first distance from the first camera and defining a range sensor coordinate system (R);
a controller operatively connected to the first camera and range sensor, the controller having a processor and a tangible, non-transitory memory device on which is recorded instructions for executing a method for obtaining a two-dimensional region of interest (u1*, v1*) in the first image;
wherein execution of the instructions by the processor causes the controller to:
acquire the first image of the scene with the first camera, the first image being represented by a plurality of first points (u1, v1) in a first image plane;
acquire a range image of the scene with the range sensor, the range image being represented by a plurality of second points (u2, v2, d) in a second image plane;
convert the range image to a three-dimensional sample of points (x2, y2, z2) in the range sensor coordinate system (R);
select a three-dimensional region of interest (x2*, y2*, z2*) in the range sensor coordinate system (R) from the three-dimensional sample of points (x2, y2, z2);
wherein the selected three-dimensional region of interest (x2*, y2*, z2*) in the range sensor coordinate system (R) includes only objects in the scene on a surface of or within a three-dimensional volume, the three-dimensional volume defining selected points;
wherein a spatial location of each of the selected points is a function of time;
wherein the selected three-dimensional region of interest (x2*, y2*, z2*) in the range sensor coordinate system (R) includes only objects in the scene that are less than a minimum distance from the range sensor; and
wherein said converting the range image to a three-dimensional sample of points (x2, y2, z2) in the range sensor coordinate system (R) includes:
obtaining a conversion matrix (P2) for converting the range sensor coordinate system (R) to the second image plane; and
using an inverse of the conversion matrix for converting the range image to the three-dimensional sample of points (x2, y2, z2).

### Claim 2 (depends on 1)

The system of claim 1, wherein:
each of the plurality of second points (u2, v2, d) in the range image includes a range distance (d) corresponding to a respective distance from the range sensor to the objects in the scene;
the selected three-dimensional region of interest (x2*, y2*, z2*) defines selected points, a spatial location of each of the selected points being a function of the range distance (d).

### Claim 3 (depends on 1)

The system of claim 1, wherein the first camera, the controller and the range sensor are part of a single device.

### Claim 4 (depends on 1)

The system of claim 1, wherein the controller is further configured to:
transform the three-dimensional region of interest from the range sensor coordinate system (R) to the first camera coordinate system (C1) [(x2*, y2*, z2*) to (x1*, y1*, z1*)]; and
map the three-dimensional region of interest (x1*, y1*, z1*) in the first camera coordinate system (C1) onto the first image plane to obtain the two-dimensional region of interest (u1*, v1*).

### Claim 5 (depends on 4)

The system of claim 4, wherein said transforming the three-dimensional region of interest from the range sensor coordinate system (R) to the first camera coordinate system (C1) [(x2*, y2*, z2*) to (x1*, y1*, z1*)] includes:
obtaining a first transformation matrix (T21) for converting the range sensor coordinate system (R) to the first camera coordinate system (C1).

### Claim 6 (depends on 4)

The system of claim 4, wherein said mapping the three-dimensional region of interest (x1*, y1*, z1*) in the first camera coordinate system (C1) onto the first image plane includes:
obtaining a projection matrix (P1) for projecting the first camera coordinate system (C1) to the first image plane.

### Claim 7 (depends on 4)

The system of claim 4, further comprising:
a second camera spaced a second distance from the range sensor and defining a second camera coordinate system (C2);
wherein the controller is configured to:
after said acquiring the range image, acquire a third image of the scene, the third image being defined by a plurality of third points (u3, v3) in a third image plane;
determine a second transformation matrix (T31) for converting the third image plane to the first image plane;
transform the plurality of third points (u3, v3) from the third image plane to the first image plane via the second transformation matrix (T31); and
prior to the conversion of the range image to the three-dimensional sample of points (x2, y2, z2), combine the third image with the first image.

### Claim 8

A method of obtaining a two-dimensional region of interest in a two-dimensional image, the method comprising:
acquiring a first image of a scene with a first camera, the first image being represented by a plurality of first points (u1, v1) in a first image plane, the first camera defining a first camera coordinate system (C1);
acquire a range image of the scene with a range sensor, the range image being represented by a plurality of second points (u2, v2, d) in a second image plane;
wherein the range sensor is spaced a first distance from the first camera and defines a range sensor coordinate system (R);
converting the range image to a three-dimensional sample of points (x2, y2, z2) in the range sensor coordinate system (R);
selecting a three-dimensional region of interest (x2*, y2*, z2*) in the range sensor coordinate system (R) from the three-dimensional sample of points (x2, y2, z2);
transforming the three-dimensional region of interest from the range sensor coordinate system (R) to the first camera coordinate system (C1) [(x2*, y2*, z2*) to (x1*, y1*, z1*)];
mapping the three-dimensional region of interest (x1*, y1*, z1*) in the first camera coordinate system (C1) onto the first image plane to obtain the two-dimensional region of interest (u1*, v1*);
wherein the selected three-dimensional region of interest (x2*, y2*, z2*) in the range sensor coordinate system (R) includes only objects in the scene on a surface of or within a three-dimensional volume, the three-dimensional volume defining selected points;
a spatial location of each of the selected points is a function of time;
wherein the selected three-dimensional region of interest (x2*, y2*, z2*) in the range sensor coordinate system (R) includes only objects in the scene that are less than a minimum distance from the range sensor; and
wherein said converting the range image to a three-dimensional sample of points (x2, y2, z2) in the range sensor coordinate system (R) includes:
obtaining a conversion matrix (P2) for converting the range sensor coordinate system (R) to the second image plane; and
using an inverse of the conversion matrix for converting the range image to the three-dimensional sample of points (x2, y2, z2).

### Claim 9 (depends on 8)

The method of claim 8, wherein said transforming the three-dimensional region of interest from the range sensor coordinate system (R) to the first camera coordinate system (C1) [(x2*, y2*, z2*) to (x1*, y1*, z1*)] includes:
obtaining a first transformation matrix (T21) for converting the range sensor coordinate system (R) to the first camera coordinate system (C1).

### Claim 10 (depends on 8)

The method of claim 8, wherein said mapping the three-dimensional region of interest (x1*, y1*, z1*) in the first camera coordinate system (C1) onto the first image plane includes:
obtaining a projection matrix (P1) for projecting the first camera coordinate system (C1) to the first image plane.

### Claim 11 (depends on 8)

The method of claim 8, further comprising:
after said acquiring the second image, acquiring a third image of the scene with a second camera, the third image being defined by a plurality of third points (u3, v3) in a third image plane;
wherein the second camera is spaced a second distance from the range sensor and defines a second camera coordinate system (C2);
determining a second transformation matrix (T31) for converting the third image plane to the first image plane;
transforming the plurality of third points (u3, v3) from the third image plane to the first image plane via the second transformation matrix (T31); and
prior to the conversion of the range image to the three-dimensional sample of points (x2, y2, z2), combining the third image with the first image.