# US 2019/0318178 A1 · Method, system and device of obtaining 3d-information of objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0318178 A1
- Document ID: US20190318178A1
- Status: Published application
- Kind: A1
- Application number: 16/381,732
- Filed: 2019-04-11
- Published: 2019-10-17
- Earliest claimed: 2018-04-12
- Assignees: Motherson Innovations Company Limited
- Applicants: Motherson Innovations Company Limited
- Inventors: Maximilian KAUFFMANN; Kai Uwe HAGENBURG
- Web page: https://patentgrep.com/patent/US20190318178A1

## CPC

- B60R 1/27: providing all-round vision, e.g. using omnidirectional cameras
- G06T 5/80: Geometric correction
- G06T 7/55: from multiple images
- G06V 20/20: in augmented reality scenes
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/586: {of parking space}
- G06V 20/64: Three-dimensional [3D] objects
- B60R 2300/105: using multiple cameras
- B60R 2300/303: using joined images, e.g. multiple camera images
- B60R 2300/307: virtually distinguishing relevant parts of a scene from the background of the scene
- B60R 2300/607: from a bird's eye viewpoint
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method for obtaining 3D information of objects shown in at least two images obtained by at least two on-vehicle sensors, a device for carrying out the respective steps of the method, a system including such a device, and a vehicle including such a device or such a system are described.

## Claims

### Claim 1

A method for obtaining 3D information of objects at least partly shown in at least two images obtained by at least two on-vehicle sensor devices, comprising:
receiving at least one first image obtained by a first sensor device;
receiving at least one second image obtained by a second sensor device;
generating a shared 3D space by combining information from at least the first image and the second image;
identifying at least one object in the shared 3D space, whereby the object is shown at least partly in the first and at least partly in the second image;
determining at least one 3D information of the object; and
performing, based on the determined 3D information, at least one action.

### Claim 2 (depends on 1)

The method of claim 1, further comprising at least one of:
receiving at least one third image or a plurality of third images obtained by at least one third sensor device or a plurality of third sensor devices;
applying distortion correction or lens distortion correction, at least partially on the first image, based on at least one predefined parameter including a lens parameter of the first sensor device;
applying distortion correction or lens distortion correction, at least partially on the second image, based on at least one predefined parameter including a lens parameter of the second sensor device;
applying distortion correction or lens distortion correction, at least partially on the third image, based on at least one predefined parameter including a lens parameter of the third sensor device; and
receiving at least partly one surround view image, generating at least partly at least one surround view image by a surround view image processor, generating at least partly at least one surround view image for at least one driver assistance system, or displaying at least partly at least one surround view image by at least one driver assistance system, wherein the surround view image is based at least partially on at least one or more of the first image, the second image and the third image.

### Claim 3 (depends on 1)

The method of claim 1, wherein at least one of
generating the shared 3D space further comprises performing 3D data mapping, at least partly of the first, second and/or third image, and/or determining at least a part of shared field of views of the respective images;
generating the shared 3D space further comprises matching objects, at least the object, at least partly shown in the first image, in the second image and/or the third image, using (a) image feature registration having large disparities, (b) tracking over time, (c) determined structures of motion of objects and/or (d) at least one random sample consensus (RANSAC)-based method;
the shared 3D space at least partly is, comprises and/or represents a model of at least one part of the vehicle's environment, particularly at least one part of the surrounding area of the vehicle;
the 3D information is determined at least partly based on the shared 3D space;
the 3D information comprises at least partly at least one depth information and/or distance;
the distance is a distance of the object measured with respect to at least one reference point, particularly at least one reference point at least partly located on or at the vehicle; and
the surround view image comprises at least partly at least one first image part, wherein the first image part is at least partly obtained by clipping the first image at least partly along at least one first border, at least one second image part, wherein the second image part is at least partly obtained by clipping the second image at least partly along at least one second border, and/or at least one third image part, wherein the third image part is at least partly obtained by clipping the third image along at least one third border.

### Claim 4 (depends on 1)

The method of claim 1, wherein performing the action comprises at least one of:
determining if the object is located at least partially inside at least one blind spot area of the surround view image, wherein the object is located inside the blind spot area if the object is outside and/or partially outside of the first image part and/or outside and/or partially outside of the second image part, and in response to determining that the object is located at least partially inside the blind spot area, adjusting at least one part of the first border and/or the second border in order to include the object at least partly in the first image part and/or in the second image part;
triggering a signaling device, for informing the driver of the vehicle about the object, if the 3D information of the object exceeds and/or falls below a first threshold;
enhancing at least one of the following application with the 3D information of the object: surround view application, parking application, especially with distances between 1.5 meters and 2 meters, far view application, especially with more than 20 meters;
rearranging at least a part of the surround view image, especially by means of changing, particularly increasing and/or decreasing, the zoom level of the surround view image, highlighting the object at least partly on the surround view image and/or isolating the object at least partly on the surround view image, dependent on the 3D information of the object, especially if the 3D information of the object exceeds and/or falls below a second threshold; and
forwarding the 3D information of the object at least partly to at least one advanced driver assistance system.

### Claim 5 (depends on 1)

A device for obtaining 3D information of objects shown in at least two images obtained by at least two on-vehicle sensors, comprising at least one processor configured to carry out the steps of the method of claim 1.

### Claim 6 (depends on 5)

A system for obtaining 3D information of objects shown in at least two images obtained by at least two on-vehicle sensors, comprising:
at least one first sensor device;
at least one second sensor device; and
at least one device according to claim 5.

### Claim 7 (depends on 6)

The system of claim 6, further comprising at least one of:
at least one third sensor device or a plurality of third sensor devices;
at least one driver assistance system and/or at least one signaling device;
at least one bird view system processor and/or at least one surround view image processor; and
at least one device adapted to execute at least one of the following application: surround view application, parking application with distances between 1.5 meters and 2 meters, far view application with more than 20 meters.

### Claim 8 (depends on 6)

The system of claim 6, wherein
the first sensor device is adapted to obtain at least partly at least a first image;
the first sensor device at least partly comprises and/or represents at least one first camera, at least one first camera monitor system (cms) camera, at least one first mirror replacement camera, at least one first back-up camera, at least one first camera of the vehicle's advanced driver assistant system, at least one first bird view image processor providing at least one bird view image, at least one first surround view image processor providing at least one surround view image, at least one first radar device, at least one first lidar device and/or at least one first ultrasonic device;
the first sensor device comprises at least partly at least one fisheye lens and/or is attached or attachable to the vehicle at a first position;
the second sensor device is adapted to obtain at least partly at least a second image;
the second sensor device at least partly comprises and/or represents at least one second camera, at least one second camera monitor system (cms) camera, at least one second mirror replacement camera, at least one second back-up camera, at least one second camera of the vehicle's advanced driver assistant system, at least one second bird view image processor providing at least one bird view image, at least one second surround view image processor providing at least one surround view image, at least one second radar device, at least one second lidar device and/or at least one second ultrasonic device;
the third sensor device at least partly comprises at least one fisheye lens and/or is attached or attachable to the vehicle at a third position;
the third sensor device is adapted to obtain at least partly at least a third image;
the third sensor device at least partly comprises and/or represents at least one third camera, at least one third camera monitor system (cms) camera, at least one third mirror replacement camera, at least one third back-up camera, at least one third camera of the vehicle's advanced driver assistant system, at least one third bird view image processor providing at least one bird view image, at least one third surround view image processor providing at least one surround view image, at least one third radar device, at least one third lidar device and/or at least one third ultrasonic device;
the second sensor device comprises at least partly at least one fisheye lens and/or is attached or attachable to the vehicle at a second position; and
the first sensor device and the second sensor device have essentially the same global field of view, the first sensor device and the third sensor device have essentially the same global field of view, the second sensor device and the third sensor device have essentially the same global field of view, the first image obtained by the first sensor device has a large area of intersection with the second image obtained by the second sensor device, the first image obtained by the first sensor device has a large area of intersection with the third image obtained by the third sensor device and/or the second image obtained by the second sensor device has a large area of intersection with the third image obtained by the third sensor device.

### Claim 9 (depends on 5)

A motor vehicle, comprising a device according to claim 5.

### Claim 10 (depends on 9)

The motor vehicle of claim 9, wherein the first sensor device is at least partly attached to the vehicle at a first position, the second sensor device is at least partly attached to the vehicle at a second position and/or the third sensor device is at least partly attached to the vehicle at a third position.

### Claim 11 (depends on 6)

A motor vehicle, comprising a system according to claim 6.

### Claim 12 (depends on 11)

The motor vehicle of claim 11, wherein the first sensor device is at least partly attached to the vehicle at a first position, the second sensor device is at least partly attached to the vehicle at a second position and/or the third sensor device is at least partly attached to the vehicle at a third position.