# US 2009/0268947 A1 · Real time environment model generation system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2009/0268947 A1
- Document ID: US20090268947A1
- Status: Published application
- Kind: A1
- Application number: 12/416,030
- Filed: 2009-03-31
- Published: 2009-10-29
- Earliest claimed: 2008-03-31
- Assignees: Harman Becker Automotive Systems GimbH
- Inventors: Roland Schaufler
- Web page: https://patentgrep.com/patent/US20090268947A1

## CPC

- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A vehicle environment monitoring system is provided that is based on a three-dimensional vector model. The three-dimensional vector model of the vehicle's environment is generated on the basis of the image data captured by at least one three-dimensional camera. Out of the image data, particular data are extracted for generating the three-dimensional vector model in order to reduce the data volume. For data extraction, a data extraction algorithm is applied that is determined in accordance with at least one parameter that relates to the situation of the vehicle. Therefore, targeted data extraction is performed for generating a three-dimensional model that is particularly adapted for an application that is desired in the current vehicle situation. The applications of the vector model include driver assistance, external monitoring and vehicle control, as well as recording in an event data recorder. In one implementation, a sequence of three-dimensional vector models, representing a three-dimensional space-and-time model, is generated.

## Claims

### Claim 1

A method of monitoring the environment of a camera-equipped vehicle, comprising the steps of:
capturing a 3D-image of the environment of the vehicle, the 3D-image representing a predetermined area of the vehicle environment;
reducing the amount of information acquired in said image capturing step by extracting data from said 3D-image employing a data extraction algorithm; and
generating a 3D-vector model of the vehicle environment from said data extracted from said 3D-image characterized by the step of:
determining the data extraction algorithm to be employed in said reducing step based on at least one parameter characterizing the vehicle situation.

### Claim 2 (depends on 1)

The method of claim 1, where said extraction algorithm determining step determines the data extraction algorithm to be adapted according to the processing time available for generating the 3D-vector model.

### Claim 3 (depends on 1)

The method of claim 1, where said extraction algorithm determining step determines the data extraction algorithm to be adapted according to information reflected in the 3D-vector model.

### Claim 4 (depends on 1)

The method of claim 1, where said data extraction algorithm includes object recognition.

### Claim 5

(canceled)

### Claim 6 (depends on 1)

The method of claim 1, where said data extraction algorithm extracts data only from a part of the 3D-image corresponding to a certain visual field.

### Claim 7 (depends on 1)

The method of claim 1, further comprising the step of further processing said 3D-vector model for assisting a driver in controlling the vehicle based on the processing result.

### Claim 8 (depends on 7)

The method of claim 7, where said extraction algorithm determining step includes the step of determining a maximum available amount of time for performing said steps of generating and further processing said 3D-vector model based on the vehicle situation and further takes into account said maximum available amount of time to determine said data extraction algorithm.

### Claim 9

(canceled)

### Claim 10 (depends on 9)

The method of claim 9, where said extraction algorithm determining step includes the step of determining a maximum amount of data to be extracted in correspondence with said maximum amount of time.

### Claim 11 (depends on 10)

The method of claim 10, where said extraction algorithm determining step further includes the step of determining priorities for different kinds of data that can be extracted from said 3D-image based on the particular relevance of different kinds of data in the vehicle situation characterized by said at least one parameter.

### Claim 12 (depends on 1)

The method of claim 1, where said extraction algorithm determining step further includes the step of analyzing the vehicle situation based on the captured 3D-image, where a value of said at least one parameter is obtained from said analyzing step.

### Claim 13 (depends on 1)

The method of claim 1, further comprising the step of detecting vehicle status information, where the detected vehicle status information includes a value of said at least one parameter.

### Claim 14 (depends on 13)

The method of claim 13, where said vehicle status information includes velocity information of said vehicle, and where the size of the visual field from which said data extraction algorithm extracts data decreases as the velocity of the vehicle increases.

### Claim 15

(canceled)

### Claim 16

(canceled)

### Claim 17 (depends on 1)

The method of claim 1, where said extraction algorithm determining step includes the step of selecting a particular data extraction algorithm out of a plurality of data extraction algorithms based on a parameter describing the vehicle situation.

### Claim 18 (depends on 1)

The method of claim 1, where said 3D-vector model represents objects in the environment of the vehicle by vectors and object classification information, said object classification information associates an object with standard object classes of an object model.

### Claim 19 (depends on 18)

The method of claim 18, where said object model is pre-stored in a database containing standard information of such objects that are most likely to occur and most relevant to the vehicle's environment.

### Claim 20

(canceled)

### Claim 21 (depends on 1)

The method of claim 1, further comprising the step of:
generating a 3D-space-and-time model, where said 3D-space-and-time model includes a sequence of 3D-vector models.

### Claim 22

(canceled)

### Claim 23 (depends on 21)

The method of claim 21, further comprising the step of generating a response to the information of the vehicle environment represented by said 3D-space-and-time model.

### Claim 24 (depends on 23)

The method of claim 23, where said response beg is directed towards avoiding an emergency situation to be expected by a driver.

### Claim 25

(canceled)

### Claim 26

(canceled)

### Claim 27 (depends on 24)

The method of claim 24, where said response includes an automatic interference with the vehicle control.

### Claim 28 (depends on 23)

The method of claim 23, where said response is directed towards assisting a driver in parking the vehicle.

### Claim 29 (depends on 23)

The method of claim 23, where said response includes taking countermeasures to minimize the consequences of a threatening accident.

### Claim 30

(canceled)

### Claim 31 (depends on 1)

The method of claim 1, where said image capturing step captures a plurality of 3D-images from various camera positions, where said plurality of 3D-images re-present an environment area including a predetermined range around the vehice.

### Claim 32

(canceled)

### Claim 33 (depends on 1)

The method of claim 1, where said 3D-vector model generating step further employs data from sensors.

### Claim 34 (depends on 33)

The method of claim 33, further comprising the step of sensor fusion to transform sensor data into a unique format and remove inconsistent and/or redundant data.

### Claim 35

(canceled)

### Claim 36 (depends on 1)

The method of claim 1, further comprising the step of transmitting the 3D vector model to an external location with respect to said vehicle.

### Claim 37

(canceled)

### Claim 38 (depends on 36)

The method of claim 36, where said 3D-vector model is transmitted via a wireless communication.

### Claim 39 (depends on 36)

The method of claim 36, where said 3D-vector model is transmitted to and visualized on an external computer.

### Claim 40 (depends on 36)

The method of claim 36, further comprising the step of evaluating said 3D-vector model at an external computer.

### Claim 41 (depends on 40)

The method of claim 40, further comprising the step of re-transmitting results of said evaluation back to the vehicle.

### Claim 42 (depends on 40)

The method of claim 40, further comprising the step of remotely controlling the vehicle by employing the evaluation results.

### Claim 43

(canceled)

### Claim 44 (depends on 36)

The method of claim 36, where said transmitting step transmits a sequence of 3D-vector models representing a 3D-space-and-time model.

### Claim 45 (depends on 1)

The method of claim 1, further comprising the step of saving the 3D-vector model in an event data recorder.

### Claim 46 (depends on 45)

The method of claim 45, further comprising the steps of:
generating a sequence of 3D-vector models during a predetermined period of time; and
saving said sequence of 3D-vector models into the event data recorder.

### Claim 47 (depends on 46)

The method of claim 46, further comprising the step of associating the 3D-vector models with time stamps before saving into the event data recorder.

### Claim 48 (depends on 47)

The method of claim 47, where the 3D-vector models are saved for a predetermined storing time;
the method further comprising the step of detecting an impact caused by an extraordinary event, where the 3D-vector models remain permanently saved upon said detection of an impact.

### Claim 49 (depends on 48)

The method of claim 48, where said 3D-vector models are stored in a cyclic storage having a predetermined storage capacity;
the method further comprising the step of cyclically overwriting stored 3D-vector models having the earliest time stamp, until said detection of au impact.

### Claim 50

An environment monitoring device for monitoring the environment of a vehicle, comprising:
a 3D-camera for capturing an image of the environment of the vehicle, the image representing a predetermined area of the vehicle environment; and
a first processing unit for processing the information acquired by said 3D-camera, said first processing unit including:
a data extracting unit for extracting data from said image by employing a data extraction algorithm; and
a model generation unit for generating a 3D-vector model of the vehicle environment from said data extracted from said image; characterized in that said first processing unit further includes a determining unit for determining the data extraction algorithm to be employed by said data extracting unit based on at least one parameter characterizing the vehicle situation.

### Claim 51 (depends on 50)

The environment monitoring device of claim 50, where said determining unit determines the data extraction algorithm to be adapted according to the processing time available for generating the 3D-vector model.

### Claim 52 (depends on 50)

The environment monitoring device of claim 50, where said determining unit determines the data extraction algorithm to be adapted according to information reflected in the 3D-vector model.

### Claim 53 (depends on 50)

The environment monitoring device of claim 50, where said data extraction algorithm includes object recognition.

### Claim 54

(canceled)

### Claim 55 (depends on 50)

The environment monitoring device of claim 50, where said data extraction algorithm extract data only from a part of the image corresponding to a certain visual field.

### Claim 56 (depends on 50)

The environment monitoring device of claim 50, further comprising a second processing unit for further processing said 3D-vector model for assisting a driver in controlling the vehicle based on the processing result.

### Claim 57 (depends on 56)

The environment monitoring device of claim 56, where said determining units determines a maximum available amount of time for generating and further processing said 3D-vector model based on the vehicle situation, and further takes into account said maximum available amount of time for determining said data extraction algorithm.

### Claim 58

(canceled)

### Claim 59 (depends on 58)

The environment monitoring device of claim 58, where said determining unit determines a maximum amount of data to be extracted in correspondence with said maximum amount of time.

### Claim 60 (depends on 59)

The environment monitoring device of claim 59, where said determining unit further determines priorities for different kinds of data that can be extracted from said image based on the particular relevance of different kinds of data in the vehicle situation characterized by said at least one parameter.

### Claim 61 (depends on 50)

The environment monitoring device of claim 50, where said determining unit further analyzes the vehicle situation based on the captured image for obtaining a value of said at least one parameter.

### Claim 62 (depends on 50)

The environment monitoring device of claim 50, further comprising a detector for detecting vehicle status information, where the detected vehicle status information includes a value of said at least one parameter.

### Claim 63 (depends on 62)

The environment monitoring device of claim 62, where said vehicle status information includes velocity information of said vehicle, and where the size of the visual field from which said data extraction algorithm extracts data decreases as the velocity of the vehicle is increases.

### Claim 64

(canceled)

### Claim 65

(canceled)

### Claim 66 (depends on 50)

The environment monitoring device of claim 50, where said determining unit selects a particular data extraction algorithm out of a plurality of data extraction algorithms based on a parameter describing the vehicle situation.

### Claim 67 (depends on 50)

The environment monitoring device of claim 50, where said 3D-vector model represents objects in the environment of the vehicle by vectors and object classification information, said object classification information associates an object with standard object classes of an object model.

### Claim 68 (depends on 67)

The environment monitoring device of claim 67, further comprising a database for pre-storing said object model.

### Claim 69 (depends on 67)

The environment monitoring device of claim 67, where said extraction algorithm obtains object classification information from the image.

### Claim 70 (depends on 50)

The environment monitoring device of claim 50, where said model generation unit generates a 3D-space-and-time model that includes a sequence of 3D-vector models.

### Claim 71

(canceled)

### Claim 72 (depends on 70)

The environment monitoring device of claim 70, further comprising a second processing unit for generating a response to the information of the vehicle environment represented by said 3D-space-and-time model.

### Claim 73 (depends on 72)

The environment monitoring device of claim 72, where said response is directed towards avoiding an emergency situation to be expected by a driver.

### Claim 74 (depends on 73)

The environment monitoring device of claim 73, further comprising a device for notifying a driver of an expected emergency situation based on the response generated by said second processing unit.

### Claim 75 (depends on 73)

The environment monitoring device of claim 73, further comprising a device for assisting a driver to avoid an emergency situation based on the response generated by said second processing unit.

### Claim 76 (depends on 75)

The environment monitoring device of claim 75, where said device is a controller for automatically interfering with the vehicle control based on the response generated by said second processing unit.

### Claim 77 (depends on 72)

The environment monitoring device of claim 72, where said response is directed towards assisting a driver in parking the vehicle.

### Claim 78 (depends on 72)

The environment monitoring device of claim 72, further comprising a controller for taking countermeasures to minimize the consequences of a threatening accident based on the response generated by said second processing unit.

### Claim 79

(canceled)

### Claim 80 (depends on 50)

The environment monitoring device of claim 50, comprising a plurality of 3D cameras for capturing images from various camera positions, where the images captured by said plurality of 3D-cameras representing an environment area include a predetermined range around the vehicle.

### Claim 81

(canceled)

### Claim 82 (depends on 50)

The environment monitoring device of claim 50, further comprising sensors, where said model generation unit further employs data from said sensors for generating said 3D-vector model.

### Claim 83 (depends on 82)

The environment monitoring device of claim 82, further comprising a sensor fusion unit for transforming sensor data into a unique format and removing inconsistent and/or redundant data.

### Claim 84

(canceled)

### Claim 85 (depends on 50)

The environment monitoring device of claim 50, further comprising a transmitter for transmitting the 3D-vector model to an external location with respect to said vehicle.

### Claim 86

(canceled)

### Claim 87 (depends on 85)

The environment monitoring device of claim 85, where said transmitter transmits said 3D-vector model via wireless communication.

### Claim 88 (depends on 85)

The environment monitoring device of claim 85, where said transmitter transmits said 3D-vector model for visualizing on an external computer.

### Claim 89 (depends on 85)

The environment monitoring device of claim 85, where said transmitter transmits said 3D-vector model for evaluating at an external computer.

### Claim 90 (depends on 89)

The environment monitoring device of claim 89, further comprising receiving unit for receiving results of said evaluation transmitted back to the vehicle.

### Claim 91 (depends on 89)

The environment monitoring device of claim 89, further comprising a controller for controlling the vehicle by employing the evaluation results received from the external computer.

### Claim 92

(canceled)

### Claim 93 (depends on 85)

The environment monitoring device of claim 85, where said transmitter transmits a sequence of 3D-vector models representing a 3D-space-and-time model.

### Claim 94 (depends on 50)

The environment monitoring device of claim 50, further comprising an event data recorder for saving the 3D-vector model.

### Claim 95 (depends on 94)

The environment monitoring device of claim 94, where said model generation unit generates a sequence of 3D-vector models during a predetermined period of time, and where said sequence of 3D-vector models is saved into the event data recorder.

### Claim 96 (depends on 95)

The environment monitoring device of claim 95, where the 3D-vector models are associated with time stamps before saving.

### Claim 97 (depends on 96)

The environment monitoring device of claim 96, further comprising an impact detector for detecting an impact caused by an extraordinary event, where the 3D-vector models are saved in the event data recorder for a predetermined storing time and the 3D-vector models remain permanently saved when an impact is detected by said impact detector.

### Claim 98

(canceled)
