# US 2019/0318041 A1 · Method and apparatus for generating situation awareness graphs using cameras from different vehicles

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0318041 A1
- Document ID: US20190318041A1
- Status: Published application
- Kind: A1
- Application number: 15/950,917
- Filed: 2018-04-11
- Published: 2019-10-17
- Earliest claimed: 2018-04-11
- Assignees: GM Global Technology Operations LLC
- Applicants: GM Global Technology Operations LLC
- Inventors: Fan Bai; Mohannad Murad; Joseph G. Machak; Wende Zhang
- Web page: https://patentgrep.com/patent/US20190318041A1

## CPC

- G06F 16/9024: {Graphs; Linked lists (G06F16/9027 takes precedence)}
- B60W 40/04: Traffic conditions
- G06F 16/53: Querying
- G06F 18/24155: {Bayesian classification}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- H04W 4/44: for communication between vehicles and infrastructures, e.g. vehicle-to-cloud [V2C] or vehicle-to-home [V2H]
- H04W 4/46: for vehicle-to-vehicle communication [V2V]
- B60W 30/0956: {the prediction being responsive to traffic or environmental parameters}
- B60W 60/00276: {for two or more other traffic participants}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

In various embodiments, methods, systems, and vehicles are provided that include obtaining first camera images from a first camera onboard a first vehicle; generating, via one or more computer processors, a first situation awareness graph with respect to objects near the first vehicle, using the first camera images; obtaining second camera images from a second camera of a second device that is in proximity to the first vehicle; generating, via one or more computer processors, a second situation awareness graph with the respect to the objects, using the second camera images; and generating, via one or more computer processor, a global situation awareness graph with respect to the objects, by merging the first situation awareness graph with the second situation awareness graph, using respective first and second weights for the first and second situation awareness graphs.

## Claims

### Claim 1

A method comprising:
obtaining first camera images from a first camera onboard a first vehicle;
generating, via one or more computer processors, a first situation awareness graph with respect to objects near the first vehicle, using the first camera images;
obtaining second camera images from a second camera of a second device that is in proximity to the first vehicle;
generating, via one or more computer processors, a second situation awareness graph with the respect to the objects, using the second camera images; and
generating, via one or more computer processor, a global situation awareness graph with respect to the objects, by merging the first situation awareness graph with the second situation awareness graph, using respective first and second weights for the first and second situation awareness graphs.

### Claim 2 (depends on 1)

The method of claim 1, wherein:
the step of generating the first situation awareness graph comprises generating a first static situation awareness graph with respect to objects near the first vehicle at a particular time, using the first camera images;
the step of generating the second situation awareness graph comprises generating a second static situation awareness graph with respect to the objects at the particular time, using the second camera images; and
the step of generating the global situation awareness graph comprises generating a static global awareness graph with respect to the objects at the particular time, by merging the first static situation awareness graph with the second static situation awareness graph, using respective first and second weights for the first and second static situation awareness graphs.

### Claim 3 (depends on 2)

The method of claim 2, wherein the step generating the static global situation awareness graph comprises generating the static global situation awareness graph using the respective first and second weights for the first and second static situation awareness graphs based on a principal components analysis of respective Mahalanobis Distances from the first and second static situation awareness graphs.

### Claim 4 (depends on 1)

The method of claim 1, wherein the step of generating the global situation awareness graph comprises generating, via one or more computer processors, a time-evolving dynamic global situational awareness map with respect to the objects using the first situation awareness graph and the second situation awareness graph over multiple periods of time, using respective first and second weights for the first and second situation awareness graphs.

### Claim 5 (depends on 1)

The method of claim 1, wherein the step of generating the global situation awareness graph comprises generating, via one or more computer processors, a multi-layer dynamic bipartite graph for predicting the trajectory of detected objects, using the first situation awareness graph and the second situation awareness graph over multiple periods of time, using respective first and second weights for the first and second situation awareness graphs.

### Claim 6 (depends on 1)

The method of claim 1, wherein the step of generating the global situation awareness graph comprises generating, via one or more computer processors, a multi-layer dynamic probabilistic graph for predicting the trajectory of detected objects, using initial and posterior probabilities from the first situation awareness graph and the second situation awareness graph over multiple periods of time, utilizing a Bayesian particle filter.

### Claim 7 (depends on 1)

The method of claim 1, wherein the second device comprises a second vehicle that is in proximity to the first vehicle.

### Claim 8 (depends on 1)

The method of claim 1, wherein the second device comprises infrastructure that is in proximity to the vehicle.

### Claim 9 (depends on 7)

The method of claim 7, further comprising:
estimating, via one or more processors, one or more parameters as to one or more occluded vehicles of the detected objects, based on the global situation awareness graph; and
controlling the first vehicle, the second vehicle, or both, in a manner to avoid contact with the one or more occluded vehicles, using the one or more parameters that were estimated based on the global situation awareness graph.

### Claim 10

A system comprising:
a first camera onboard a first vehicle, the first camera configured to generate first camera images from the first vehicle; and
one or more computer processors configured to at least facilitate:
generating a first situation awareness graph with respect to objects near the first vehicle, using the first camera images; and
generating a global situation awareness graph with respect to the objects, by merging the first situation awareness graph with a second situation awareness graph that was generated using second camera images from a second camera of a second device that is in proximity to the first vehicle, using respective first and second weights for the first and second situation awareness graphs.

### Claim 11 (depends on 10)

The system of claim 10, further comprising:
a transceiver configured to receive the second camera images from the second device, the second situation awareness graph, or both.

### Claim 12 (depends on 11)

The system of claim 11, wherein the second device comprises a second vehicle that is in proximity to the first vehicle, and the transceiver is further configured to transmit, via instructions from the one or more computer processors, the first camera images, the first situation awareness graph, or both, for use by the second vehicle.

### Claim 13 (depends on 10)

The system of claim 10, wherein the one or more processors are configured to at least facilitate:
generating a first static situation awareness graph with respect to objects near the first vehicle at a particular time, using the first camera images;
generating a second static situation awareness graph with respect to the objects at the particular time, using the second camera images; and
generating a static global awareness graph with respect to the objects at the particular time, by merging the first static situation awareness graph with the second static situation awareness graph, using respective first and second weights for the first and second static situation awareness graphs.

### Claim 14 (depends on 13)

The system of claim 13, wherein the one or more processors are configured to at least facilitate generating the static global situation awareness graph using the respective first and second weights for the first and second static situation awareness graphs based on a principal components analysis of respective Mahalanobis Distances from the first and second static situation awareness graphs.

### Claim 15 (depends on 10)

The system of claim 10, wherein the one or more processors are configured to at least facilitate generating a time-evolving dynamic global situational awareness map with respect to the objects using the first situation awareness graph and the second situation awareness graph over multiple periods of time, using respective first and second weights for the first and second situation awareness graphs.

### Claim 16 (depends on 10)

The system of claim 10, wherein the one or more processors are configured to at least facilitate generating a multi-layer dynamic bipartite graph for predicting the trajectory of detected objects, using the first situation awareness graph and the second situation awareness graph over multiple periods of time, using respective first and second weights for the first and second situation awareness graphs.

### Claim 17 (depends on 9)

The system of claim 9, wherein the one or more processors are configured to at least facilitate generating a multi-layer dynamic probabilistic graph for predicting the trajectory of detected objects, using initial and posterior probabilities from the first situation awareness graph and the second situation awareness graph over multiple periods of time, utilizing a Bayesian particle filter.

### Claim 18 (depends on 9)

The system of claim 9, wherein the second device comprises a second vehicle, and the one or more processors are configured to at least facilitate:
estimating one or more parameters as to one or more occluded vehicles of the detected objects, based on the global situation awareness graph; and
controlling the first vehicle, the second vehicle, or both, in a manner to avoid contact with the one or more occluded vehicles, using the one or more parameters that were estimated based on the global situation awareness graph.

### Claim 19

A vehicle comprising:
a first camera onboard the vehicle, the first camera configured to generate first camera images from the vehicle; and
one or more computer processors configured to at least facilitate:
generating a first situation awareness graph with respect to objects near the vehicle, using the first camera images; and
generating a global situation awareness graph with respect to the objects, by merging the first situation awareness graph with a second situation awareness graph that was generated using second camera images from a second camera of a second device that is in proximity to the vehicle, using respective first and second weights for the first and second situation the first camera images.

### Claim 20 (depends on 19)

The vehicle of claim 19, further comprising:
a transceiver configured to receive the second camera images from the second device, the second situation awareness graph, or both, and to transmit, via instructions from the one or more computer processors, the first camera images, the first situation awareness graph, or both, for use by the second device.
