# US 2021/0081683 A1 · Vehicular control system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0081683 A1
- Document ID: US20210081683A1
- Status: Published application
- Kind: A1
- Application number: 16/949,812
- Filed: 2020-11-16
- Published: 2021-03-18
- Earliest claimed: 2009-02-27
- Assignees: Magna Electronics Inc.
- Applicants: Magna Electronics Inc.
- Inventors: Niall R. Lynam; Axel Nix
- Web page: https://patentgrep.com/patent/US20210081683A1

## CPC

- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- B60Q 9/005: {using a video camera}
- B60R 1/12: Mirror assemblies combined with other articles, e.g. clocks
- B60R 1/26: to the rear of the vehicle
- B60R 1/27: providing all-round vision, e.g. using omnidirectional cameras
- B60R 1/28: with an adjustable field of view
- B60W 10/04: including control of propulsion units
- B60W 10/18: including control of braking systems
- B60W 10/20: including control of steering systems
- B60W 40/02: related to ambient conditions
- B60W 50/0098: {Details of control systems ensuring comfort, safety or stability not otherwise provided for}
- G05D 1/81: Handing over between on-board automatic and on-board manual control
- H04N 23/63: by using electronic viewfinders
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- B60R 2001/1215: {with information displays}
- B60R 2001/1253: {with cameras, video cameras or video screens}
- B60R 2300/105: using multiple cameras
- B60R 2300/302: combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
- B60R 2300/303: using joined images, e.g. multiple camera images
- B60R 2300/605: the adjustment being automatic
- B60R 2300/607: from a bird's eye viewpoint
- B60R 2300/806: for aiding parking
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60Q: ARRANGEMENT OF SIGNALLING OR LIGHTING DEVICES, THE MOUNTING OR SUPPORTING THEREOF OR CIRCUITS THEREFOR, FOR VEHICLES IN GENERAL
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A vehicular control system includes a central control module vehicle, a plurality of image capture sensors, at least one radar sensor and at least one lidar sensor disposed at an equipped. The central control module receives vehicle data relating to operation of the vehicle. During a driving maneuver of the vehicle, and responsive at least in part to fusion at the central control module of (i) captured image data, (ii) captured radar data and (iii) captured lidar data, a pedestrian or other vehicle may be detected or the speed of the equipped vehicle may be controlled. Responsive to the central control module determining that the detected pedestrian constitutes a potentially hazardous driving condition for the equipped vehicle, the central control module may control at least one selected from the group consisting of (i) braking of the equipped vehicle and (ii) steering of the equipped vehicle.

## Claims

### Claim 1

A vehicular control system, said vehicular control system comprising:
a central control module disposed at a vehicle equipped with said vehicular control system;
wherein said central control module comprises at least one data processor;
a plurality of image capture sensors disposed at the equipped vehicle;
wherein said plurality of image capture sensors comprises at least (i) a forward viewing image capture sensor disposed behind a windshield of the equipped vehicle and viewing through the windshield at least forward of the equipped vehicle, (ii) a rearward viewing image capture sensor disposed at the equipped vehicle and viewing at least rearward of the equipped vehicle, (iii) a driver-side sideward viewing image capture sensor disposed at a driver side of the equipped vehicle and viewing at least sideward of the driver side of the equipped vehicle and (iv) a passenger-side sideward viewing image capture sensor disposed at a passenger side of the equipped vehicle and viewing at least sideward of the passenger side of the equipped vehicle;
wherein image data captured by said forward viewing image capture sensor and image data captured by at least one other image capture sensor of said plurality of image capture sensors is provided to and processed at said central control module;
at least one radar sensor disposed at the equipped vehicle and sensing forward of the equipped vehicle;
wherein radar data captured by said at least one radar sensor is provided to and processed at said central control module;
at least one lidar sensor disposed at the equipped vehicle and sensing forward of the equipped vehicle;
wherein lidar data captured by said at least one lidar sensor is provided to and processed at said central control module;
wherein said central control module receives vehicle data relating to operation of the equipped vehicle;
wherein, during a driving maneuver of the equipped vehicle, and responsive at least in part to fusion at said central control module of (i) captured image data, (ii) captured radar data and (iii) captured lidar data, a pedestrian present exterior of the equipped vehicle is detected; and
wherein, responsive at least in part to processing at said central control module of received vehicle data, and responsive to said central control module determining that the detected pedestrian constitutes a potentially hazardous driving condition for the equipped vehicle, said central control module controls at least one selected from the group consisting of (i) braking of the equipped vehicle and (ii) steering of the equipped vehicle.

### Claim 2 (depends on 1)

The vehicular control system of claim 1, wherein said at least one radar sensor comprises a front radar sensor mounted at a front portion of the equipped vehicle.

### Claim 3 (depends on 1)

The vehicular control system of claim 1, wherein said central control module controls braking of the equipped vehicle responsive at least in part to processing at said central control module of image data captured by the at least one other image capture sensor of said plurality of image capture sensors.

### Claim 4 (depends on 3)

The vehicular control system of claim 3, wherein said central control module controls braking of the equipped vehicle responsive at least in part to processing of radar data captured by at least one other radar sensor disposed at the equipped vehicle and sensing exterior of the equipped vehicle.

### Claim 5 (depends on 1)

The vehicular control system of claim 1, wherein said at least one data processor and a fusion core are co-established on a common substrate.

### Claim 6 (depends on 1)

The vehicular control system of claim 1, wherein said at least one radar sensor comprises at least (i) a front radar sensor mounted at a front portion of the equipped vehicle and having a field of sensing at least forward of the equipped vehicle, (ii) a driver-side radar sensor mounted at a driver-side portion of the equipped vehicle and having a field of sensing including at least a driver-side traffic lane to the left of a traffic lane the equipped vehicle is travelling in and (iii) a passenger-side radar sensor mounted at a passenger-side portion of the equipped vehicle and having a field of sensing including at least a passenger-side traffic lane to the right of the traffic lane the equipped vehicle is travelling in.

### Claim 7 (depends on 1)

The vehicular control system of claim 1, wherein said at least one lidar sensor comprises a scanning lidar sensor.

### Claim 8 (depends on 1)

The vehicular control system of claim 1, wherein said at least one lidar sensor comprises a three-dimensional sensing lidar sensor.

### Claim 9 (depends on 8)

The vehicular control system of claim 8, wherein said at least one lidar sensor comprises an array of infrared emitting light emitters.

### Claim 10 (depends on 1)

The vehicular control system of claim 1, wherein image data captured by said rearward viewing image capture sensor is provided to said central control module, image data captured by said driver-side sideward viewing image capture sensor is provided to said central control module and image data captured by said passenger-side sideward viewing image capture sensor is provided to said central control module.

### Claim 11 (depends on 1)

The vehicular control system of claim 1, wherein data pertaining to a geographical location of the equipped vehicle is provided to said central control module, and wherein said central control module controls the equipped vehicle responsive to (i) processing at said central control module of vehicle data received at said central control module, (ii) processing at said central control module of image data captured by at least said forward viewing image capture sensor, (iii) processing at said central control module of radar data captured by said at least one radar sensor, (iv) processing at said central control module of lidar data captured by said at least one lidar sensor and (v) processing at said central control module of data pertaining to the geographical location of the equipped vehicle.

### Claim 12 (depends on 1)

The vehicular control system of claim 1, wherein image data captured by at least two image capture sensors of said plurality of image capture sensors is processed at said central control module to determine a driving situation, and wherein, in determining said driving situation, processing of image data at said central control module is augmented by processing at said central control module of radar data captured by said at least one radar sensor.

### Claim 13 (depends on 12)

The vehicular control system of claim 12, wherein captured image data is communicated from said at least two of said plurality of image capture sensors to said central control module via respective Ethernet links.

### Claim 14 (depends on 1)

The vehicular control system of claim 1, wherein, responsive at least in part to fusion at said central control module of (i) captured image data, (ii) captured radar data and (iii) captured lidar data, said central control module controls braking of the equipped vehicle.

### Claim 15 (depends on 14)

The vehicular control system of claim 14, wherein, responsive at least in part to fusion at said central control module of (i) captured image data, (ii) captured radar data and (iii) captured lidar data, said central control module controls steering of the equipped vehicle.

### Claim 16 (depends on 14)

The vehicular control system of claim 14, wherein, responsive at least in part to fusion at said central control module of (i) captured image data, (ii) captured radar data and (iii) captured lidar data, said central control module controls acceleration of the equipped vehicle.

### Claim 17 (depends on 14)

The vehicular control system of claim 14, wherein, responsive at least in part to fusion at said central control module of (i) captured image data, (ii) captured radar data and (iii) captured lidar data, said central control module controls speed of the equipped vehicle.

### Claim 18 (depends on 14)

The vehicular control system of claim 14, wherein said central control module controls braking of the equipped vehicle responsive at least in part to image processing at said central control module of image data captured by at least said forward viewing image capture sensor of said plurality of image capture sensors, and responsive at least in part to processing at said central control module of radar data captured by said at least one radar sensor, and responsive at least in part to processing at said central control module of lidar data captured by said at least one lidar sensor.

### Claim 19 (depends on 1)

The vehicular control system of claim 1, wherein said driver-side sideward viewing image capture sensor is disposed at a driver-side exterior rearview mirror assembly of the equipped vehicle, and wherein said passenger-side sideward viewing image capture sensor is disposed at a passenger-side exterior rearview mirror assembly of the equipped vehicle.

### Claim 20 (depends on 1)

The vehicular control system of claim 1, wherein (i) said forward viewing image capture sensor of said plurality of image capture sensors comprises an at least 1 Megapixel image capture sensor, (ii) said rearward viewing image capture sensor of said plurality of image capture sensors comprises an at least 1 Megapixel image capture sensor, (iii) said driver-side sideward viewing image capture sensor of said plurality of image capture sensors comprises an at least 1 Megapixel image capture sensor and (iv) said passenger-side sideward viewing image capture sensor of said plurality of image capture sensors comprises an at least 1 Megapixel image capture sensor.

### Claim 21 (depends on 1)

The vehicular control system of claim 1, wherein said central control module controls the equipped vehicle responsive to data pertaining to a geographical location of the equipped vehicle.

### Claim 22 (depends on 1)

The vehicular control system of claim 1, wherein the received vehicle data comprises at least one selected from the group consisting of (i) vehicle speed data, (ii) vehicle steering data, (iii) vehicle yaw rate data and (iv) vehicle acceleration data.

### Claim 23 (depends on 1)

The vehicular control system of claim 1, wherein the received vehicle data comprises (i) vehicle speed data and (ii) vehicle steering data.

### Claim 24

A vehicular control system, said vehicular control system comprising:
a central control module disposed at a vehicle equipped with said vehicular control system;
wherein said central control module comprises at least one data processor;
a camera module disposed at an in-cabin side of a windshield of the equipped vehicle, said camera module comprising a forward viewing image capture sensor viewing through the windshield at least forward of the equipped vehicle;
wherein said forward viewing image capture sensor comprises an at least 1 Megapixel image capture sensor;
wherein image data captured by said forward viewing image capture sensor is provided to and processed at said central control module;
a front radar sensor mounted at a front portion of the equipped vehicle and sensing forward of the equipped vehicle;
wherein radar data captured by said front radar sensor is provided to and processed at said central control module;
a lidar sensor disposed at the equipped vehicle and sensing forward of the equipped vehicle;
wherein said lidar sensor comprises a three-dimensional sensing lidar sensor;
wherein lidar data captured by said lidar sensor is provided to and processed at said central control module;
wherein said central control module receives vehicle data relating to operation of the equipped vehicle;
wherein, during a driving maneuver of the equipped vehicle, and responsive at least in part to fusion at said central control module of (i) captured image data, (ii) captured radar data and (iii) captured lidar data, another vehicle present exterior of the equipped vehicle is detected; and
wherein, responsive at least in part to processing at said central control module of received vehicle data, and responsive to said central control module determining that the detected other vehicle constitutes a potentially hazardous driving condition for the equipped vehicle, said central control module controls at least one selected from the group consisting of (i) braking of the equipped vehicle and (ii) steering of the equipped vehicle.

### Claim 25 (depends on 24)

The vehicular control system of claim 24, wherein, responsive to said central control module determining that the detected other vehicle constitutes a potentially hazardous driving condition for the equipped vehicle, said central control module controls braking of the equipped vehicle and steering of the equipped vehicle.

### Claim 26 (depends on 24)

The vehicular control system of claim 24, wherein said central control module controls the equipped vehicle responsive to data pertaining to a geographical location of the equipped vehicle.

### Claim 27 (depends on 26)

The vehicular control system of claim 26, wherein the received vehicle data comprises at least one selected from the group consisting of (i) vehicle speed data, (ii) vehicle steering data, (iii) vehicle yaw rate data and (iv) vehicle acceleration data.

### Claim 28 (depends on 24)

The vehicular control system of claim 24, wherein the received vehicle data comprises vehicle speed data and vehicle steering data.

### Claim 29

A vehicular control system, said vehicular control system comprising:
a central control module disposed at a vehicle equipped with said vehicular control system;
wherein said central control module comprises at least one data processor;
a camera module disposed at an in-cabin side of a windshield of the equipped vehicle, said camera module comprising a forward viewing image capture sensor viewing through the windshield at least forward of the equipped vehicle;
wherein said forward viewing image capture sensor comprises an at least 1 Megapixel image capture sensor;
wherein image data captured by said forward viewing image capture sensor is provided to and processed at said central control module;
a front radar sensor mounted at a front portion of the equipped vehicle and sensing forward of the equipped vehicle;
wherein radar data captured by said front radar sensor is provided to and processed at said central control module;
a lidar sensor disposed at the equipped vehicle and sensing forward of the equipped vehicle;
wherein said lidar sensor comprises a three-dimensional sensing lidar sensor;
wherein lidar data captured by said lidar sensor is provided to and processed at said central control module;
wherein said central control module receives vehicle data relating to operation of the equipped vehicle;
wherein the received vehicle data comprises vehicle speed data and vehicle steering data; and
wherein, responsive at least in part to fusion at said central control module of (i) captured image data, (ii) captured radar data, (iii) captured lidar data and (iv) data pertaining to the geographical location of the equipped vehicle, said central control module controls speed of the equipped vehicle.

### Claim 30 (depends on 29)

The vehicular control system of claim 29, wherein the received vehicle data comprises vehicle yaw rate data.

### Claim 31 (depends on 29)

The vehicular control system of claim 29, wherein the received vehicle data comprises vehicle acceleration data.

### Claim 32 (depends on 29)

The vehicular control system of claim 29, wherein, responsive at least in part to fusion at said central control module of (i) captured image data, (ii) captured radar data and (iii) captured lidar data, said central control module controls braking of the equipped vehicle.

### Claim 33 (depends on 32)

The vehicular control system of claim 32, wherein, responsive at least in part to fusion at said central control module of (i) captured image data, (ii) captured radar data and (iii) captured lidar data, said central control module controls steering of the equipped vehicle.

### Claim 34 (depends on 33)

The vehicular control system of claim 33, wherein, responsive at least in part to fusion at said central control module of (i) captured image data, (ii) captured radar data and (iii) captured lidar data, said central control module controls acceleration of the equipped vehicle.

### Claim 35 (depends on 29)

The vehicular control system of claim 29, wherein, during a driving maneuver of the equipped vehicle, and responsive at least in part to fusion at said central control module of (i) captured image data, (ii) captured radar data and (iii) captured lidar data, a pedestrian exterior of the equipped vehicle is detected.

### Claim 36 (depends on 35)

The vehicular control system of claim 35, wherein, responsive at least in part to processing at said central control module of received vehicle data, and responsive to said central control module determining that the detected pedestrian constitutes a potentially hazardous driving condition for the equipped vehicle, said central control module controls braking of the equipped vehicle.

### Claim 37 (depends on 35)

The vehicular control system of claim 35, wherein, responsive at least in part to processing at said central control module of received vehicle data, and responsive to said central control module determining that the detected pedestrian constitutes a potentially hazardous driving condition for the equipped vehicle, said central control module controls steering of the equipped vehicle.
