# US 2019/0168772 A1 · Driving assistance method, and driving assistance device, driving control device, vehicle, driving assistance program, and recording medium using said method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0168772 A1
- Document ID: US20190168772A1
- Status: Published application
- Kind: A1
- Application number: 16/255,338
- Filed: 2019-01-23
- Published: 2019-06-06
- Earliest claimed: 2015-04-21
- Assignees: Panasonic Intellectual Property Management Co., Ltd.
- Applicants: Panasonic Intellectual Property Management Co., Ltd.
- Inventors: KOICHI EMURA; MASANAGA TSUJI; TOSHIYA MORI; WATARU NAKAI
- Web page: https://patentgrep.com/patent/US20190168772A1

## CPC

- B60W 50/10: Interpretation of driver requests or demands
- B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
- B60K 35/21: using visual output, e.g. blinking lights or matrix displays
- B60K 35/22: Display screens
- B60K 35/23: Head-up displays [HUD] (optical aspects of head-up displays G02B27/01)
- B60K 35/265: {Voice}
- B60K 35/50: Instruments characterised by their means of attachment to or integration in the vehicle (B60K35/231 takes precedence)
- B60K 35/60: Instruments characterised by their location or relative disposition in or on vehicles (arrangements of lighting devices on dashboards B60Q3/10)
- B60K 35/80: Arrangements for controlling instruments
- B60K 35/90: Calibration of instruments, e.g. setting initial or reference parameters; Testing of instruments, e.g. detecting malfunction
- B60R 16/02: electric {constitutive elements}
- B60W 30/09: Taking automatic action to avoid collision, e.g. braking and steering
- B60W 30/18163: {Lane change; Overtaking manoeuvres}
- B60W 40/09: Driving style or behaviour
- B60W 50/08: Interaction between the driver and the control system
- B60W 50/082: {Selecting or switching between different modes of propelling}
- B60W 50/085: {Changing the parameters of the control units, e.g. changing limit values, working points by control input}
- B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention
- B60W 60/0027: {using trajectory prediction for other traffic participants}
- B60W 60/0053: {from vehicle to occupant}
- G01C 21/3484: {Personalized, e.g. from learned user behaviour or user-defined profiles}
- G01C 21/3605: {Destination input or retrieval}
- G01C 21/3617: {using user history, behaviour, conditions or preferences, e.g. predicted or inferred from previous use or current movement}
- G05D 1/0257: {using a radar (radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931)}
- G06F 3/048: Interaction techniques based on graphical user interfaces [GUI]
- G06V 20/20: in augmented reality scenes
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/59: inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- G06V 40/172: {Classification, e.g. identification}
- G08G 1/0962: having an indicator mounted inside the vehicle, e.g. giving voice messages
- G08G 1/096888: {where input information is obtained using learning systems, e.g. history databases}
- B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
- B60K 2360/166: Navigation
- B60K 2360/167: Vehicle dynamics information
- B60K 2360/175: Autonomous driving
- B60K 2360/179: Distances to obstacles or vehicles
- B60W 2050/0067: {Confirmation by the driver}
- B60W 2050/0075: {Automatic parameter input, automatic initialising or calibrating means}
- B60W 2050/146: {Display means}
- B60W 2520/10: Longitudinal speed
- B60W 2540/10: Accelerator pedal position
- B60W 2540/12: Brake pedal position
- B60W 2540/18: Steering angle
- B60W 2540/20: Direction indicator values
- B60W 2540/215: Selection or confirmation of options
- B60W 2540/30: Driving style
- B60W 2554/00: Input parameters relating to objects
- B60W 2554/802: Longitudinal distance
- B60W 2554/804: Relative longitudinal speed
- B60W 2556/10: Historical data
- B60W 2556/55: using telemetry
- B60W 2720/10: Longitudinal speed
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})

## Abstract

The driving assistance device acquires, from an autonomous driving controller that determines an action of a vehicle during autonomous driving of the vehicle, action information indicating a first action that the vehicle is caused to execute. The driving assistance device acquires, from a detector that detects a surrounding situation and a travel state of the vehicle, detection information indicating a detection result. The driving assistance device determines a second action which is executable in place of the first action, based on the detection information. The driving assistance device generates a first image representing the first action and a second image representing the second action. The driving assistance device outputs the first image and the second image to a notification device such that the first image and the second image are displayed within a fixed field of view of a driver of the vehicle.

## Claims

### Claim 1

A driving assistance device comprising:
an action information input unit, which, in operation, acquires, from an autonomous driving controller that determines an action of a vehicle during autonomous driving of the vehicle, action information indicating a first action that the vehicle is caused to execute;
a detection information input unit, which, in operation, acquires, from a detector that detects a surrounding situation and a travel state of the vehicle, detection information indicating a detection result;
a candidate determination unit, which, in operation, determines, based on the detection information, at least one second action that the vehicle is caused to execute after the first action indicated by the action information, the first action and the at least one second action adjusting movement of the vehicle;
an image generator, which, in operation, generates a first image representing the first action indicated by the action information and a second image representing the at least one second action; and
an image output unit, which, in operation, outputs the first image and the second image to a display in the vehicle.

### Claim 2 (depends on 1)

The driving assistance device according to claim 1, further comprising:
a command output unit, which, in operation, outputs, to the autonomous driving controller, a command to cause the vehicle to execute the at least one second action, based on a reception status of a predetermined signal related to a designation of the action of the vehicle.

### Claim 3 (depends on 2)

The driving assistance device according to claim 2, wherein the command output unit outputs the command to the autonomous driving controller, based on whether or not the predetermined signal is received within a predetermined time for which the first image and the second image are displayed on the display in the vehicle.

### Claim 4 (depends on 3)

The driving assistance device according to claim 3, further comprising:
an operation signal input unit, which, in operation, receives the predetermined signal,
wherein the command output unit outputs the command to the autonomous driving controller in response to the predetermined signal not being received within the predetermined time.

### Claim 5 (depends on 3)

The driving assistance device according to claim 3, further comprising:
an operation signal input unit, which, in operation, receives the predetermined signal, wherein
the command output unit outputs the command to the autonomous driving controller in response to the predetermined signal being received within the predetermined time, and
the predetermined signal is a signal indicating that the second image is selected.

### Claim 6 (depends on 2)

The driving assistance device according to claim 2, wherein the image output unit, in operation, outputs, to the display, an inquiry image for designating execution or reservation of the at least one second action, based on the reception status of the predetermined signal.

### Claim 7 (depends on 6)

The driving assistance device according to claim 6, wherein the predetermined signal is a signal indicating an operation for designating the at least one second action,
in response to the predetermined signal being received, the image output unit outputs the inquiry image to the display,
in response to an operation for designating execution being received while the inquiry image is displayed, the command output unit outputs the command to the autonomous driving controller at a first timing, and
in response to an operation for designating reservation being received while the inquiry image is displayed, the command output unit outputs the command to the autonomous driving controller at a second timing that is later than the first timing.

### Claim 8 (depends on 7)

The driving assistance device according to claim 7, wherein in response to the operation for designating the reservation being received while the inquiry image is displayed, the image output unit outputs, to the display, a third image representing that the at least one second action is currently reserved.

### Claim 9 (depends on 6)

The driving assistance device according to claim 6, further comprising:
an operation signal input unit, which, in operation, receives the predetermined signal, wherein
the predetermined signal is a signal indicating that the second image is selected,
in response to the predetermined signal being received within a predetermined time for which the first image and the second image are displayed on the display in the vehicle, the image generator generates the inquiry image, and
the inquiry image is an image for making an inquiry to a driver of the vehicle as to whether to immediately execute or to execute after a predetermined time the at least one second action represented by the second image which has been selected.

### Claim 10 (depends on 9)

The driving assistance device according to claim 9, wherein, in response to the operation signal input unit receiving a signal indicating that immediate execution of the at least one second action represented by the second image is selected in the inquiry image, the command output unit transmits, to the autonomous driving controller, a control command to cause the vehicle to immediately execute the at least one second action represented by the second image in place of the first action represented by the first image.

### Claim 11 (depends on 9)

The driving assistance device according to claim 9, wherein, in response to the operation signal input unit receiving a signal indicating that execution of the at least one second action represented by the second image after a predetermined time is selected in the inquiry image, the command output unit transmits, to the autonomous driving controller, a control command to cause the vehicle to execute the at least one second action represented by the second image after a predetermined time.

### Claim 12 (depends on 1)

The driving assistance device according to claim 1, wherein
the image generator, in operation, generates an additional image representing a time until the at least one second action is executed, and
the image output unit, in operation, outputs the additional image to the display in the vehicle to display the second image including the additional image.

### Claim 13 (depends on 1)

The driving assistance device according to claim 1, further comprising:
an accumulation unit, which, in operation, accumulates statistical information indicating relevance between the surrounding situation and the travel state of the vehicle and the action of the vehicle,
wherein the candidate determination unit, in operation, determines the at least one second action based on the statistical information and the detection information.

### Claim 14 (depends on 1)

The driving assistance device according to claim 1, further comprising:
a communication interface, which, in operation, communicates with, through a communication network, an accumulation unit, provided outside of the vehicle, configured to accumulate statistical information indicating relevance between the surrounding situation and the travel state of the vehicle and the action of the vehicle,
wherein the candidate determination unit, in operation, accesses the statistical information in the accumulation unit through the communication interface, and determines the at least one second action based on the statistical information and the detection information.

### Claim 15 (depends on 1)

The driving assistance device according to claim 1, further comprising:
a command output unit, which, in operation, outputs, to the autonomous driving controller, a command to cause the vehicle to execute the at least one second action, after the first image and the second image are displayed on the display in the vehicle for a predetermined time.

### Claim 16 (depends on 15)

The driving assistance device according to claim 15, wherein
after the command is output to the autonomous driving controller, the action information input unit acquires first action information indicating a first action which has been updated according to the command.

### Claim 17

A driving control device comprising:
an autonomous driving controller, which, in operation, determines an action of a vehicle during autonomous driving of the vehicle;
a detection information input unit, which, in operation, acquires, from a detector that detects a surrounding situation and a travel state of the vehicle, detection information indicating a detection result;
a candidate determination unit, which, in operation, determines, based on the detection information, at least one second action that the vehicle can be caused to execute after a first action that the autonomous driving controller causes the vehicle to execute, the first action and the at least one second action adjusting movement of the vehicle;
an image generator, which, in operation, generates a first image representing the first action and a second image representing the at least one second action; and
an image output unit which outputs the first image and the second image to a display in the vehicle.

### Claim 18 (depends on 17)

The driving control device according to claim 17, further comprising:
a command output unit, which, in operation, outputs, to the autonomous driving controller, a command to cause the vehicle to execute the at least one second action, based on a reception status of a predetermined signal related to a designation of the action of the vehicle

### Claim 19

A driving assistance method executed by a computer, the method comprising:
acquiring, from an autonomous driving controller that determines an action of a vehicle during autonomous driving of the vehicle, action information indicating a first action that the vehicle is caused to execute;
acquiring, from a detector that detects a surrounding situation and a travel state of the vehicle, detection information indicating a detection result;
determining, based on the detection information, at least one second action that the vehicle can be caused to execute after the first action indicated by the action information, the first action and the at least one second action adjusting movement of the vehicle;
generating a first image representing the first action indicated by the action information and a second image representing the second action; and
outputting the first image and the second image to a display in the vehicle.

### Claim 20 (depends on 19)

The driving assistance method according to claim 19, further comprising:
outputting, to the autonomous driving controller, a command to cause the vehicle to execute the at least one second action, based on a reception status of a predetermined signal related to a designation of the action of the vehicle.