# US 10,864,906 B2 · Method of switching vehicle drive mode from automatic drive mode to manual drive mode depending on accuracy of detecting object

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,864,906 B2
- Document ID: US10864906B2
- Status: Granted
- Kind: B2
- Application number: 15/446,525
- Filed: 2017-03-01
- Published: 2017-09-14
- Granted: 2020-12-15
- Earliest claimed: 2016-03-10
- Assignees: Panasonic Intellectual Property Corporation of America
- Applicants: Panasonic Intellectual Property Corporation of America
- Inventors: Ryota Fujimura; Kazuki Kozuka; Tetsuji Fuchikami
- Web page: https://patentgrep.com/patent/US10864906B2

## CPC

- B60W 30/09: Taking automatic action to avoid collision, e.g. braking and steering
- B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention
- B60W 60/0051: {from occupants to vehicle}
- B60W 60/0053: {from vehicle to occupant}
- B60W 60/0059: {Estimation of the risk associated with autonomous or manual driving, e.g. situation too complex, sensor failure or driver incapacity}
- G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00)
- G05D 1/0238: {using obstacle or wall sensors (G05D1/0246 and G05D1/0289 take precedence; lidar systems designed for anti-collision purposes G01S17/93)}
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G08G 1/16: Anti-collision systems (road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08)
- B60W 2050/0072: {Controller asks driver to take over}
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2420/408: Radar; Laser, e.g. lidar
- B60W 2520/06: Direction of travel
- B60W 2520/10: Longitudinal speed
- B60W 2552/00: Input parameters relating to infrastructure
- B60W 2552/20: Road profile, i.e. the change in elevation or curvature of a plurality of continuous road segments
- B60W 2552/53: Road markings, e.g. lane marker or crosswalk
- B60W 2554/00: Input parameters relating to objects
- B60W 2554/4029: Pedestrians
- B60W 2554/4041: Position
- B60W 2555/60: Traffic rules, e.g. speed limits or right of way
- B60W 2556/00: Input parameters relating to data
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})

## Abstract

An apparatus includes a memory, and circuitry which, in operation, performs operations including, storing, in the memory, an object occurrence map defining an occurrence area where there is a possibility that an object appears, detecting the object included in a captured image of a scene seen in a running direction of a vehicle, switching a vehicle drive mode, based on a result of the detection of the object and the object occurrence map, from an automatic drive mode in which the vehicle is automatically driven to a manual drive mode in which the vehicle is driven manually by a driver, and controlling driving of the vehicle in the switched manual drive mode.

## Claims

### Claim 1

An apparatus for use by a vehicle configured to operate in an automatic drive mode, in which the vehicle is automatically driven, and a manual drive mode, in which the vehicle is driven manually by a driver, comprising:
a first memory; and
circuitry that, in operation, performs operations including
storing, in the first memory, an object occurrence map defining an object occurrence area representing a predefined area of a scene of predetermined dimensions
i) outside the vehicle path when viewed from the vehicle in a running direction of the vehicle,
ii) smaller than the entire area of the scene outside the vehicle path when viewed from the vehicle in the running direction of the vehicle, and
iii) based on data of past detections of an object in the area of the scene outside the vehicle path, where there is a higher probability that the object will exist in the scene than in other areas outside the vehicle and detectable by a sensor of the vehicle;
detecting the object included in a captured image of the scene seen in the running direction of the vehicle;
determining whether the detected object is outside the object occurrence area;
determining that the detecting of the object is unreliable when it is determined that the object is outside the object occurrence area;
switching a vehicle drive mode to the manual drive mode in response to determining that the detecting of the object is unreliable as a result of the detected object being detected to be outside the object occurrence area;
maintaining the vehicle in the automatic drive mode in response to determining that the detected object is not outside the object occurrence area; and
controlling driving of the vehicle in the switched manual drive mode when the drive mode is switched and controlling the driving of vehicle in the automatic drive mode when the drive mode is not switched.

### Claim 2 (depends on 1)

The apparatus according to claim 1, wherein the object occurrence map is managed in relation to map information.

### Claim 3 (depends on 1)

The apparatus according to claim 1,
wherein the operations further include
acquiring a running circumstance around the vehicle; and
depending on the acquired running circumstance, generating a viewpoint-converted object occurrence map from the object occurrence map so as to be represented from the same viewpoint as that of the captured image,
wherein the switching operation includes
superimposing the viewpoint-converted object occurrence map on the captured image, and
in a case where a location of the object detected in the captured image is outside the occurrence area in the viewpoint-converted object occurrence map, switching the vehicle drive mode from the automatic drive mode to the manual drive mode.

### Claim 4 (depends on 3)

The apparatus according to claim 3, wherein
the map includes a three-dimensional map represented in a three-dimensional coordinate system,
the occurrence area includes an occurrence space in which there is a possibility that the object appears on the three-dimensional map, and
the generating operation includes converting the object occurrence map in the three-dimensional coordinate system to the viewpoint-converted object occurrence map in a two-dimensional coordinate system as seen from the same viewpoint as that of the captured image.

### Claim 5 (depends on 1)

The apparatus according to claim 1,
wherein, in the storing operation, a plurality of object occurrence maps are stored that are respectively represented in the two-dimensional coordinate system as seen from a same viewpoint as that of the captured image, and that respectively define occurrence areas depending on shapes of a plurality of roads, and
wherein the operations further include
acquiring a shape of a road included in the captured image of the scene seen in the travelling direction of the vehicle, and
determining one object occurrence map corresponding to the acquired shape of the road from the plurality of object occurrence maps.

### Claim 6 (depends on 1)

The apparatus according to claim 1,
wherein the map includes a three-dimensional map represented in a three-dimensional coordinate system, and the occurrence area includes an occurrence space in which there is a possibility that the object appears on the three-dimensional map,
wherein in the storing operation, a plurality of object occurrence maps are stored that are respectively represented in the three-dimensional coordinate system, and that respectively define occurrence spaces depending on shapes of a plurality of roads, and
wherein the operations further include
acquiring a shape of a road included in the captured image of the scene seen in the travelling direction of the vehicle,
determining one object occurrence map corresponding to the acquired shape of the road from the plurality of object occurrence maps, and
in a case where the one object occurrence map is determined, converting the one object occurrence map in the three-dimensional coordinate system to an object occurrence map in the same two-dimensional coordinate system as that of the captured image.

### Claim 7 (depends on 1)

The apparatus according to claim 1, further comprising:
a second memory,
wherein in the storing operation, a plurality of object occurrence maps are stored that are respectively represented in the two-dimensional coordinate system as seen from the same viewpoint as that of the captured image, and that respectively define occurrence areas depending on shapes of a plurality of roads,
wherein the operations further include
determining, depending on a location of the object detected in the captured image, one object occurrence map from the plurality of object occurrence maps, and
storing, in the second memory, an identification number corresponding to the determined one object occurrence map, and
wherein in the switching operation, in a case where the identification number stored in the second memory is switched a predetermined number of times during a predetermined period of time, the vehicle drive mode is switched from the automatic drive mode to the manual drive mode.

### Claim 8 (depends on 1)

The apparatus according to claim 1, wherein
the object is a person, and
the occurrence area is an area where the person passes.

### Claim 9 (depends on 1)

The apparatus according to claim 1, wherein the circuitry also performs operations including:
detecting the shape of a road on which the vehicle travels from a scene outside the running vehicle, captured in the running direction of the vehicle; and
determining the dimensions and shape of an object occurrence map to match the detected shape of the road,
wherein the storing operation stores, in the first memory, the object occurrence map whose dimensions and shape are based on the detected shape of the road, the object occurrence map whose dimensions and shape are based on the detected shape of the road defining the occurrence area representing the predefined area of the scene of predetermined dimensions based on data of past detections of the object in the scene, where there is a probability that the object will exist in the scene.

### Claim 10 (depends on 1)

The apparatus according to claim 1, wherein the circuitry also performs operations including:
detecting traffic signal information on the state of a traffic signal of a road on which the vehicle travels; and
detecting whether two objects of the same type are present at the same time in the object occurrence map defining the occurrence area representing the predefined area of the scene of predetermined dimensions outside the vehicle in the direction in which the vehicle travels, wherein the switching operation switches the vehicle drive mode, based on a result of the detection of the traffic signal information and the detection of whether two objects of the same type are present in an object occurrence map, from the automatic drive mode, in which the vehicle is automatically driven, to the manual drive mode, in which the vehicle is driven manually by the driver.

### Claim 11 (depends on 1)

The apparatus according to claim 1, wherein the circuitry, in operation, further performs an operation including when the vehicle drive mode is in the manual drive mode, and it is determined that the location of the detected person is not outside the occurrence area, switching the vehicle drive mode from the manual drive mode to the automatic drive mode.

### Claim 12 (depends on 1)

The apparatus according to claim 1, wherein the circuitry, in operation, further performs operations including:
generating the object occurrence map for superimposition on the captured image; and
superimposing the object occurrence map for superimposition on the captured image and determining whether the detected object is outside the object occurrence area,
wherein the circuitry operations are performed in the following order,
1) the detecting operation,
2) the generating operation,
3) the superimposing operation, and
4) the switching and maintaining operations.

### Claim 13 (depends on 1)

The apparatus according to claim 1, wherein the circuitry, in operation, further performs an operation including switching from the automatic drive mode to the manual drive mode in response to switching of a determined object occurrence map image a predetermined number of times during a predetermined period of time as the object occurrence map is determined repeatedly depending on the location of a detected object as the vehicle travels on the vehicle path.

### Claim 14

A method for use by a vehicle configured to operate in an automatic drive mode, in which the vehicle is automatically driven, and a manual drive mode, in which the vehicle is driven manually by a driver, comprising:
detecting an object included in a captured image of a scene seen in a running direction of a vehicle;
determining whether the detected object is outside an object occurrence area of an object occurrence map representing a predefined area of the scene of predetermined dimensions
i) outside the vehicle path when viewed from the vehicle in a running direction of the vehicle,
ii) smaller than the entire area of the scene outside the vehicle path when viewed from the vehicle in the running direction of the vehicle, and
iii) based on data of past detections of the object in the area of the scene outside the vehicle path, where there is a higher probability that the object will exist in the scene than in other areas outside the vehicle and detectable by a sensor of the vehicle;
determining that the detecting of the object is unreliable when it is determined that the object is outside the object occurrence area;
switching a vehicle drive mode from the automatic drive mode to the manual drive mode in response to determining that the detecting of the object is unreliable as a result of the detected object being detected to be outside the object occurrence area;
maintaining the vehicle in the automatic drive mode in response to determining that the detected object is not outside the object occurrence area; and
controlling driving of the vehicle in the switched manual drive mode when the drive mode is switched and controlling the driving of vehicle in the automatic drive mode when the drive mode is not switched.

### Claim 15

A non-transitory computer-readable storage medium storing a program, the program causing a computer to execute a method when the program is executed by the computer, the method for use by a vehicle configured to operate in an automatic drive mode, in which the vehicle is automatically driven, and a manual drive mode, in which the vehicle is driven manually by a driver, including:
detecting an object included in a captured image of a scene seen in a running direction of a vehicle,
determining whether the detected object is outside an object occurrence area of an object occurrence map representing a predefined area of the scene of predetermined dimensions
i) outside the vehicle path when viewed from the vehicle in a running direction of the vehicle,
ii) smaller than the entire area of the scene outside the vehicle path when viewed from the vehicle in the running direction of the vehicle, and
iii) based on data of past detections of the object in the area of the scene outside the vehicle path, where there is a higher probability that the object will exist in the scene than in other areas outside the vehicle and detectable by a sensor of the vehicle;
determining that the detecting of the object is unreliable when it is determined that the object is outside the object occurrence area;
switching a vehicle drive mode from the automatic drive mode to the manual drive mode in response to determining that the detecting of the object is unreliable as a result of the detected object being detected to be outside the object occurrence area;
maintaining the vehicle in the automatic drive mode in response to determining that the detected object is not outside the object occurrence area; and
controlling driving of the vehicle in the switched manual drive mode when the drive mode is switched and controlling the driving of vehicle in the automatic drive mode when the drive mode is not switched.

### Claim 16

An apparatus for use in a vehicle, configured to operate in an automatic drive mode, in which the vehicle is automatically driven, and a manual drive mode, in which the vehicle is driven manually by a driver, comprising:
a first memory; and
circuitry that, in operation, performs operations including
storing, in the first memory, an object occurrence map defining an object occurrence area of a captured image of a scene seen in a running direction of a vehicle, and representing a predefined area of predetermined dimensions of the scene
i) outside the vehicle path when viewed from the vehicle in a running direction of the vehicle,
ii) smaller than the entire area of the scene outside the vehicle path when viewed from the vehicle in the running direction of the vehicle, and
iii) in which where there is a higher probability that the object exists than in another area of the captured image of the scene;
detecting the object included in the captured image of the scene seen in the running direction of the vehicle;
determining whether the detected object is outside the occurrence area;
determining that the detecting of the object is unreliable when it is determined that the object is outside the object occurrence area;
switching a vehicle drive mode from the automatic drive mode to the manual drive mode in response to determining that detecting of the object is unreliable as a result of the detected object being detected to be outside the object occurrence area;
maintaining the vehicle in the automatic drive mode in response to determining that the detected object is not outside the object occurrence area; and
controlling driving of the vehicle in the switched manual drive mode when the drive mode is switched and controlling the driving of vehicle in the automatic drive mode when the drive mode is not switched.

### Claim 17

An apparatus for use by a vehicle configured to operate in an automatic drive mode, in which the vehicle is automatically driven, and a manual drive mode, in which the vehicle is driven manually by a driver, comprising:
a first memory; and
circuitry that, in operation, performs operations including
storing, in the first memory, an object occurrence map defining an object occurrence area of a captured image of a scene seen in a running direction of a vehicle, and representing a predefined area of predetermined dimensions of the scene
i) outside the vehicle path when viewed from the vehicle in a running direction of the vehicle,
ii) smaller than the entire area of the scene outside the vehicle path when viewed from the vehicle in the running direction of the vehicle, and
iii) where an object is to be expected to exist when the vehicle travels in the running direction through the scene;
detecting the object included in the captured image of the scene seen in the running direction of the vehicle;
determining the location of the detected object in relation to the stored occurrence area;
determining that the detecting of the object is unreliable when it is determined that the object is outside the object occurrence area;
switching a vehicle drive mode from the automatic drive mode to the manual drive mode in response to determining that the detecting of the object is unreliable as a result of the detected object being detected to be outside the object occurrence area;
maintaining the vehicle in the automatic drive mode in response to determining that the detected object is not outside the object occurrence area; and
controlling driving of the vehicle in the switched manual drive mode when the drive mode is switched and controlling the driving of vehicle in the automatic drive mode when the drive mode is not switched.