# US 11,299,166 B2 · Method and system for personalized driving lane planning in autonomous driving vehicles

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,299,166 B2
- Document ID: US11299166B2
- Status: Granted
- Kind: B2
- Application number: 15/856,728
- Filed: 2017-12-28
- Published: 2019-06-20
- Granted: 2022-04-12
- Earliest claimed: 2017-12-18
- Assignees: PlusAI, Inc.
- Applicants: PlusAI, Inc.
- Inventors: Hao Zheng; David Wanqian Liu; Timothy Patrick Daly, Jr.
- Web page: https://patentgrep.com/patent/US11299166B2

## CPC

- B60W 40/09: Driving style or behaviour
- B60W 30/18163: {Lane change; Overtaking manoeuvres}
- B60W 50/0097: {Predicting future conditions}
- B60W 60/00: Drive control systems specially adapted for autonomous road vehicles
- B60W 60/001: {Planning or execution of driving tasks}
- G01C 21/3484: {Personalized, e.g. from learned user behaviour or user-defined profiles}
- G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera}
- G01C 21/3658: {Lane guidance}
- G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00)
- G05D 1/0212: {with means for defining a desired trajectory (involving a plurality of land vehicles G05D1/0287)}
- G05D 1/0246: {using a video camera in combination with image processing means}
- G05D 1/249: from positioning sensors located off-board the vehicle, e.g. from cameras
- G05D 1/644: Optimisation of travel parameters, e.g. of energy consumption, journey time or distance
- G05D 1/646: Following a predefined trajectory, e.g. a line marked on the floor or a flight path
- G05D 1/81: Handing over between on-board automatic and on-board manual control
- B60W 2040/0872: {Driver physiology}
- B60W 2040/089: {Driver voice}
- B60W 2050/0028: {Mathematical models, e.g. for simulation}
- B60W 2050/0029: {Mathematical model of the driver}
- B60W 2540/043: Identity of occupants
- B60W 2540/30: Driving style
- B60W 2556/50: of positioning data, e.g. GPS [Global Positioning System] data
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES

## Abstract

The present teaching relates to method, system, medium, and implementation of lane planning in an autonomous vehicle. Sensor data are received that capture ground images of a road the autonomous vehicle is on. Based on the sensor data, a current lane of the road that autonomous vehicle is currently occupying is detected. Information indicating presence of a passenger in the vehicle is obtained and used to retrieve a personalized lane control model related to the passenger. Lane control for the autonomous vehicle is planned based on the detected current lane and self-aware capability parameters in accordance with the personalized lane control model. The self-aware capability parameters are used to predict operational capability of the autonomous vehicle with respect to a current location of the autonomous vehicle. The personalized lane control model is generated based on recorded human driving data.

## Claims

### Claim 1

A method, comprising:
receiving, via a processor, a first set of images generated from real time visual sensor data and depicting a ground surface on which an autonomous vehicle is positioned;
detecting, in real time and based on the first set of images, marks on the ground surface representing a lane that the autonomous vehicle is currently occupying based on at least one lane detection model obtained via machine learning;
receiving, via the processor, data representing a presence of a passenger within the autonomous vehicle;
selecting a personalized lane control model from a plurality of personalized lane control models related to the passenger, each personalized lane control model from the plurality of personalized lane control models being trained via machine learning based on recorded human driving data related to the passenger under different driving conditions;
generating a planned lane control for the autonomous vehicle based on the lane that the autonomous vehicle is currently occupying and self-aware capability parameters associated with the selected personalized lane control model, the generating the planned lane control including using the self-aware capability parameters to predict an operational capability of the autonomous vehicle with respect to a current location of the autonomous vehicle; and
sending a signal to cause implementation of the planned lane control at the autonomous vehicle.

### Claim 2 (depends on 1)

The method of claim 1, wherein the planned lane control includes at least one of lane following or lane changing.

### Claim 3 (depends on 1)

The method of claim 1, wherein the self-aware capability parameters include intrinsic capability parameters and extrinsic capability parameters, the intrinsic capability parameters specifying conditions internal to the autonomous vehicle that limit the operational capability of the autonomous vehicle, the extrinsic capability parameters specifying conditions external to the autonomous vehicle that impact the operational capability of the autonomous vehicle.

### Claim 4 (depends on 1)

The method of claim 1, wherein the self-aware capability parameters are dynamically updated to reflect a scenario the autonomous vehicle is currently associated with.

### Claim 5 (depends on 1)

The method of claim 1, wherein the lane detection model is selected from a plurality of lane detection models based on the self-aware capability parameters.

### Claim 6 (depends on 1)

The method of claim 1, wherein the data representing the presence of the passenger within the autonomous vehicle is obtained by:
receiving a second set of images generated from the real time visual sensor data and depicting an internal scene of the autonomous vehicle;
analyzing the second set of images to detect the presence of the passenger; and
generating the data representing the presence of the passenger within the autonomous vehicle.

### Claim 7

A non-transitory, processor-readable medium storing instructions to cause a processor to:
receive a first set of images generated from real time visual sensor data and depicting a ground surface on which an autonomous vehicle is positioned;
detect, in real time and based on the first set of images, marks on the ground surface representing a lane that the autonomous vehicle is currently occupying based on at least one lane detection model obtained via machine learning;
receive data representing a presence of a passenger within the autonomous vehicle;
select a personalized lane control model from a plurality of personalized lane control models related to the passenger, wherein each personalized lane control model from the plurality of personalized lane control models is trained via machine learning based on recorded human driving data related to the passenger under different driving conditions;
generate a planned lane control for the autonomous vehicle based on the lane that the autonomous vehicle is currently occupying and self-aware capability parameters associated with the selected personalized lane control model, the generating the planned lane control including using the self-aware capability parameters to predict an operational capability of the autonomous vehicle with respect to a current location of the autonomous vehicle; and
send a signal to cause implementation of the planned lane control at the autonomous vehicle.

### Claim 8 (depends on 7)

The medium of claim 7, wherein the planned lane control includes at least one of lane following or lane changing.

### Claim 9 (depends on 7)

The medium of claim 7, wherein the self-aware capability parameters include intrinsic capability parameters and extrinsic capability parameters, the intrinsic capability parameters specifying conditions internal to the autonomous vehicle that limit the operational capability of the autonomous vehicle, the extrinsic capability parameters specifying conditions external to the autonomous vehicle that impact the operational capability of the autonomous vehicle.

### Claim 10 (depends on 7)

The medium of claim 7, wherein the self-aware capability parameters are dynamically updated to reflect a scenario the autonomous vehicle is currently associated with.

### Claim 11 (depends on 7)

The medium of claim 7, wherein the lane detection model is selected from a plurality of lane detection models based on the self-aware capability parameters.

### Claim 12 (depends on 7)

The medium of claim 7, wherein the data representing the presence of the passenger within the autonomous vehicle is obtained by:
receiving a second set of images generated from the real time visual sensor data and depicting an internal scene of the autonomous vehicle;
analyzing the second set of images to detect the presence of the passenger; and
generating the data representing the presence of the passenger within the autonomous vehicle.

### Claim 13

A system, comprising:
a driving lane detector implemented by a processor and configured to:
receive a first set of images generated from real time visual sensor data and depicting a ground surface on which an autonomous vehicle is positioned,
detect, in real time and based on the first set of images, marks on the ground surface representing a lane that the autonomous vehicle is currently occupying based on at least one lane detection model obtained via machine learning; and
a driving lane planning unit implemented by the processor and configured to
receive data representing a presence of a passenger within the autonomous vehicle,
select a personalized lane control model from a plurality of personalized lane control models related to the passenger, wherein each personalized lane control model from the plurality of personalized lane control models is trained via machine learning based on recorded human driving data related to the passenger under different driving conditions,
generate a planned lane control for the autonomous vehicle based on the lane that the autonomous vehicle is currently occupying and self-aware capability parameters associated with the selected personalized lane control model, the generating the planned lane control including using the self-aware capability parameters to predict an operational capability of the autonomous vehicle with respect to a current location of the autonomous vehicle; and
send a signal to cause implementation of the planned lane control at the autonomous vehicle.

### Claim 14 (depends on 13)

The system of claim 13, wherein the planned lane includes at least one of lane following or lane changing.

### Claim 15 (depends on 13)

The system of claim 13, wherein the self-aware capability parameters include intrinsic capability parameters and extrinsic capability parameters, the intrinsic capability parameters specifying conditions internal to the autonomous vehicle that limit the operational capability of the autonomous vehicle, the extrinsic capability parameters specifying conditions external to the autonomous vehicle that impact the operational capability of the autonomous vehicle.

### Claim 16 (depends on 13)

The system of claim 13, wherein the self-aware capability parameters are dynamically updated to reflect a scenario the autonomous vehicle is currently associated with.

### Claim 17 (depends on 13)

The system of claim 13, wherein the lane detection model is selected from a plurality of lane detection models based on the self-aware capability parameters.

### Claim 18 (depends on 13)

The system of claim 13, wherein the data representing the presence of the passenger within the autonomous vehicle is obtained by:
receiving a second set of images generated from the real time visual sensor data and depicting an internal scene of the autonomous vehicle;
analyzing the second set of images to detect the presence of the passenger; and
generating the data representing the presence of the passenger within the autonomous vehicle.