# US 2024/0005641 A1 · Data Augmentation for Detour Path Configuring

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005641 A1
- Document ID: US20240005641A1
- Status: Published application
- Kind: A1
- Application number: 18/083,187
- Filed: 2022-12-16
- Published: 2024-01-04
- Earliest claimed: 2022-06-30
- Assignees: PlusAl, Inc.
- Applicants: PlusAl, Inc.
- Inventors: Abhishek BHATIA; Inderjot Singh Saggu; Ankur Agarwal
- Web page: https://patentgrep.com/patent/US20240005641A1

## CPC

- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- B60W 60/0015: {specially adapted for safety}
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/096: Transfer learning
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 3/60: Rotation of whole images or parts thereof
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06V 10/82: using neural networks
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2552/50: Barriers
- B60W 2552/53: Road markings, e.g. lane marker or crosswalk
- B60W 2554/4026: Cycles
- B60W 2554/4029: Pedestrians
- B60W 2555/60: Traffic rules, e.g. speed limits or right of way
- B60W 2556/45: External transmission of data to or from the vehicle
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/048: Activation functions
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06T 2207/20081: Training; Learning
- G06T 2207/30256: Lane; Road marking
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

This application is directed to augmenting training images used for generating vehicle driving models. A computer system obtains a first image of a road, identifies within the first image a drivable area of the road, obtains an image of a traffic safety object, and determines a detour path on the drivable area. The computer system determines positions of a plurality of traffic safety objects to be placed adjacent to the detour path, and generates a second image from the first image by adaptively overlaying a respective copy of the image of the traffic safety object at each of the positions of the plurality of traffic safety objects on the drivable area within the first image. The second image is added to a corpus of training images to be used by a machine learning system to generate a model for facilitating driving a vehicle (e.g., at least partial autonomously).

## Claims

### Claim 1

A method for augmenting training images used for generating autonomous vehicle driving models, comprising:
at a computer system including one or more processors and memory:
obtaining a first image of a road that depicts a drivable area;
identifying within the first image the drivable area of the road;
obtaining an image of a traffic safety object;
generating a second image from the first image by adaptively overlaying a plurality of copies of the image of the traffic safety object on the drivable area of the road in the first image; and
adding the second image to a corpus of training images to be used to generate a model for facilitating at least partial autonomous driving of a vehicle.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
training, using machine learning, the model using the corpus of training images, including the second image; and
distributing the model to one or more vehicles, including a first vehicle;
wherein, in use, the model is configured to process road images captured by the first vehicle to facilitate at least partially autonomously driving the first vehicle.

### Claim 3 (depends on 1)

The method of claim 1, wherein adaptively overlaying a plurality of copies of the image of the traffic safety object on the drivable area of the road in the first image comprises scaling a respective size of each copy of the image of the traffic safety object based on a respective position where the respective traffic safety object is to be placed.

### Claim 4 (depends on 1)

The method of claim 1, wherein adaptively overlaying a plurality of copies of the image of the traffic safety object on the drivable area of the road in the first image comprises adjusting an orientation of each copy of the image of the traffic safety object based on a direction normal to the drivable area at a corresponding position.

### Claim 5 (depends on 1)

The method of claim 1, wherein adaptively overlaying a plurality of copies of the image of the traffic safety object on the drivable area of the road in the first image comprises adjusting one or more image properties of each copy of the image of the traffic safety object.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
obtaining information for a plurality of road features, including one or more of: a vehicle, a lane area, a shoulder area, an edge marking, a lane marking, a shoulder barrier structure, a road divider, a traffic light, a traffic sign, a road sign, a pedestrian, and a bicycle, wherein the plurality of copies of the image of the traffic safety object are overlaid on the drivable area of the road based on the information for the plurality of road features.

### Claim 7 (depends on 1)

The method of claim 1, further comprising determining a detour path on the drivable area, wherein the detour path is defined by two substantially parallel detour lane lines, and positions of one or more traffic safety objects are substantially on or near the detour lane lines.

### Claim 8 (depends on 1)

The method of claim 1, further comprising determining a detour path on the drivable area, including:
determining a first detour lane line that is adjacent to the detour path and configured to define the detour path on the drivable area; and
identifying positions for one or more traffic safety objects on or near the first detour lane line.

### Claim 9 (depends on 8)

The method of claim 8, wherein determining the first detour lane line comprises:
determining one or more of: a total number of detour paths, a length of the first detour lane line, a number of objects on the first detour lane line, object spacings between each two immediately adjacent traffic safety objects, curvatures of the first detour lane line at the one or more traffic safety objects, and randomly generated deviations from the first detour lane line.

### Claim 10 (depends on 8)

The method of claim 8, wherein determining the detour path on the drivable area comprises:
determining a second detour lane line parallel to the first detour lane line, the second detour lane line configured to define the detour path on the road jointly with the first detour lane line; and
identifying positions for a second plurality of traffic safety objects on or near the second detour lane line.

### Claim 11 (depends on 8)

The method of claim 8, wherein the second image is associated with a ground truth, and the ground truth includes the first detour lane line, the method comprising, iteratively:
recognizing a lane line of the detour path using the model;
comparing the recognized lane line to the first detour lane line of the ground truth; and
adjusting the model to match the recognized detour lane line to the first detour lane line of the ground truth.

### Claim 12

A computer system, comprising:
one or more processors; and
memory storing one or more programs configured for execution by the one or more processors, the one or more programs comprising instructions for:
obtaining a first image of a road that depicts a drivable area;
identifying within the first image the drivable area of the road;
obtaining an image of a traffic safety object;
generating a second image from the first image by adaptively overlaying a plurality of copies of the image of the traffic safety object on the drivable area of the road in the first image; and
adding the second image to a corpus of training images to be used to generate a model for facilitating at least partial autonomous driving of a vehicle.

### Claim 13 (depends on 12)

The computer system of claim 12, the one or more programs further comprising instructions for determining positions for a plurality of traffic safety objects to be placed adjacent to a detour path, including, for each of the plurality of traffic safety objects:
identifying one or more reference nodes in a respective copy of the image of the traffic safety object;
determining a respective location of each of the one or more reference nodes in a first coordinate system corresponding to a first space; and
converting the respective location of each of the one or more reference nodes in the first coordinate system to a respective location near a position of the respective traffic safety object in an image coordinate system.

### Claim 14 (depends on 13)

The computer system of claim 13, wherein determining the positions further comprises, for each of the plurality of traffic safety objects:
obtaining a camera intrinsic parameter K and a conversion vector between the first coordinate system and a camera coordinate system;
wherein based on the camera intrinsic parameter K and the conversion vector, the respective location of each of the one or more reference nodes in the first coordinate system is converted to the respective location near the position of the respective traffic safety object in the image coordinate system.

### Claim 15 (depends on 12)

The computer system of claim 12, the one or more programs further comprising instructions for determining a detour path on the drivable area, wherein the detour path is adjacent to a road feature, and the road feature is one of a vehicle, a person, a bike, a motorcycle, a traffic sign, and a road sign.

### Claim 16 (depends on 15)

The computer system of claim 15, the one or more programs further comprising instructions for:
determining that the road feature is partially occluded by a first traffic safety object based on depth values of the road feature and the first traffic safety object, wherein the first traffic safety object is overlaid partially on the road feature.

### Claim 17

A non-transitory computer-readable storage medium storing one or more programs configured for execution by one or more processors of a computer system, the one or more programs comprising instructions for:
obtaining a first image of a road that depicts a drivable area;
identifying within the first image the drivable area of the road;
obtaining an image of a traffic safety object;
generating a second image from the first image by adaptively overlaying a plurality of copies of the image of the traffic safety object on the drivable area of the road in the first image; and
adding the second image to a corpus of training images to be used to generate a model for facilitating at least partial autonomous driving of a vehicle.

### Claim 18 (depends on 17)

The non-transitory computer-readable storage medium of claim 17, wherein the traffic safety object includes a cone structure.

### Claim 19 (depends on 17)

The non-transitory computer-readable storage medium of claim 17, the one or more programs further comprising instructions for determining positions for a plurality of traffic safety objects to be placed adjacent to a detour path, including:
determining alternative positions of the plurality of traffic safety objects in a first coordinate system; and
converting the alternative positions of the plurality of traffic safety objects in the first coordinate system to the positions of the plurality of traffic safety objects in a camera coordinate system.

### Claim 20 (depends on 19)

The non-transitory computer-readable storage medium of claim 19, wherein determining the positions further comprises, for each of the plurality of traffic safety objects:
obtaining a camera intrinsic parameter K and a conversion vector between the first coordinate system and the camera coordinate system;
wherein based on the camera intrinsic parameter K and the conversion vector, the respective location of each of the one or more reference nodes in the first coordinate system is converted to a respective location near a position of the respective traffic safety object in an image coordinate system.
