# US 11,906,974 B2 · Off-road machine-learned obstacle navigation in an autonomous vehicle environment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,906,974 B2 - Document ID: US11906974B2 - Status: Granted - Kind: B2 - Application number: 16/953,774 - Filed: 2020-11-20 - Published: 2022-05-26 - Granted: 2024-02-20 - Earliest claimed: 2020-11-20 - Assignees: Deere & Company - Applicants: Deere & Company - Inventors: Travis J. Davis - Web page: https://patentgrep.com/patent/US11906974B2 ## CPC - G05D 1/0238: {using obstacle or wall sensors (G05D1/0246 and G05D1/0289 take precedence; lidar systems designed for anti-collision purposes G01S17/93)} - G05D 1/0248: {in combination with a laser (lasers per se H01S)} - G05D 1/0223: {involving speed control of the vehicle (vehicle fittings for automatically controlling, i.e. preventing speed from exceeding an arbitrarily established velocity or maintaining speed at a particular velocity, as selected by the vehicle operator B60K31/00)} - G05D 1/0246: {using a video camera in combination with image processing means} - G05D 1/249: from positioning sensors located off-board the vehicle, e.g. from cameras - G05D 1/628: following the obstacle profile, e.g. a wall or undulated terrain - G05D 1/65: Following a desired speed profile - G06F 18/24: Classification techniques - G06N 3/02: Neural networks - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 20/64: Three-dimensional [3D] objects - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract An autonomous off-road vehicle, upon encountering an obstruction while navigating a route, can apply a first machine-learned model to identify the obstruction. In the event that the first machine-learned model cannot identify the obstruction, the autonomous off-road vehicle can apply a second machine-learned model configured to determine whether or not the obstruction can be ignored, for instance based on dimensions of the obstruction. If the obstruction can be ignored, the autonomous off-road vehicle can continue navigating the route. If the obstruction cannot be ignored, the autonomous off-road vehicle can modify the route, can stop, can flag the obstruction to a remote human operator, can modify an interface of a human operator to display a notification or a video feed from the vehicle, and the like. ## Claims ### Claim 1 A method for autonomously navigating an off-road vehicle, comprising: navigating, by an autonomous off-road vehicle, a route from a first location to a second location within an off-road environment; while navigating the route, identifying, by the autonomous off-road vehicle, an obstruction obstructing the route; applying, by the autonomous off-road vehicle, a first machine-learned model to sensor data representative of the obstruction, the first machine-learned model configured to attempt to identify a type of the obstruction and, based on the type of obstruction, determine whether the obstruction can be ignored; in response to determining that the obstruction can be ignored, continuing, by the autonomous off-road vehicle, navigating along the route; in response to determining that the obstruction cannot be ignored, modifying, by the autonomous off-road vehicle, the route to avoid the obstruction; and in response to the first machine-learned model being unable to determine whether or not the identified obstruction obstructing the route can be ignored based on being unable to identify the type of the obstruction, applying, by the autonomous off-road vehicle, a dimension-based model to depth data representative of the obstruction, the dimension-based model configured to determine whether the obstruction can be ignored based on dimensions of the obstruction, wherein the dimension-based model is trained based on a set of training data representative of interactions between vehicles and obstructions, wherein the set of training data includes information describing a change in a condition of one or both of a vehicle and an obstruction with which the vehicle interacted. ### Claim 2 (depends on 1) The method of claim 1, wherein the sensor data representative of the obstruction comprises image data captured by a camera of the autonomous off-road vehicle. ### Claim 3 (depends on 1) The method of claim 1, wherein the depth data is captured by a depth-sensing system of the autonomous off-road vehicle. ### Claim 4 (depends on 1) The method of claim 1, wherein the first machine-learned model comprises an image classifier configured to attempt to identify the type of the obstruction based on one or more images of the obstruction. ### Claim 5 (depends on 4) The method of claim 4, wherein the first machine-learned model further comprises an obstruction classifier configured to classify whether or not the obstruction can be ignored based on the identified type of the obstruction. ### Claim 6 (depends on 1) The method of claim 1, wherein the first machine-learned model comprises a neural network configured to classify the type of the obstruction, and if the neural network is able to classify the type of the obstruction, classify whether or not the obstruction can be ignored based on the identified type of the obstruction. ### Claim 7 (depends on 1) The method of claim 1, wherein the set of training data includes dimensions of the obstructions. ### Claim 8 (depends on 1) The method of claim 1, wherein the dimension-based model comprises a neural network. ### Claim 9 (depends on 1) The method of claim 1, wherein the dimension-based model, in response to determining that the obstruction cannot be ignored, further identifies an action that can be performed by the autonomous off-road vehicle. ### Claim 10 (depends on 9) The method of claim 9, wherein the action that can be performed by the autonomous off-road vehicle is based on one or more of: a distance between the autonomous off-road vehicle and the obstruction, a speed at which the autonomous off-road vehicle is traveling, a size of the obstruction, a potential harm to the autonomous off-road vehicle or the obstruction, and a cost to modify the route being navigated by the autonomous off-road vehicle. ### Claim 11 (depends on 9) The method of claim 9, wherein the action that can be performed by the autonomous off-road vehicle comprises one or more of: reducing a speed of the autonomous off-road vehicle, stopping the autonomous off-road vehicle, and modifying the route the autonomous off-road vehicle is navigating. ### Claim 12 (depends on 1) The method of claim 1, wherein the off-road environment comprises a construction site, and wherein the autonomous off-road vehicle comprises an autonomous construction vehicle. ### Claim 13 (depends on 1) The method of claim 1, wherein the off-road environment comprises an agriculture field, and wherein the autonomous off-road vehicle comprises an autonomous agriculture vehicle. ### Claim 14 (depends on 1) The method of claim 1, wherein the obstruction is a static obstruction. ### Claim 15 An autonomous off-road vehicle comprising a hardware processor and a non-transitory computer-readable storage medium storing executable instructions that, when executed by the hardware processor, cause the autonomous off-road vehicle to perform steps comprising: navigating a route from a first location to a second location within an off-road environment; while navigating the route, identifying an obstruction obstructing the route; applying a first machine-learned model to sensor data representative of the obstruction, the first machine-learned model configured to attempt to identify a type of the obstruction and, based on the type of obstruction, determine whether the obstruction can be ignored; in response to determining that the obstruction can be ignored, continuing navigating along the route; in response to determining that the obstruction cannot be ignored, modifying the route to avoid the obstruction; and in response to the first machine-learned model being unable to determine whether or not the identified obstruction obstructing the route can be ignored based on being unable to identify the type of the obstruction, applying a dimension-based model to depth data representative of the obstruction, the dimension-based model configured to determine whether the obstruction can be ignored based on dimensions of the obstruction, wherein the dimension-based model is trained based on a set of training data representative of interactions between vehicles and obstructions, wherein the set of training data includes information describing a change in a condition of one or both of a vehicle and an obstruction with which the vehicle interacted. ### Claim 16 (depends on 15) The autonomous off-road vehicle of claim 15, wherein the obstruction is a static obstruction. ### Claim 17 (depends on 15) The autonomous off-road vehicle of claim 15, wherein the sensor data representative of the obstruction comprises image data captured by a camera of the autonomous off-road vehicle, and wherein the depth data is captured by a depth-sensing system of the autonomous off-road vehicle. ### Claim 18 (depends on 15) The autonomous off-road vehicle of claim 15, wherein the first machine-learned model comprises: an image classifier configured to attempt to identify the type of the obstruction based on one or more images of the obstruction; and an obstruction classifier configured to classify whether or not the obstruction can be ignored based on the identified type of the obstruction. ### Claim 19 (depends on 15) The autonomous off-road vehicle of claim 15, wherein the first machine-learned model comprises a neural network configured to classify the type of the obstruction, and if the neural network is able to classify the type of the obstruction, classify whether or not the obstruction can be ignored based on the identified type of the obstruction.