# US 12,112,546 B2 · Vision guidance system using dynamic edge detection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,546 B2 - Document ID: US12112546B2 - Status: Granted - Kind: B2 - Application number: 17/245,522 - Filed: 2021-04-30 - Published: 2022-11-03 - Granted: 2024-10-08 - Earliest claimed: 2021-04-30 - Assignees: Deere & Company - Applicants: Deere & Company - Inventors: Aaron G. Vesperman; Jeffery J. Adams; Anthony J. Mainelli - Web page: https://patentgrep.com/patent/US12112546B2 ## CPC - A01B 69/001: {Steering by means of optical assistance, e.g. television cameras (steering devices for road marking vehicles E01C23/163)} - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - A01B 69/008: {automatic} - B62D 15/025: {Active steering aids, e.g. helping the driver by actively influencing the steering system after environment evaluation (B62D1/28 takes precedence)} - G05D 1/0038: {by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation (images analyzed by a computer and used for automatic navigation G05D1/0246)} - G05D 1/2247: {providing the operator with simple or augmented images from one or more cameras} - G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529) - G06V 10/225: {based on a marking or identifier characterising the area} - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G06V 20/188: {Vegetation} - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - A01B: SOIL WORKING IN AGRICULTURE OR FORESTRY; PARTS, DETAILS, OR ACCESSORIES OF AGRICULTURAL MACHINES OR IMPLEMENTS, IN GENERAL (making or covering furrows or holes for sowing, planting or manuring A01C5/00; mowers convertible to soil working apparatus or capable of soil working A01D42/04; mowers combined with soil working implements A01D43/12; soil working for engineering purposes E01, E02, E21) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B62D: MOTOR VEHICLES; TRAILERS (steering, or guiding on a desired track, of agricultural machines or implements A01B69/00; wheels, castors, axles, increasing wheel adhesion B60B; vehicle tyres, tyre inflation or tyre changing B60C; connections between vehicles of a train or the like B60D; vehicles for use on rail and road, amphibious or convertible vehicles B60F; suspension arrangements B60G; heating, cooling, ventilating or other air treating devices B60H; windows, windscreens, non-fixed roofs, doors or similar devices, protective coverings for vehicles not in use B60J; propulsion plant arrangements, auxiliary drives, transmissions, controls, instrumentation or dashboards B60K; electric equipment or propulsion of electrically-propelled vehicles B60L; power supply for electrically-propelled vehicles B60M; passenger accommodation not otherwise provided for B60N; adaptations for load transportation or to carry special loads or objects B60P; arrangement of signalling or lighting devices, the mounting or supporting thereof or circuits therefor, for vehicles in general B60Q; vehicles, vehicle fittings or vehicle parts not otherwise provided for B60R; servicing, cleaning, repairing, supporting, lifting, or manoeuvring, not otherwise provided for, B60S; brake arrangements, brake control systems or parts thereof B60T; air-cushion vehicles B60V; motor-cycles, accessories therefor B62J, B62K; testing of vehicles G01M) - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A row vision system modifies automated operation of a vehicle based on edges detected between surfaces in the environment in which the vehicle travels. The vehicle may be a farming vehicle (e.g., a tractor) that operates using automated steering to perform farming operations that track an edge formed by a row of field work completed next to the unworked field area. A row vision system may access images of the field ahead of the tractor and apply models that identify surface types and detect edges between the identified surfaces (e.g., between worked and unworked ground). Using the detected edges, the system determines navigation instructions that modify the automated steering (e.g., direction) to minimize the error between current and desired headings of the vehicle, enabling the tractor to track the row of crops, edge of field, or edge of field work completed. ## Claims ### Claim 1 A method comprising: accessing a set of images captured by a vehicle while navigating via autonomous steering through an area of different surface types, the set of images comprising images of a ground surface in front the vehicle; receiving, from a remote operator, an input representative of a location within the set of images displayed to the remote operator; identifying a set of candidate edges within an image portion corresponding to the location within the set of images, each candidate edge corresponding to a candidate boundary between two different surface types; determining, for each of the set of candidate edges, a distance between the candidate edge and the location within the set of images represented by the input received from the remote operator; applying an edge selection model to the set of candidate edges, the edge selection model configured to select an edge of the set of candidate edges based at least in part on the determined distance for each candidate edge; and modifying a route being navigated by the vehicle based at least in part on the selected candidate edge. ### Claim 2 (depends on 1) The method of claim 1, wherein the image portion within the set of images comprises a bounding box centered on the location represented by the received input. ### Claim 3 (depends on 1) The method of claim 1, wherein the set of candidate edges are identified using an edge detection model corresponding to one or both of the two different surface types. ### Claim 4 (depends on 3) The method of claim 3, wherein the two different surface types comprise soil and one of a crop, grass, and pavement, and wherein the edge detection model comprises a machine-learned model that is trained on images of manually tagged boundaries between soil and the one of a crop, grass, and pavement. ### Claim 5 (depends on 1) The method of claim 1, wherein the edge selection model is configured to weight the candidate edges at least in part based on the distance between each candidate edge and the location represented by the received input within the set of images, wherein a first candidate edge is weighted lower than a second candidate edge that is closer to the location represented by the received input than the first candidate edge. ### Claim 6 (depends on 1) The method of claim 1, wherein the edge selection model comprises a machine-learned model that is trained on images each with a set of candidate edges and a manually-selected edge of the set of candidate edges. ### Claim 7 (depends on 1) The method of claim 1, further comprising: identifying, by the vehicle, each of the two different surface types; and selecting, by the vehicle, the edge selection model from a set of edge selection models based on the identified surface types. ### Claim 8 (depends on 7) The method of claim 7, wherein an identified surface comprises a type of crop, and wherein the selected edge selection model is trained based on images of the type of crop. ### Claim 9 (depends on 1) The method of claim 1, wherein the set of candidate edges are identified by and the edge selection model is applied by the vehicle. ### Claim 10 (depends on 1) The method of claim 1, wherein the set of candidate edges are identified by and the edge selection model is applied by a remote computing system communicatively coupled to the vehicle. ### Claim 11 (depends on 1) The method of claim 1, wherein modifying the route being navigated by the vehicle comprises moving the vehicle such that an edge of a tool or instrument being pulled by the vehicle is aligned with the selected edge. ### Claim 12 (depends on 1) The method of claim 1, wherein additional sets of candidate edges are iteratively identified as the vehicle captures additional images, and wherein edges are iteratively selected from among the sets of candidate edges. ### Claim 13 A system comprising a hardware processor and a non-transitory computer-readable storage medium storing executable instructions that, when executed by the processor, are configured to cause the system to perform steps comprising: accessing a set of images captured by a vehicle while navigating via autonomous steering through an area of different surface types, the set of images comprising images of a ground surface in front the vehicle; receiving, from a remote operator, an input representative of a location within the set of images displayed to the remote operator; identifying a set of candidate edges within an image portion corresponding to the location within the set of images, each candidate edge corresponding to a candidate boundary between two different surface types; determining, for each of the set of candidate edges, a distance between the candidate edge and the location within the set of images represented by the input received from the remote operator; applying an edge selection model to the set of candidate edges, the edge selection model configured to select an edge of the set of candidate edges based at least in part on the determined distance for each candidate edge; and modifying a route being navigated by the vehicle based at least in part on the selected candidate edge. ### Claim 14 (depends on 13) The system of claim 13, wherein the image portion within the set of images comprises a bounding box centered on the location represented by the received input. ### Claim 15 (depends on 13) The system of claim 13, wherein the set of candidate edges are identified using an edge selection model corresponding to one or both of the two different surface types, wherein the two different surface types comprise soil and one of a crop, grass, and pavement, and wherein the edge selection model comprises a machine-learned model that is trained on images of manually tagged boundaries between soil and the one of a crop, grass, and pavement. ### Claim 16 (depends on 13) The system of claim 13, wherein the edge selection model is configured to weight the candidate edges at least in part based on the distance between each candidate edge and the location represented by the received input within the set of images, wherein a first candidate edge is weighted lower than a second candidate edge that is closer to the location represented by the received input than the first candidate edge. ### Claim 17 (depends on 13) The system of claim 13, wherein the edge selection model comprises a machine-learned model that is trained on images each with a set of candidate edges and a manually-selected edge of the set of candidate edges, and wherein the edge selection model is selected among a set of edge selection models based on an identified surface type of the two different surface types. ### Claim 18 (depends on 13) The system of claim 13, wherein the hardware processor and the non-transitory computer-readable storage medium are implemented within one of the vehicle and a remote computing system coupled to the vehicle. ### Claim 19 (depends on 13) The system of claim 13, wherein modifying the route being navigated by the vehicle comprises moving the vehicle such that an edge of a tool or instrument being pulled by the vehicle is aligned with the selected edge. ### Claim 20 (depends on 13) The system of claim 13, wherein additional sets of candidate edges are iteratively identified as the vehicle captures additional images, and wherein edges are iteratively selected from among the sets of candidate edges. ### Claim 21 A method comprising: accessing a set of images captured by a vehicle while navigating via autonomous steering through an area of different surface types, the set of images comprising images of a ground surface in front the vehicle; identifying a set of candidate edges within the set of images, each candidate edge corresponding to a candidate boundary between two different surface types; displaying, to a remote operator via a device of the remote operator, the set of images overlayed with the identified set of candidate edges; receiving, from the remote operator, a selection of a candidate edge from the set of candidate edges displayed to the remote operator to be overlayed on the set of images; and modifying a route being navigated by the vehicle based at least in part on the selected candidate edge.