# US 12,223,677 B1 · Map-anchored object detection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,677 B1 - Document ID: US12223677B1 - Status: Granted - Kind: B1 - Application number: 18/354,415 - Filed: 2023-07-18 - Published: 2025-01-23 - Granted: 2025-02-11 - Earliest claimed: 2023-07-18 - Assignees: Aurora Operations, Inc. - Applicants: Aurora Operations, Inc. - Inventors: Louis Foucard - Web page: https://patentgrep.com/patent/US12223677B1 ## CPC - G06T 7/74: {involving reference images or patches} - B60W 60/001: {Planning or execution of driving tasks} - G06V 10/811: {the classifiers operating on different input data, e.g. multi-modal recognition} - G06V 10/82: using neural networks - G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road} - B60W 2420/00: Indexing codes relating to the type of sensors based on the principle of their operation - B60W 2554/00: Input parameters relating to objects - B60W 2556/00: Input parameters relating to data - G06T 2207/20081: Training; Learning - G06T 2207/30204: Marker - G06T 2207/30256: Lane; Road marking - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract An example method includes (a) obtaining sensor data descriptive of an environment of an autonomous vehicle; (b) obtaining a plurality of travel way markers from map data descriptive of the environment; (c) determining, using a machine-learned object detection model and based on the sensor data, an association between one or more travel way markers of the plurality of travel way markers and an object in the environment; and (d) generating, using the machine-learned object detection model, an offset with respect to the one or more travel way markers of a spatial region of the environment associated with the object. ## Claims ### Claim 1 A computer-implemented method, comprising: (a) obtaining sensor data descriptive of an environment of an autonomous vehicle; (b) obtaining a plurality of travel way markers from map data descriptive of the environment; (c) determining, using a machine-learned object detection model and based on the sensor data, an association between one or more travel way markers of the plurality of travel way markers and an object in the environment; and (d) generating, using the machine-learned object detection model, an offset with respect to the one or more travel way markers of a spatial region of the environment associated with the object; wherein (c) comprises: inputting the travel way markers and the sensor data to the machine-learned object detection model; and obtaining object data from the machine-learned object detection model at projected locations of the travel way markers in a reference frame of the sensor data, wherein the object data indicates that the object is likely to be present at a projected location of the one or more travel way markers. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the travel way markers include lane markers. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the lane markers comprise centerline markers. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein obtaining the object data comprises subsampling, based on the travel way markers, a detection map generated by the machine-learned object detection model. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein one or more portions of the machine-learned object detection model are configured to sparsely use portions of an output layer based on locations in the sensor data corresponding to the projected locations. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the machine-learned object detection model is trained by: obtaining ground truth travel way marker labels indicating a ground truth association between the object and one or more of the travel way markers; and determining, based on comparing the object data and the ground truth travel way marker labels, a sparse loss for the machine-learned object detection model. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein (d) comprises: determining an offset of a centroid of a boundary of the spatial region; and determining one or more dimensions of the boundary. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein (d) comprises: determining a first offset of a centroid of a first boundary of the spatial region in two dimensions; and determining a second offset of a centroid of a second boundary of the spatial region in three dimensions. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, comprising: based on determining that a velocity of the object is below a threshold, outputting a characteristic for the object indicating that the object is a static object; and outputting the characteristic to a motion planning system of the autonomous vehicle. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, comprising: based on determining that a velocity of the object is below a threshold and that the object is located adjacent to a travel way in the environment, outputting a characteristic for the object indicating that the object is a static object; and outputting the characteristic to a motion planning system of the autonomous vehicle. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein (b) comprises: sampling discrete travel way markers from continuous travel way map data. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein the spatial region of the environment is beyond an effective range of a LIDAR sensor of the autonomous vehicle. ### Claim 13 (depends on 1) The computer-implemented method of claim 1, wherein: the machine-learned object detection model was trained using training sensor data having a training field of view and training travel way markers having a training resolution; the sensor data is associated with a runtime field of view; and the travel way markers are obtained in (c) at a runtime resolution selected based on a comparison of the training field of view and the runtime field of view. ### Claim 14 (depends on 1) The computer-implemented method of claim 1, comprising: projecting, using a projection transform, the travel way markers into a reference frame of the sensor data; determining one or more offsets of the spatial region with respect to the travel way markers; based on the determined one or more offsets, determining a projection error for the projected travel way markers; and recalibrating the projection transform based on the determined projection error. ### Claim 15 (depends on 1) The computer-implemented method of claim 1, comprising: identifying a lane in which the object is located based on the one or more travel way markers. ### Claim 16 (depends on 1) The computer-implemented method of claim 1, wherein (d) comprises: regressing, by the machine-learned object detection model, the offset. ### Claim 17 An autonomous vehicle control system for controlling an autonomous vehicle, the autonomous vehicle control system comprising: one or more processors; and one or more non-transitory computer-readable media storing instructions that are executable by the one or more processors to cause the autonomous vehicle control system to perform operations, the operations comprising: (a) obtaining sensor data descriptive of an environment of the autonomous vehicle; (b) obtaining a plurality of travel way markers from map data descriptive of the environment; (c) determining, using a machine-learned object detection model and based on the sensor data, an association between one or more travel way markers of the plurality of travel way markers and an object in the environment; and (d) generating, using the machine-learned object detection model, an offset with respect to the one or more travel way markers of a spatial region of the environment associated with the object; wherein (c) comprises: inputting the travel way markers and the sensor data to the machine-learned object detection model; and obtaining object data from the machine-learned object detection model at projected locations of the travel way markers in a reference frame of the sensor data, wherein the object data indicates that the object is likely to be present at a projected location of the one or more travel way markers. ### Claim 18 (depends on 17) The autonomous vehicle control system of claim 17, wherein obtaining the object data comprises subsampling, based on the travel way markers, a detection map generated by the machine-learned object detection model. ### Claim 19 (depends on 17) The autonomous vehicle control system of claim 17, wherein (d) comprises: determining a first offset of a centroid of a first boundary of the spatial region in two dimensions; and determining a second offset of a centroid of a second boundary of the spatial region in three dimensions. ### Claim 20 (depends on 17) The autonomous vehicle control system of claim 17, wherein the machine-learned object detection model is trained by: obtaining ground truth travel way marker labels indicating a ground truth association between the object and one or more of the travel way markers; and determining, based on comparing the object data and the ground truth travel way marker labels, a sparse loss for the machine-learned object detection model. ### Claim 21 One or more non-transitory computer-readable media storing instructions that are executable by one or more processors to cause an autonomous vehicle control system to perform operations, the operations comprising: (a) obtaining sensor data descriptive of an environment of an autonomous vehicle; (b) obtaining a plurality of travel way markers from map data descriptive of the environment; (c) determining, using a machine-learned object detection model and based on the sensor data, an association between one or more travel way markers of the plurality travel way markers and an object in the environment; and (d) generating, using the machine-learned object detection model, an offset with respect to the one or more travel way markers of a spatial region of the environment associated with the object; wherein (c) comprises: inputting the travel way markers and the sensor data to the machine-learned object detection model; and obtaining object data from the machine-learned object detection model at projected locations of the travel way markers in a reference frame of the sensor data, wherein the object data indicates that the object is likely to be present at a projected location of the one or more travel way markers.