# US 2023/0053785 A1 · Vision-based machine learning model for aggregation of static objects and systems for autonomous driving > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0053785 A1 - Document ID: US20230053785A1 - Status: Published application - Kind: A1 - Application number: 17/820,849 - Filed: 2022-08-18 - Published: 2023-02-23 - Earliest claimed: 2021-08-19 - Assignees: Tesla, Inc. - Applicants: Tesla, Inc. - Inventors: Micael Carvalho; John Emmons; Patrick Cho; Bradley Emi; Saachi Jain; Nishant Desai; Tony Duan - Web page: https://patentgrep.com/patent/US20230053785A1 ## CPC - B60W 60/001: {Planning or execution of driving tasks} - G06T 7/13: Edge detection - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/776: Validation; Performance evaluation - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road} - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2552/53: Road markings, e.g. lane marker or crosswalk - G06T 2207/20081: Training; Learning - G06T 2207/30241: Trajectory - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Systems and methods for a vision-based machine learning model for aggregation of static objects and systems for autonomous driving. An example method includes obtaining images from image sensors positioned about a vehicle. Features associated with the images are determined, with the features being output based on a forward pass through a machine learning model. The features are projected into a vector space associated with a birds-eye view based on the machine learning model. The projected features are aggregated with other projected features associated with prior images. Images depicting static objects in the birds-eye view are output. ## Claims ### Claim 1 A method implemented by a vehicle processor system, the method comprising: obtaining images from a multitude of image sensors positioned about a vehicle; determining features associated with the images, wherein the features are output based on a forward pass through a machine learning model; projecting, based on the machine learning model, the features into a vector space associated with a birds-eye view; aggregating, based on a video module, the projected features with other projected features associated with prior images; and outputting, based on a plurality of heads of the machine learning model, images depicting static objects in the birds-eye view. ### Claim 2 (depends on 1) The method of claim 1, wherein the birds-eye view represents a top-down view in which static objects are positioned about a location of the vehicle. ### Claim 3 (depends on 1) The method of claim 1, wherein each image sensor is associated with a backbone network, and wherein the backbone network represents a portion of the machine learning model which determines a portion of the features which are associated with an individual image sensor. ### Claim 4 (depends on 3) The method of claim 3, wherein the features are projected into the vector space based on an attention network, and wherein the attention network receives an aggregated input of the features from the backbone networks. ### Claim 5 (depends on 1) The method of claim 1, wherein the video module obtains the projected features and other projected features from a feature queue, and wherein the feature queue spatially indexes information. ### Claim 6 (depends on 1) The method of claim 1, wherein the machine learning model is a hydra network in which a plurality of trunks receives respective portions of output from the video module, and wherein each head is associated with an individual trunk. ### Claim 7 (depends on 1) The method of claim 1, wherein a first image of the output images includes pixels which are assigned values based on likelihoods of the pixels depicting road edges. ### Claim 8 (depends on 1) The method of claim 1, wherein a second image of the output images includes pixels which are assigned respective colors, and wherein each color is indicative of a direction of travel. ### Claim 9 (depends on 1) The method of claim 1, wherein a third image of the output images depicts connections between a lane in which the vehicle is located with one or mother other lanes across an intersection. ### Claim 10 (depends on 9) The method of claim 9, wherein the machine learning model determines splines which connect the lane with the other lanes. ### Claim 11 A system comprising one or more processors and non-transitory computer storage media storing instructions that when executed by the one or more processors, cause the processors to perform operations, wherein the system is included in an autonomous or semi-autonomous vehicle, and wherein the operations comprise: obtaining images from a multitude of image sensors positioned about a vehicle; determining features associated with the images, wherein the features are output based on a forward pass through a machine learning model; projecting, based on the machine learning model, the features into a vector space associated with a birds-eye view; aggregating, based on a video module, the projected features with other projected features associated with prior images; and outputting, based on a plurality of heads of the machine learning model, images depicting static objects in the birds-eye view. ### Claim 12 (depends on 11) The system of claim 11, wherein the birds-eye view represents a top-down view in which static objects are positioned about a location of the vehicle. ### Claim 13 (depends on 11) The system of claim 11, wherein each image sensor is associated with a backbone network, and wherein the backbone network represents a portion of the machine learning model which determines a portion of the features which are associated with an individual image sensor. ### Claim 14 (depends on 13) The system of claim 13, wherein the features are projected into the vector space based on an attention network, and wherein the attention network receives an aggregated input of the features from the backbone networks. ### Claim 15 (depends on 11) The system of claim 11, wherein the video module obtains the projected features and other projected features from a feature queue, and wherein the feature queue spatially indexes information. ### Claim 16 (depends on 11) The system of claim 11, wherein a first image of the output images includes pixels which are assigned values based on likelihoods of the pixels depicting road edges. ### Claim 17 (depends on 11) The system of claim 11, wherein a second image of the output images includes pixels which are assigned respective colors, and wherein each color is indicative of a direction of travel. ### Claim 18 (depends on 11) The system of claim 11, wherein a third image of the output images depicts connections between a lane in which the vehicle is located with one or mother other lanes across an intersection. ### Claim 19 (depends on 18) The system of claim 18, wherein the machine learning model determines splines which connect the lane with the other lanes. ### Claim 20 Non-transitory computer storage media storing instructions that when executed by a system of one or more processors which are included in an autonomous or semi-autonomous vehicle, cause the system to perform operations comprising: obtaining images from a multitude of image sensors positioned about a vehicle; determining features associated with the images, wherein the features are output based on a forward pass through a machine learning model; projecting, based on the machine learning model, the features into a vector space associated with a birds-eye view; aggregating, based on a video module, the projected features with other projected features associated with prior images; and outputting, based on a plurality of heads of the machine learning model, images depicting static objects in the birds-eye view.