# US 12,214,808 B2 · Systems and methods for detecting actors with respect to an autonomous vehicle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,214,808 B2 - Document ID: US12214808B2 - Status: Granted - Kind: B2 - Application number: 18/178,641 - Filed: 2023-03-06 - Published: 2023-08-03 - Granted: 2025-02-04 - Earliest claimed: 2020-02-19 - Assignees: Aurora Operations, Inc. - Applicants: Aurora Operations, Inc. - Inventors: Abhishek Sen; Ashton James Fagg; Brian C. Becker; Yang Xu; Nathan Nicolas Pilbrough; Carlos Vallespi-Gonzalez - Web page: https://patentgrep.com/patent/US12214808B2 ## CPC - B60W 60/0027: {using trajectory prediction for other traffic participants} - G05D 1/0212: {with means for defining a desired trajectory (involving a plurality of land vehicles G05D1/0287)} - G01S 7/4808: {Evaluating distance, position or velocity data} - G01S 17/58: Velocity or trajectory determination systems; Sense-of-movement determination systems - G01S 17/66: Tracking systems using electromagnetic waves other than radio waves - G01S 17/931: of land vehicles - G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)} - G05D 1/0214: {in accordance with safety or protection criteria, e.g. avoiding hazardous areas (monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13)} - G05D 1/617: Safety or protection, e.g. defining protection zones around obstacles or avoiding hazards (arrangements for controlling the position or course of two or more vehicles for avoiding collisions therebetween G05D1/693; arrangements for reacting to or preventing system or operator failure G05D1/80) - G05D 1/81: Handing over between on-board automatic and on-board manual control - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - B60W 2420/408: Radar; Laser, e.g. lidar - B60W 2554/4026: Cycles - B60W 2554/4029: Pedestrians - B60W 2554/4042: Longitudinal speed - B60W 2554/4043: Lateral speed - B60W 2554/4044: Direction of movement, e.g. backwards - G01S 17/89: for mapping or imaging - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract An autonomous vehicle computing system can include a primary perception system configured to receive a plurality of sensor data points as input generate primary perception data representing a plurality of classifiable objects and a plurality of paths representing tracked motion of the plurality of classifiable objects. The autonomous vehicle computing system can include a secondary perception system configured to receive the plurality of sensor data points as input, cluster a subset of the plurality of sensor data points of the sensor data to generate one or more sensor data point clusters representing one or more unclassifiable objects that are not classifiable by the primary perception system, and generate secondary path data representing tracked motion of the one or more unclassifiable objects. The autonomous vehicle computing system can generate fused perception data based on the primary perception data and the one or more unclassifiable objects. ## Claims ### Claim 1 A computer-implemented method, comprising: obtaining sensor data comprising a plurality of sensor data points descriptive of an environment of an autonomous vehicle; generating, based on the sensor data, primary perception data by a primary perception system, the primary perception data representing a plurality of classifiable objects and a plurality of paths representing tracked motion of the plurality of classifiable objects, wherein each classifiable object is classified by the primary perception system as a predefined class of a plurality of predefined classes of objects; generating, by a secondary perception system that is different from the primary perception system, one or more sensor data point clusters and secondary path data based on the plurality of sensor data points, the secondary path data representing tracked motion of the one or more sensor data point clusters; and determining fused perception data representing the tracked motion of the plurality of classifiable objects and the tracked motion of one or more unclassifiable objects, wherein the one or more unclassifiable objects are identified by deduplicating the plurality of classifiable objects with the one or more sensor data point clusters. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the plurality of predefined classes comprises at least one of: (i) a vehicle class, (ii) a pedestrian class, or (iii) a bicycle class. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the method further comprises: predicting one or more predicted paths representing predicted motion of at least one of the plurality of classifiable objects or the plurality of unclassified objects based on the fused perception data; determining a motion plan for navigation of the autonomous vehicle based on the fused perception data and the one or more predicted paths; and implementing the motion plan to navigate the autonomous vehicle. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein generating the primary perception data comprises: identifying, by the primary perception system, the plurality of classifiable objects based on the sensor data; determining, by the primary perception system, the plurality of paths representing the tracked motion of the plurality of classifiable objects; and classifying, by the primary perception system, each of the plurality of classified objects with respect to the plurality of predefined classes. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein generating the secondary path data comprises: determining a clustered sensor data point path for a respective sensor data point cluster by aggregating respective sensor data point paths of each sensor data point within the respective sensor data point cluster. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein generating the secondary path data further comprises: validating the secondary path data based on the clustered sensor data point path. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein validating the secondary path data comprises: determining a velocity of the cluster sensor data point path; and comparing the velocity of the cluster sensor data point path to a velocity threshold. ### Claim 8 (depends on 7) The computer-implemented method of claim 7, further comprising: determining that the velocity of the clustered sensor data point path has a velocity magnitude greater than the velocity threshold; and in response to determining that the velocity magnitude is greater than the velocity threshold, generating, by the secondary perception system, the secondary path data. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein generating the one or more sensor data point clusters comprises: inputting, by the secondary perception system, the plurality of sensor data points into a machine-learned clustering model; and in response to inputting the plurality of sensor data points into the machine-learned clustering model, obtaining the one or more sensor data point clusters. ### Claim 10 (depends on 9) The computer-implemented method of claim 9, wherein the machine-learned clustering model comprises an unsupervised clustering model. ### Claim 11 (depends on 9) The computer-implemented method of claim 9, wherein the machine-learned clustering model comprises a connected component clustering model. ### Claim 12 (depends on 9) The computer-implemented method of claim 9, wherein the machine-learned clustering model comprises at least one of a greedy associator model, a long short-term memory model, or an interacting multiple model. ### Claim 13 (depends on 1) The computer-implemented method of claim 1, wherein the sensor data comprises LIDAR data, and wherein the plurality of sensor data points comprises one or more LIDAR sensor data points. ### Claim 14 (depends on 1) The computer-implemented method of claim 1, further comprising: removing one or more nonviable sensor data points from the plurality of sensor data points. ### Claim 15 An autonomous vehicle computing system, comprising: a primary perception system configured to receive sensor data comprising a plurality of sensor data points as input and, in response to receipt of the sensor data as input, generate primary perception data representing a plurality of classifiable objects and a plurality of paths representing tracked motion of the plurality of classifiable objects, wherein each classifiable object is classified by the primary perception system as a predefined class of a plurality of predefined classes of objects; a secondary perception system configured to receive the sensor data comprising the plurality of sensor data points as input, and, in response to receipt of the sensor data as input, generate one or more sensor data point clusters and secondary path data representing tracked motion of the one or more sensor data point clusters; one or more processors; and one or more non-transitory, computer-readable media storing instructions that, when implemented, cause the one or more processors to perform operations, the operations comprising: obtaining the sensor data comprising the plurality of sensor data points, the sensor data descriptive of an environment of an autonomous vehicle; providing the sensor data as input to the primary perception system and, in response to providing the sensor data as input to the primary perception system, obtaining the primary perception data; providing the sensor data as input to the secondary perception system and, in response to providing the sensor data as input to the secondary perception system, obtaining the or more sensor data point clusters and the secondary path data; and determining fused perception data representing the tracked motion of the plurality of classifiable objects and the tracked motion of one or more unclassifiable objects, wherein the one or more unclassifiable objects are identified by deduplicating the plurality of classifiable objects with the one or more sensor data point clusters. ### Claim 16 (depends on 15) The autonomous vehicle computing system of claim 15, wherein the plurality of predefined classes comprises at least one of: (i) a vehicle class, (ii) a pedestrian class, or (iii) a bicycle class. ### Claim 17 (depends on 15) The autonomous vehicle computing system of claim 15, wherein the operations further comprise: predicting one or more predicted paths representing predicted motion of at least one of the plurality of classifiable objects or the plurality of unclassified objects based on the fused perception data; determining a motion plan for navigation of the autonomous vehicle based on the fused perception data and the one or more predicted paths; and implementing the motion plan to navigate the autonomous vehicle. ### Claim 18 An autonomous vehicle, comprising: a primary perception system configured to receive sensor data comprising a plurality of sensor data points as input and, in response to receipt of the sensor data as input, generate primary perception data representing a plurality of classifiable objects and a plurality of paths representing tracked motion of the plurality of classifiable objects, wherein each classifiable object is classified by the primary perception system as a predefined class of a plurality of predefined classes of objects; a secondary perception system configured to receive the sensor data comprising the plurality of sensor data points as input, and, in response to receipt of the sensor data as input, generate one or more sensor data point clusters and secondary path data representing tracked motion of the one or more sensor data point clusters; one or more processors; and one or more non-transitory, computer-readable media storing instructions that, when implemented, cause the one or more processors to perform operations, the operations comprising: obtaining the sensor data comprising the plurality of sensor data points, the sensor data descriptive of an environment of the autonomous vehicle; providing the sensor data as input to the primary perception system and, in response to providing the sensor data as input to the primary perception system, obtaining the primary perception data; providing the sensor data as input to the secondary perception system and, in response to providing the sensor data as input to the secondary perception system, obtaining the or more sensor data point clusters and the secondary path data; and determining fused perception data representing the tracked motion of the plurality of classifiable objects and the tracked motion of one or more unclassifiable objects, wherein the one or more unclassifiable objects are identified by deduplicating the plurality of classifiable objects with the one or more sensor data point clusters. ### Claim 19 (depends on 18) The autonomous vehicle of claim 18, wherein the plurality of predefined classes comprises at least one of (i) a vehicle class, (ii) a pedestrian class, or (iii) a bicycle class. ### Claim 20 (depends on 18) The autonomous vehicle of claim 18, wherein the operations further comprise: predicting one or more predicted paths representing predicted motion of at least one of the plurality of classifiable objects or the plurality of unclassified objects based on the fused perception data; determining a motion plan for navigation of the autonomous vehicle based on the fused perception data and the one or more predicted paths; and implementing the motion plan to navigate the autonomous vehicle.