# US 11,906,625 B2 · Surround vehicle tracking and motion prediction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,906,625 B2 - Document ID: US11906625B2 - Status: Granted - Kind: B2 - Application number: 16/960,290 - Filed: 2019-01-08 - Published: 2021-02-25 - Granted: 2024-02-20 - Earliest claimed: 2018-01-08 - Assignees: The Regents of the University of California - Applicants: The Regents of the University of California - Inventors: Akshay Rangesh; Mohan M. Trivedi; Nachiket Deo - Web page: https://patentgrep.com/patent/US11906625B2 ## CPC - G01S 17/89: for mapping or imaging - B60W 30/0956: {the prediction being responsive to traffic or environmental parameters} - B60W 60/00272: {relying on extrapolation of current movement} - G01S 17/66: Tracking systems using electromagnetic waves other than radio waves - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/931: of land vehicles - G06F 18/24: Classification techniques - G06F 18/251: {of input or preprocessed data} - G06T 7/248: {involving reference images or patches} - G06T 7/251: {involving models} - G06T 7/292: Multi-camera tracking - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - B60W 50/0097: {Predicting future conditions} - B60W 2050/0028: {Mathematical models, e.g. for simulation} - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2420/408: Radar; Laser, e.g. lidar - B60W 2554/404: Characteristics - B60W 2556/35: Data fusion - G06T 2207/10016: Video; Image sequence - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20076: Probabilistic image processing - G06T 2207/30241: Trajectory - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06T 2207/30261: Obstacle - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A surround multi-object tracking and surround vehicle motion prediction framework is provided. A full-surround camera array and LiDAR sensor based approach provides for multi-object tracking for autonomous vehicles. The multi-object tracking incorporates a fusion scheme to handle object proposals from the different sensors within the calibrated camera array. A motion prediction framework leverages the instantaneous motion of vehicles, an understanding of motion patterns of freeway traffic, and the effect of inter-vehicle interactions. The motion prediction framework incorporates probabilistic modeling of surround vehicle trajectories. Additionally, subcategorizing trajectories based on maneuver classes leads to better modeling of motion patterns. A model takes into account interactions between surround vehicles for simultaneously predicting each of their motion. ## Claims ### Claim 1 A computer-implemented method, comprising: receiving, by at least one data processor and from one or more sensors positioned on a vehicle, data indicative of a detected object in an area at least partially surrounding the vehicle; fusing, by the at least one data processor, object proposals of the detected object identified from the received data to form a fused object proposal such that the fused object proposal is representative of the detected object, wherein fusing object proposals comprises: projecting, by the at least one data processor, the object proposals onto a global coordinate frame with respect to the vehicle; and combining, by the at least one data processor and based on the projection of the object proposals, a subset of object proposals determined to belong to the detected object; generating, by the processing device, a track of the detected object, wherein the track is based on the fused object proposal and updated received data indicative of the detected object; determining, by the at least one data processor, a maneuver class of the detected object, the maneuver class indicative of motion of the detected object and based on the generated track; determining, by the at least one data processor, a plurality of future trajectories for the detected object, the plurality of future trajectories based on the generated track and at least one of a motion model or a probabilistic model; and determining, by the processing device, an expected future trajectory of the detected object based on the corresponding maneuver class and the plurality of future trajectories. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the received data comprises: a video feed, from a plurality of cameras positioned on the vehicle, of the area at least partially surrounding the vehicle; and at least one point cloud, from at least one LiDAR sensor positioned on the vehicle, the at least one point cloud comprising LiDAR data associated with the area at least partially surrounding the vehicle. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein generating the track of the detected object comprises: identifying, by the at least one data processor, a status of the detected object; and identifying, by the at least one data processor and if the status is an active status, a sequence of positions of the detected object based on the updated received data. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein the status of the detected object is identified by a Markov Decision Process. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the maneuver class of the detected object is based on a Hidden Markov Model applied to a period of time of the generated track. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein determining the plurality of future trajectories for the detected object comprises: averaging, by the at least one data processor, predicted future locations and covariances provided by the probabilistic model. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the motion model comprises a constant velocity motion model, a constant acceleration motion model, and a constant turn rate and velocity motion model. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the expected future trajectory for the detected object is based on a feasibility assessment of each of the plurality of future trajectories, wherein the feasibility assessment is based on a configuration of other detected objects within a predefined range of the vehicle. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, further comprising: generating, by the at least one data processor, an output representative of the expected future trajectory of the detected object. ### Claim 10 A system, comprising: at least one data processor; and at least one memory storing instructions which, when executed by the at least one data processor, result in operations comprising: receiving, from one or more sensors positioned on a vehicle, data indicative of a detected object in an area at least partially surrounding the vehicle; fusing object proposals of the detected object identified from the received data to form a fused object proposal such that the fused object proposal is representative of the detected object, wherein fusing object proposals comprises: projecting the object proposals onto a global coordinate frame with respect to the vehicle; and combining, based on the projection of the object proposals, a subset of object proposals determined to belong to the detected object; generating a track of the detected object, wherein the track is based on the fused object proposal and updated received data indicative of the detected object; determining a maneuver class of the detected object, the maneuver class indicative of motion of the detected object and based on the generated track; determining a plurality of future trajectories for the detected object, the plurality of future trajectories based on the generated track and at least one of a motion model or a probabilistic model; and determining an expected future trajectory of the detected object based on the corresponding maneuver class and the plurality of future trajectories. ### Claim 11 (depends on 10) The system of claim 10, wherein the received data comprises: a video feed, from a plurality of cameras positioned on the vehicle, of the area at least partially surrounding the vehicle; and at least one point cloud, from at least one LiDAR sensor positioned on the vehicle, the at least one point cloud comprising LiDAR data associated with the area at least partially surrounding the vehicle. ### Claim 12 (depends on 10) The system of claim 10, wherein generating the track of the detected object comprises: identifying a status of the detected object; and identifying, if the status is an active status, a sequence of positions of the detected object based on the updated received data. ### Claim 13 (depends on 12) The system of claim 12, wherein the status of the detected object is identified by a Markov Decision Process. ### Claim 14 (depends on 10) The system of claim 10, wherein the maneuver class of the detected object is based on a Hidden Markov Model applied to a period of time of the generated track. ### Claim 15 (depends on 10) The system of claim 10, wherein determining the plurality of future trajectories for the detected object comprises: averaging, by the at least one data processor, predicted future locations and covariances provided by the probabilistic model. ### Claim 16 (depends on 10) The system of claim 10, wherein the motion model comprises a constant velocity motion model, a constant acceleration motion model, and a constant turn rate and velocity motion model. ### Claim 17 (depends on 10) The system of claim 10, wherein the expected future trajectory for the detected object is based on a feasibility assessment of each of the plurality of future trajectories, wherein the feasibility assessment is based on a configuration of other detected objects within a predefined range of the vehicle. ### Claim 18 (depends on 10) The system of claim 10, wherein execution of the at least one memory storing instructions further result in: generating a control signal indicative of the expected future trajectory of the detected object. ### Claim 19 A non-transitory computer-readable storage medium including program code, which when executed by at least one data processor, causes operations comprising: receiving, by at least one data processor a processing and from one or more sensors positioned on a vehicle, data indicative of a detected object in an area at least partially surrounding the vehicle; fusing, by the at least one data processor, object proposals of the detected object identified from the received data to form a fused object proposal such that the fused object proposal is representative of the detected object, wherein fusing object proposals comprises: projecting, by the at least one data processor, the object proposals onto a global coordinate frame with respect to the vehicle; and combining, by the at least one data processor and based on the projection of the object proposals, a subset of object proposals determined to belong to the detected object; generating, by the at least one data processor, a track of the detected object, wherein the track is based on the fused object proposal and updated received data indicative of the detected object; determining, by the at least one data processor, a maneuver class of the detected object, the maneuver class indicative of motion of the detected object and based on the generated track; determining, by the at least one data processor, a plurality of future trajectories for the detected object, the plurality of future trajectories based on the generated track and at least one of a motion model or a probabilistic model; and determining, by the at least one data processor, an expected future trajectory of the detected object based on the corresponding maneuver class and the plurality of future trajectories.