# US 11,299,151 B2 · Enhanced vehicle tracking > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,299,151 B2 - Document ID: US11299151B2 - Status: Granted - Kind: B2 - Application number: 16/862,058 - Filed: 2020-04-29 - Published: 2020-12-03 - Granted: 2022-04-12 - Earliest claimed: 2018-03-15 - Assignees: Woven Planet North America, Inc. - Applicants: Woven Planet North America, Inc. - Inventors: Peter Ondruska; Lukas Platinsky; Suraj Mannakunnel Surendran - Web page: https://patentgrep.com/patent/US11299151B2 ## CPC - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - B60W 30/0956: {the prediction being responsive to traffic or environmental parameters} - G01C 21/3635: {Guidance using 3D or perspective road maps} - G01C 21/3647: {Guidance involving output of stored or live camera images or video streams} - G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00) - G05D 1/0212: {with means for defining a desired trajectory (involving a plurality of land vehicles G05D1/0287)} - G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)} - G06F 18/295: {Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models} - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06V 10/82: using neural networks - G06V 10/85: {Markov-related models; Markov random fields} - G06V 20/20: in augmented reality scenes - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - B60W 2520/06: Direction of travel - B60W 2520/10: Longitudinal speed - G05D 1/0231: {using optical position detecting means (position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16)} - G05D 1/027: {comprising intertial navigation means, e.g. azimuth detector (inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165)} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The present invention relates to a method and system for accurately predicting future trajectories of observed objects in dense and ever-changing city environments. More particularly, the present invention relates to substantially continuously tracking and estimating the future movements of an observed object. As an example, an observed object may be a moving vehicle, for example along a path or road. Aspects and/or embodiments seek to provide an end to end method and system for substantially continuously tracking and predicting future movements of a newly observed object, such as a vehicle, using motion prior data extracted from map data. ## Claims ### Claim 1 A computer-implemented method comprising: determining, by a computing system, from one or more captured images of an object in an environment, a trajectory of the object in the environment; determining, by the computing system, one or more prior trajectories of one or more prior objects in the environment; determining, by the computing system, an amount of similarity between the trajectory of the object and the one or more prior trajectories of the one or more prior objects; and based on the amount of similarity, predicting at least one future pose of the object in the environment. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the determining the amount of similarity comprises: determining, by the computing system, at least one projected position associated with the at least one future pose of the object; projecting, by the computing system, the at least one projected position to at least one of the one or more captured images of the object; and determining, by the computing system, whether the object follows the one or more prior trajectories based on a reprojection error between at least one position of the object in the at least one of the one or more captured images and the at least one projected position. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the object is determined to be associated with a new trajectory based on the reprojection error between the at least one position of the object and the at least one projected position failing to satisfy a threshold error. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the determining the amount of similarity between the trajectory of the object to the one or more prior trajectories of the one or more prior objects comprises: determining a pose of at least one of the one or more captured images; aligning the pose with a prior pose of a prior image associated with the one or more prior trajectories; and aligning the trajectory of the object with the one or more prior trajectories based on the aligning the pose with the prior pose of the prior image. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, further comprising: hypothesizing that the object will follow at least one of the one or more prior trajectories based on a first position of the object in a first image of the one or more captured images and at least one prior position of the one or more prior trajectories; determining that a likelihood that the object will follow the at least one of the one or more prior trajectories satisfies a threshold likelihood based on a second position of the object in a second image of the one or more captured images and at least one subsequent prior position of the one or more prior trajectories; and in response to the determining that the likelihood that the object will follow the at least one of the one or more prior trajectories satisfies the threshold likelihood, constraining the one or more prior trajectories to a subset of the one or more prior trajectories, wherein the subset of the one or more prior trajectories satisfy the threshold likelihood. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the at least one future pose of the object is in accordance with the object following the one or more prior trajectories. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the at least one future pose is predicted based on at least one of: a distance between an observed position of the object and a sample position of one of the one or more prior trajectories, a difference between an observed heading angle of the object and a sample heading angle associated with the one of the one or more prior trajectories, or a difference between an observed linear speed of the object and a sample linear speed of the one of the one or more prior trajectories. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the trajectory of the object is associated with a first pose at a first time and the at least one future pose is based on a distribution of prior poses associated with the one or more prior trajectories, wherein the prior poses correspond to a second time subsequent to the first time. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the one or more prior trajectories are based on at least one of: data sets captured for a structure-from-motion map of a locality associated with the environment or captured camera images, wherein each of the captured camera images include pose associated with the one or more prior objects. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, further comprising: determining, by the computing system, at least one of a position, rotation, or velocity of the object using the at least one future pose. ### Claim 11 A system comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform: determining from one or more captured images of an object in an environment, a trajectory of the object in the environment; determining one or more prior trajectories of one or more prior objects in the environment; determining an amount of similarity between the trajectory of the object and the one or more prior trajectories of the one or more prior objects; and based on the amount of similarity, predicting at least one future pose of the object in the environment. ### Claim 12 (depends on 11) The system of claim 11, wherein the determining the amount of similarity comprises: determining at least one projected position associated with the at least one future pose of the object; projecting the at least one projected position to at least one of the one or more captured images of the object; and determining whether the object follows the one or more prior trajectories based on a reprojection error between at least one position of the object in the at least one of the one or more captured images and the at least one projected position. ### Claim 13 (depends on 12) The system of claim 12, wherein the object is determined to be associated with a new trajectory based on the reprojection error between the at least one position of the object and the at least one projected position failing to satisfy a threshold error. ### Claim 14 (depends on 11) The system of claim 11, wherein the determining the amount of similarity between the trajectory of the object to the one or more prior trajectories of the one or more prior objects comprises: determining a pose of at least one of the one or more captured images; aligning the pose with a prior pose of a prior image associated with the one or more prior trajectories; and aligning the trajectory of the object with the one or more prior trajectories based on the aligning the pose with the prior pose of the prior image. ### Claim 15 (depends on 11) The system of claim 11, wherein the at least one processor is further configured to cause the system to perform: hypothesizing that the object will follow at least one of the one or more prior trajectories based on a first position of the object in a first image of the one or more captured images and at least one prior position of the one or more prior trajectories; determining that a likelihood that the object will follow the at least one prior trajectory satisfies a threshold likelihood based on a second position of the object in a second image and subsequent prior positions of the one or more prior objects; and in response to the determining that the likelihood that the object will follow the at least one of the one or more prior trajectories satisfies the threshold likelihood, constraining the one or more prior trajectories to a subset of the one or more prior trajectories, wherein the subset of the one or more prior trajectories satisfy the threshold likelihood. ### Claim 16 A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the at least one processor of the computing system to perform: determining from one or more captured images of an object in an environment, a trajectory of the object in the environment; determining one or more prior trajectories of one or more prior objects in the environment; determining an amount of similarity between the trajectory of the object and the one or more prior trajectories of the one or more prior objects; and based on the amount of similarity, predicting at least one future pose of the object in the environment. ### Claim 17 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the determining the amount of similarity comprises: determining at least one projected position associated with the at least one future pose of the object; projecting the at least one projected position to at least one of the one or more captured images of the object; and determining whether the object follows the one or more prior trajectories based on a reprojection error between at least one position of the object in the at least one of the one or more captured images and the at least one projected position. ### Claim 18 (depends on 17) The non-transitory computer-readable storage medium of claim 17, wherein the object is determined to be associated with a new trajectory based on the reprojection error between the at least one position of the object and the at least one projected position failing to satisfy a threshold error. ### Claim 19 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the determining the similarity between the trajectory of the object to the one or more prior trajectories of the one or more prior objects comprises: determining a pose of at least one of the one or more captured images; aligning the pose with a prior pose of a prior image associated with the one or more prior trajectories; and aligning the trajectory of the object with the one or more prior trajectories based on the aligning the pose with the prior pose of the prior image. ### Claim 20 (depends on 16) The non-transitory computer-readable storage medium of claim 16, further comprising: hypothesizing that the object will follow at least one of the one or more prior trajectories based on a first position of the object in a first image of the one or more captured images and at least one prior position of the one or more prior trajectories; determining that a likelihood that the object will follow the at least one of the one or more prior trajectories satisfies a threshold likelihood based on a second position of the object in a second image of the one or more captured images and at least one subsequent prior positions of the one or more prior trajectories; and in response to the determining that the likelihood that the object will follow the at least one of the one or more prior trajectories satisfies the threshold likelihood, constraining the set of prior trajectories to a subset of the one or more prior trajectories, wherein the subset of the one or more prior trajectories satisfy the threshold likelihood.