# US 2020/0401819 A1 · Image based locationing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401819 A1 - Document ID: US20200401819A1 - Status: Published application - Kind: A1 - Application number: 16/911,287 - Filed: 2020-06-24 - Published: 2020-12-24 - Earliest claimed: 2019-06-22 - Inventors: Erik Volkerink; Ajay Khoche - Web page: https://patentgrep.com/patent/US20200401819A1 ## CPC - G06T 7/285: using a sequence of stereo image pairs - G06T 7/593: from stereo images - G06T 7/73: using feature-based methods - G06V 10/147: Details of sensors, e.g. sensor lenses (fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19) - G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching} - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20021: Dividing image into blocks, subimages or windows - G06T 2207/30241: Trajectory - G06T 2207/30244: Camera pose - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract This disclosure relates to systems and methods of obtaining accurate motion and orientation estimates for a vehicle traveling at high speed based on images of a road surface. A purpose of these systems and methods is to provide a supplementary or alternative means of locating a vehicle on a map, particularly in cases where other locationing approaches (e.g., GPS) are unreliable or unavailable. ## Claims ### Claim 1 A method of processing a sequence of images to determine an image based trajectory of a vehicle along a road, comprising by a data processing system: rectifying each stereoscopic pair of images to a common epipolar plane; detecting features in each of the rectified stereoscopic pair of images in each successive frame; matching corresponding features in each rectified stereoscopic pair of images, wherein the matching comprises matching points in one image of the rectified stereoscopic pair with corresponding features in the other image of the stereoscopic image pair to produce a feature disparity map; calculating a depth at each location point to obtain a sparse three-dimensional depth map of the road; and determining motion and orientation of the vehicle between successive stereoscopic image frames based on execution of an optical flow process by the data processing system that determines estimates of motion and orientation between successive stereoscopic pairs of images. ### Claim 2 (depends on 1) The method of claim 1, comprising by the data processing system, converting the image of the stereoscopic pair of images to a grayscale format. ### Claim 3 (depends on 1) The method of claim 1, wherein the disparity map measures pixel displacement between the matched features. ### Claim 4 (depends on 1) The method of claim 1, further comprising estimating a current trajectory of the vehicle based on the estimates of motion and orientation between successive stereoscopic pairs of images. ### Claim 5 (depends on 1) The method of claim 1, further comprising by the data processing system, dividing an image region above a road surface into multiple tracks, and processing the multiple tracks to obtain multiple independent estimates of a trajectory of the vehicle. ### Claim 6 (depends on 5) The method of claim 5, further comprising by the data processing system, combining multiple ones of the estimates to determine estimates of motion and orientation of the vehicle between successive stereoscopic frames. ### Claim 7 (depends on 6) The method of claim 6, further comprising by the data processing system, identifying and rejecting as outliers motion and orientation estimates that are inconsistent with a majority of other motion and orientation estimates. ### Claim 8 (depends on 5) The method of claim 5, further comprising by the data processing system, processing each of the tracks independently of other tracks to obtain a respective set of features for the stereographic image in the frame for each track. ### Claim 9 (depends on 8) The method of claim 8, further comprising by the data processing system, determining a respective set of disparity and depth maps for each track. ### Claim 10 (depends on 9) The method of claim 9, further comprising by the data processing system, dividing an imaged road region into a rectangular array multiple tracks wide and multiple tracks long, and processing the tracks to obtain respective independent sets of estimates of motion and orientation along a trajectory of the vehicle. ### Claim 11 (depends on 10) The method of claim 10, wherein the dividing comprises dividing the imaged road region into the rectangular array by capturing a stereographic image set and windowing the imaged road surface into the sets of tracks. ### Claim 12 (depends on 10) The method of claim 10, wherein the dividing comprises dividing the imaged road region into the rectangular array using optical elements configured to direct light reflecting from a surface of the road to respective ones of the sets of tracks. ### Claim 13 (depends on 10) The method of claim 10, wherein the dividing comprises dividing the imaged road region into the rectangular array using respective pairs of stereoscopic image devices array according to the track layout. ### Claim 14 A method of processing a sequence of successive stereographic image frames to determine estimates of motion and orientation of a vehicle, the method comprising: dividing an image region above a road surface into multiple tracks; for each track, by the data processing system, estimating motion and orientation between corresponding features in respective stereographic image pairs; removing, by the data processing system, inconsistent motion and orientation estimates as outliers; determining, by the data processing system, estimates of motion and orientation based on an aggregation of consistent motion and orientation estimates; and processing the multiple tracks to obtain multiple independent estimates of a trajectory of the vehicle. ### Claim 15 (depends on 14) The method of claim 14, wherein consistent motion and orientation estimates are averaged to determine the motion and orientation estimates. ### Claim 16 A computer program product for execution by a computer system and comprising at least one non-transitory computer-readable medium having computer readable program code portions embodied therein to process a sequence of images to determine an image based trajectory of a vehicle along a road, the computer-readable program code portions comprising: an executable code portion to rectify each stereoscopic pair of images to a common epipolar plane; an executable code portion to detect features in each of the rectified stereoscopic pair of images in each successive frame; an executable code portion to match corresponding features in each rectified stereoscopic pair of images, wherein the matching comprises matching points in one image of the rectified stereoscopic pair with corresponding features in the other image of the stereoscopic image pair to produce a feature disparity map; an executable code portion to calculate a depth at each location point to obtain a sparse three-dimensional depth map of the road; and an executable code portion to determine motion and orientation of the vehicle between successive stereoscopic image frames based on execution of an optical flow process by the data processing system that determines estimates of motion and orientation between successive stereoscopic pairs of images.