# US 12,120,429 B2 · Method and system for real-time geo referencing stabilization > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,120,429 B2 - Document ID: US12120429B2 - Status: Granted - Kind: B2 - Application number: 17/889,650 - Filed: 2022-08-17 - Published: 2024-03-21 - Granted: 2024-10-15 - Earliest claimed: 2022-04-19 - Assignees: Palantir Technologies Inc. - Applicants: Palantir Technologies Inc. - Inventors: Jay Harshadbhai Patel - Web page: https://patentgrep.com/patent/US12120429B2 ## CPC - G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)} - H04N 23/683: {performed by a processor, e.g. controlling the readout of an image memory} - B64C 39/024: {of the remote controlled vehicle type, i.e. RPV} - G06F 18/24147: {Distances to closest patterns, e.g. nearest neighbour classification} - G06T 7/35: using statistical methods - G06T 7/38: Registration of image sequences - G06T 15/06: Ray-tracing - G06V 20/17: taken from planes or by drones - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - B64U 2101/30: for imaging, photography or videography - G06T 2207/10016: Video; Image sequence - G06T 2207/10032: Satellite or aerial image; Remote sensing - G06T 2207/30181: Earth observation - G06T 2207/30241: Trajectory - G06T 2207/30242: Counting objects in image - G06V 2201/10: Recognition assisted with metadata - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - B64C: AEROPLANES; HELICOPTERS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B64U: UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR ## Abstract A computing system and methods are provided for georeferencing stabilization. An exemplary method includes: obtaining a video stream capturing an area from a camera of a drone, where the video stream includes a plurality of frames, each including a field of view of the image capturing device and metadata of the image capturing device when the frame is captured; constructing a geographic (geo) lattice for the field of view in each of the plurality of frames, the geo lattice comprises a plurality of points, each being associated with raw coordinates determined based on the corresponding metadata; and building a lattice map with stabilized geo coordinates by (1) aligning the frames, (2) averaging the raw geo coordinates for given intersection points, and (3) building the lattice map based on the averaged geo coordinates of the intersection points. ## Claims ### Claim 1 A computer-implemented method for georeferencing stabilization, comprising: obtaining, from an image capturing device on a vehicle, a video stream capturing an area, wherein the video stream comprises a plurality of frames, each of the plurality of frames comprises a field of view of the image capturing device and metadata of the image capturing device when the frame is captured; constructing a geographic (geo) lattice for the field of view in each of the plurality of frames, wherein the geo lattice comprises a plurality of points, each point being associated with a pair of raw coordinates determined based on the corresponding metadata of the image capturing device; building a lattice map for the area based on the plurality of geo lattices, wherein the lattice map comprises points within the plurality of geo lattices, and building the lattice map comprises: performing image registration based on the plurality of frames to obtain a plurality of aligned geo lattices; for each point in the lattice map, obtaining one or more pairs of raw coordinates corresponding to the point from the plurality of aligned geo lattices; computing a pair of stabilized geographic coordinates for the point by averaging any of the one or more pairs of raw coordinates; and building the lattice map for the area by associating the point with the pair of stabilized geographic coordinates; receiving a request for geographic coordinates of a ground point in the area; identifying multiple lattice map points in the lattice map surrounding the ground point in the request; and computing the geographic coordinates of the ground point by interpolating the multiple pairs of stabilized geographic coordinates associated with the multiple lattice map points in the lattice map. ### Claim 2 (depends on 1) The method of claim 1, wherein building the lattice map for the area comprises: building the lattice map based on an estimated projection of the field of view of the image capturing device. ### Claim 3 (depends on 2) The method of claim 2, wherein building the lattice map with an estimated projection comprises: orthorectifying the plurality of geo lattices using random sample consensus (RANSAC) to estimate homography matrix for a Nadir projection of the plurality of frames captured by the image capturing device. ### Claim 4 (depends on 2) The method of claim 2, further comprising: receiving an incoming frame from the video stream when the vehicle pans, wherein the incoming frame captures a new location of the area absent from the plurality of frames; constructing a new geo lattice for the incoming frame; extrapolating the lattice map for the area based on the Nadir projection to obtain estimated geo lattices for locations beyond the field of view of the image capturing device; and updating the lattice map by averaging raw coordinates of the new geo lattice with coordinates of the estimated geo lattices. ### Claim 5 (depends on 1) The method of claim 1, wherein the geo lattice is constructed by using ray tracing. ### Claim 6 (depends on 1) The method of claim 1, wherein the image registration comprises: computing a homography matrix between a first frame and a second frame of the plurality of frames; and mapping the plurality of points in the geo lattice of the first frame to the plurality of points in the geo lattice of the second frame based on the homography matrix. ### Claim 7 (depends on 6) The method of claim 6, further comprising: determining an error rate of the mapping between the plurality of points in the geo lattice of the first frame to the plurality of points in the geo lattice of the second frame based on the homography matrix; and in response to the error rate being greater than a threshold, skipping one of the first frame or the second frame in building the lattice map. ### Claim 8 (depends on 6) The method of claim 6, wherein the computing of the homography matrix comprises: extracting features from the first frame and the second frame using accelerated-KAZE feature detection and description algorithm; matching the extracted features using K-nearest neighbor search; and constructing the homography matrix by using RANSAC homography estimation based on the matched extracted features. ### Claim 9 (depends on 1) The method of claim 1, wherein the image registration comprises: dividing the plurality of frames into multiple overlapping groups of frames, each group comprising two consecutive frames, and two consecutive groups share one frame; computing multiple homography matrices based on the multiple overlapping groups of frames using parallel processing; and caching the multiple homography matrices for computing a transformation matrix between any two frames of the plurality of frames. ### Claim 10 (depends on 1) The method of claim 1, wherein the image registration is performed on every K frames, where K is an integer greater than one, and the image registration comprises: computing a homography matrix for a first frame of the K frames and a k-th frame of the K frames; and computing intermediate homography matrices for every two frames of the K frames by applying linear interpolation to the homography matrix. ### Claim 11 (depends on 1) The method of claim 1, wherein the lattice map comprises a first channel corresponding to latitude, a second channel corresponding to longitude, and a third channel corresponding to a number of observations of a corresponding point in the plurality of geo lattices. ### Claim 12 (depends on 1) The method of claim 1, wherein the vehicle comprises a drone or a Synthetic-aperture radar (SAR). ### Claim 13 A computing system for georeferencing stabilization, comprising: one or more processors; and memory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising: obtaining, from an image capturing device on a vehicle, a video stream capturing an area, wherein the video stream comprises a plurality of frames, each of the plurality of frames comprises a field of view of the image capturing device and metadata of the image capturing device when the frame is captured; constructing a geographic (geo) lattice for the field of view in each of the plurality of frames, wherein the geo lattice comprises a plurality of points, each being associated with a pair of raw coordinates determined based on the corresponding metadata of the image capturing device; building a lattice map for the area based on the plurality of geo lattices, wherein the lattice map comprises within the plurality of geo lattices, and building the lattice map comprises: performing image registration based on the plurality of frames to obtain a plurality of aligned geo lattices; for each point in the lattice map, obtaining one or more pairs of raw coordinates corresponding to the point from the plurality of aligned geo lattices; computing a pair of stabilized geographic coordinates for the point by averaging any of the one or more pairs of raw coordinates; and building the lattice map for the area by associating the point with the pair of stabilized geographic coordinates; receiving a request for geographic coordinates of a ground point in the area; identifying multiple lattice map points in the lattice map surrounding the ground point in the request; and computing the geographic coordinates of the ground point by interpolating the multiple pairs of stabilized geographic coordinates associated with the multiple lattice map points in the lattice map. ### Claim 14 (depends on 13) The computing system of claim 13, wherein building the lattice map for the area comprises: building the lattice map based on an estimated projection of the field of view of the image capturing device. ### Claim 15 (depends on 13) The computing system of claim 13, wherein the image registration comprises: computing a homography matrix between a first frame and a second frame of the plurality of frames; and mapping the plurality of points in the geo lattice of the first frame to the plurality of points in the geo lattice of the second frame based on the homography matrix. ### Claim 16 (depends on 15) The computing system of claim 15, wherein the computing of the homography matrix comprises: extracting features from the first frame and the second frame using accelerated-KAZE feature detection and description algorithm; matching the extracted features using K-nearest neighbor search; and constructing the homography matrix by using RANSAC homography estimation based on the matched extracted features. ### Claim 17 (depends on 13) The computing system of claim 13, wherein the image registration comprises: dividing the plurality of frames into multiple overlapping groups of frames, each group comprising two consecutive frames, and two consecutive groups share one frame; computing multiple homography matrices based on the multiple overlapping groups of frames using parallel processing; and caching the multiple homography matrices for computing a transformation matrix between any two frames of the plurality of frames. ### Claim 18 A non-transitory computer-readable storage medium configured with instructions executable by one or more processors to cause the one or more processors to perform operations comprising: obtaining, from an image capturing device on a vehicle, a video stream capturing an area, wherein the video stream comprises a plurality of frames, each of the plurality of frames comprises a field of view of the image capturing device and metadata of the image capturing device when the frame is captured; constructing a geographic (geo) lattice for the field of view in each of the plurality of frames, wherein the geo lattice comprises a plurality of points, each being associated with a pair of raw coordinates determined based on the corresponding metadata of the image capturing device; computing a transformation matrix between a first frame and a second frame of the plurality of frames; building a lattice map for the area based on the plurality of geo lattices, wherein the lattice map comprises points within the plurality of geo lattices, and building the lattice map comprises: performing image registration based on the plurality of frames to obtain a plurality of aligned geo lattices; for each point in the lattice map, obtaining one or more pairs of raw coordinates corresponding to the point from the plurality of aligned geo lattices; computing a pair of stabilized geographic coordinates for the point by averaging any of the one or more pairs of raw coordinates; mapping one or more points in the geo lattice of the first frame to one or more points in the geo lattice of the second frame based on the transformation matrix; determining an error rate of the mapping between one or more points in the geo lattice of the first frame and one or more corresponding points in the geo lattice of the second frame; in response to the error rate being greater than a threshold, skipping one of the first frame or the second frame in the building of the lattice map; and building the lattice map for the area by associating the point with the pair of stabilized geographic coordinates.