# US 12,008,786 B2 · Method, apparatus, and computer program product for point-to-point translation between images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,786 B2 - Document ID: US12008786B2 - Status: Granted - Kind: B2 - Application number: 17/132,790 - Filed: 2020-12-23 - Published: 2022-06-23 - Granted: 2024-06-11 - Earliest claimed: 2020-12-23 - Assignees: HERE Global B.V. - Applicants: Here Global B.V. - Inventors: David Lawlor; Niranjan Kotha; Zhanwei Chen - Web page: https://patentgrep.com/patent/US12008786B2 ## CPC - G06T 7/579: from motion - G06T 7/74: {involving reference images or patches} - G01C 11/02: Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures - G01C 21/3841: {Data obtained from two or more sources, e.g. probe vehicles} - G01C 21/3848: {Data obtained from both position sensors and additional sensors} - G01C 21/3852: {Data derived from aerial or satellite images} - G06F 16/2379: {Updates performed during online database operations; commit processing} - G06F 16/29: Geographical information databases - G06T 3/20: Linear translation of whole images or parts thereof, e.g. panning - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 2207/30244: Camera pose - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method, apparatus and computer program product are provided for establishing correspondences between images using through generation of a translation between the different perspectives. Methods may include: receiving first sensor data from a first image sensor, where the first sensor data includes a first image of an environment captured from a first perspective; receiving second sensor data from a second image sensor of a second image of the environment captured from a second perspective; identifying image correspondence points between the first sensor data and the second sensor data; computing pairwise vectors between corresponding pairs of first projected points of the first sensor data and second projected points of the second sensor data; clustering the pairwise vectors according to magnitude and orientation; and generating a translation vector from clusters of pairwise vectors, where the translation vector represents a shift of the second image sensor data to correspond to ground truth data. ## Claims ### Claim 1 An apparatus comprising at least one processor and at least one non-transitory memory including computer program code instructions, the computer program code instructions configured to, when executed, cause the apparatus to at least: receive three-dimensional coordinates of points in a first image of an environment captured in first sensor data by a first image sensor from a first perspective; project the three-dimensional coordinates of points in the first image of the environment onto a plane of projection to form first projected points; receive second sensor data from a second image sensor, wherein the second sensor data comprises a second image of the environment captured from a second perspective; project rays from the second sensor data into the plane of projection to generate second projected points, wherein the rays comprise lines that extend from a position of the second sensor through image correspondence points of the second sensor data; compute pairwise vectors between corresponding pairs of the first projected points of the first sensor data and the second projected points of the second sensor data; and generate a translation vector from the pairwise vectors, wherein the translation vector represents a shift of the second image sensor data to correspond to a ground truth data. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein the first image of the environment captured from the first perspective is captured along a first axis, wherein the second image of the environment captured from the second perspective is captured along a second axis, wherein the first axis and the second axis are separated by at least 45 degrees. ### Claim 3 (depends on 1) The apparatus of claim 1, wherein causing the apparatus to generate a translation vector from the pairwise vectors comprises causing the apparatus to: cluster the pairwise vectors according to magnitude and orientation; and generate a translation vector from clusters of the pairwise vectors. ### Claim 4 (depends on 1) The apparatus of claim 1, wherein the apparatus is further caused to translate the second projected points using the translation vector to obtain correspondences between the second projected points and the first projected points. ### Claim 5 (depends on 4) The apparatus of claim 4, wherein the apparatus is further caused to: establish an orientation of the second image sensor based, at least in part, on the correspondences between the second projected points and the first projected points. ### Claim 6 (depends on 4) The apparatus of claim 4, wherein the apparatus is further caused to: identify a precise location of the second image based, at least in part, on the correspondences between the second projected points and the first projected points. ### Claim 7 (depends on 6) The apparatus of claim 6, wherein the apparatus is further caused to: provide for building or updating a map in a map database based on the precise location of the second image. ### Claim 8 (depends on 1) The apparatus of claim 1, wherein the apparatus is further caused to: assign correspondences between the first projected points and the second projected points using the translation vector. ### Claim 9 (depends on 8) The apparatus of claim 8, wherein causing the apparatus to assign correspondences between the first projected points and the second projected points using the translation vector comprises causing the apparatus to assign correspondences between a respective second projected point with a respective closest one of the first projected points and excluding second projected points with a distance to a respective closest one of the first projected points above a predefined threshold distance. ### Claim 10 A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising program code instructions to: receive three-dimensional coordinates of points in a first image of an environment captured in first sensor data by a first image sensor from a first perspective; project the three-dimensional coordinates of points in the first image of the environment onto a plane of projection to form first projected points; receive second sensor data from a second image sensor, wherein the second sensor data comprises a second image of the environment captured from a second perspective; project rays from the second sensor data into the plane of projection to generate second projected points, wherein the rays comprise lines that extend from a position of the second sensor through image correspondence points of the second sensor data; compute pairwise vectors between corresponding pairs of the first projected points of the first sensor data and the second projected points of the second sensor data; and generate a translation vector from the pairwise vectors wherein the translation vector represents a shift of the second image sensor data to correspond to a ground truth data. ### Claim 11 (depends on 10) The computer program product of claim 10, wherein the first image of the environment captured from the first perspective is captured along a first axis, wherein the second image of the environment captured from the second perspective is captured along a second axis, wherein the first axis and the second axis are separated by at least 45 degrees. ### Claim 12 (depends on 10) The computer program product of claim 10, wherein the program code instructions to generate a translation vector from the pairwise vectors comprise program code instructions to: cluster the pairwise vectors according to magnitude and orientation; and generate a translation vector from clusters of the pairwise vectors. ### Claim 13 (depends on 10) The computer program product of claim 10, further comprising program code instructions to translate the second projected points using the translation vector to obtain correspondences between the second projected points and the first projected points. ### Claim 14 (depends on 13) The computer program product of claim 13, further comprising program code instructions to: establish an orientation of the second image sensor based, at least in part, on the correspondences between the second projected points and the first projected points. ### Claim 15 (depends on 13) The computer program product of claim 13, further comprising program code instructions to: identify a precise location of the second image based, at least in part, on the correspondences between the second projected points and the first projected points. ### Claim 16 (depends on 15) The computer program product of claim 15, further comprising program code instructions to: provide for building or updating a map in a map database based on the precise location of the second image. ### Claim 17 (depends on 10) The computer program product of claim 10, further comprising program code instructions to: assign correspondences between the first projected points and the second projected points using the translation vector. ### Claim 18 (depends on 17) The computer program product of claim 17, wherein the program code instructions to assign correspondences between the first projected points and the second projected points using the translation vector comprise program code instructions to assign correspondences between a respective second projected point with a respective closest one of the first projected points and excluding second projected points with a distance to a respective closest one of the first projected points above a predefined threshold distance. ### Claim 19 A method comprising: receiving first sensor data from a first image sensor, wherein the first sensor data comprises a first image of an environment captured from a first perspective; receiving second sensor data from a second image sensor, wherein the second sensor data comprises a second image of the environment captured from a second perspective; identifying image correspondence points between the first sensor data and the second sensor data; computing pairwise vectors between corresponding pairs of first projected points of the first sensor data and second projected points of the second sensor data; clustering the pairwise vectors according to magnitude and orientation; and generating a translation vector from clusters of pairwise vectors, wherein the translation vector represents a shift of the second image sensor data to correspond to a ground truth data. ### Claim 20 (depends on 19) The method of claim 19, wherein computing pairwise vectors between corresponding pairs of first projected points of the first sensor data and second projected points of the second sensor data comprises: projecting the three-dimensional coordinates of points in the first image of the environment onto a plane of projection to form the first projected points; projecting rays from the second sensor data into the plane of projection to generate the second projected points, wherein the rays comprise lines that extend from a position of the second sensor through image correspondence points of the second sensor data; and computing pairwise vectors between corresponding pairs of the first projected points of the first sensor data and the second projected points of the second sensor data.