# US 12,008,740 B2 · Feature matching using features extracted from perspective corrected image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,740 B2 - Document ID: US12008740B2 - Status: Granted - Kind: B2 - Application number: 17/396,568 - Filed: 2021-08-06 - Published: 2022-02-17 - Granted: 2024-06-11 - Earliest claimed: 2020-08-12 - Assignees: Niantic, Inc. - Applicants: Niantic, Inc. - Inventors: Carl Sebastian Toft; Daniyar Turmukhambetov; Gabriel J. Brostow - Web page: https://patentgrep.com/patent/US12008740B2 ## CPC - G06T 5/80: Geometric correction - G06T 3/00: Geometric image transformations in the plane of the image - G06T 7/187: involving region growing; involving region merging; involving connected component labelling - G06T 7/50: Depth or shape recovery - G06T 7/74: {involving reference images or patches} - H04N 5/2628: {Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation} - A63F 13/216: using geographical information, e.g. location of the game device or player using GPS - A63F 13/65: automatically by game devices or servers from real world data, e.g. measurement in live racing competition - A63F 2300/69: Involving elements of the real world in the game world, e.g. measurement in live races, real video - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30244: Camera pose - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR ## Abstract An image localization system receives an image of a scene and generates a depth map for the image by inputting the image to a model trained for generating depth maps for images. The system determines surface normal vectors for the pixels in the depth map. The system clusters the surface normal vectors to identify regions in the image corresponding to planar surfaces. The system partitions the image into patches, each of which is a region of connected pixels in the image and corresponds to a cluster of surface normal vectors. The system rectifies the perspective distortion of patches and extracts perspective corrected features from the rectified patches. The system matches the perspective corrected features of the image with perspective corrected features of other images for three-dimensional re-localization. ## Claims ### Claim 1 A computer-implemented method comprising: receiving an image of a scene, the image including pixels; determining a depth map of the image, the depth map including a plurality of pixels; for each of the pixels in the depth map, computing a surface normal vector that indicates an estimated surface normal of the pixel in the depth map; classifying the pixels in the depth map into clusters based on the surface normal vectors of the pixels in the depth map, each cluster including a subset of the pixels in the depth map; generating patches of connected pixels in the image based on the clusters of the depth map, wherein one of the patches of the image corresponds to one of the clusters of the depth map; rectifying a perspective distortion of each respective patch of the patches based on the surface normal vectors of the pixels of the depth map in the corresponding cluster to generate a rectified patch, wherein rectifying the perspective distortion of a first patch of the patches comprises: determining a first angle difference between an optical axis of a camera that captured the image and a surface normal of the first patch; subsequent to determining the first angle difference is greater than a threshold, identifying a portion of the first patch based on a second angle difference between the optical axis of the camera and a surface normal of the portion of the first patch, wherein the second angle difference is not greater than the threshold; and rectifying a perspective of the identified portion of the first patch without rectifying a remaining portion of the first patch; and extracting perspective corrected features from the rectified patches, the perspective corrected features to be used for feature matching. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein determining the depth map of the image comprises: determining the depth map of the image by inputting the image into a model, the model trained to receive images and output depth maps of the received images. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the model is a convolutional neural network. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein classifying the pixels into the clusters based on the surface normal vectors of the pixels comprises: using k-means clustering to classify the pixels into the clusters based on the surface normal vectors of the pixels. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein classifying the pixels into the clusters based on the surface normal vectors of the pixels comprises: classifying the surface normal vectors into three clusters based on the surface normal vectors of the pixels, the three clusters corresponding to three planes orthogonal to each other in a three-dimensional space. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein rectifying the perspective distortion of each respective patch of the patches based on the surface normal vectors of the pixels in the corresponding cluster to generate the rectified patch comprises: computing a homography that simulates a virtual camera that has a same camera center as the camera that captured the image and has an optical axis along an estimated surface normal of the corresponding cluster; and using the homography to perform a projective transformation to the patch to generate the rectified patch. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein the estimated surface normal of the corresponding cluster is a mean of the surface normal vectors of the pixels in the corresponding cluster. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the threshold is 80 degrees. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein extracting the perspective corrected features from the rectified patches comprises: identifying a non-planar portion of the image that is not included in any of the patches; extracting features from the non-planar portion of the image; and extracting features, the features including the perspective corrected features and features extracted from the non-planar portion of the image. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, further comprising: matching the perspective corrected features with features of a second image to re-localize the image with respect to the second image. ### Claim 11 A non-transitory computer-readable memory storing executable computer program instructions, the instructions executable to perform operations comprising: receiving an image of a scene, the image including pixels; determining a depth map of the image, the depth map including a plurality of pixels; for each of the pixels in the depth map, computing a surface normal vector that indicates an estimated surface normal of the pixel in the depth map; classifying the pixels in the depth map into clusters based on the surface normal vectors of the pixels in the depth map, each cluster including a subset of the pixels in the depth map; generating patches of connected pixels in the image based on the clusters of the depth map, wherein one of the patches of the image corresponds to one of the clusters of the depth map; rectifying a perspective distortion of each respective patch of the patches based on the surface normal vectors of the pixels of the depth map in the corresponding cluster to generate a rectified patch, wherein rectifying the perspective distortion of a first patch of the patches comprises: determining a first angle difference between an optical axis of a camera that captured the image and a surface normal of the first patch; subsequent to determining the first angle difference is greater than a threshold, identifying a portion of the first patch based on a second angle difference between the optical axis of the camera and a surface normal of the portion of the first patch, wherein the second angle difference is not greater than the threshold; and rectifying a perspective of the identified portion of the first patch without rectifying a remaining portion of the first patch; and extracting perspective corrected features from the rectified patches, the perspective corrected features to be used for feature matching. ### Claim 12 (depends on 11) The non-transitory computer-readable memory of claim 11, wherein determining the depth map of the image comprises: determining the depth map of the image by inputting the image into a model, the model trained to receive images and output depth maps of the received images. ### Claim 13 (depends on 11) The non-transitory computer-readable memory of claim 11, wherein classifying the pixels into the clusters based on the surface normal vectors of the pixels comprises: using k-means clustering to classify the pixels into the clusters based on the surface normal vectors of the pixels. ### Claim 14 (depends on 11) The non-transitory computer-readable memory of claim 11, wherein classifying the pixels into the clusters based on the surface normal vectors of the pixels comprises: classifying the surface normal vectors into three clusters based on the surface normal vectors of the pixels, the three clusters corresponding to three planes orthogonal to each other in a three-dimensional space. ### Claim 15 (depends on 11) The non-transitory computer-readable memory of claim 11, wherein rectifying the perspective distortion of each respective patch of the patches based on the surface normal vectors of the pixels in the corresponding cluster to generate the rectified patch comprises: computing a homography that simulates a virtual camera that has a same center as the camera that captured the image and has an optical axis along an estimated surface normal of the corresponding cluster; and using the homography to perform a projective transformation to the patch to generate the rectified patch. ### Claim 16 (depends on 11) The non-transitory computer-readable memory of claim 11, wherein the threshold is 80 degrees. ### Claim 17 (depends on 11) The non-transitory computer-readable memory of claim 11, wherein extracting the perspective corrected features from the rectified patches comprises: identifying a non-planar portion of the image that is not included in any of the patches; extracting features from the non-planar portion of the image; and extracting features, the features including the perspective corrected features and features extracted from the non-planar portion of the image. ### Claim 18 (depends on 11) The non-transitory computer-readable memory of claim 11, further comprising: matching the perspective corrected features with features of a second image to re-localize the image with respect to the second image. ### Claim 19 A computer system, comprising: a computer processor for executing computer program instructions; and a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations comprising: receiving an image of a scene, the image including pixels; determining a depth map of the image, the depth map including a plurality of pixels; for each of the pixels in the depth map, computing a surface normal vector that indicates an estimated surface normal of the pixel in the depth map; classifying the pixels in the depth map into clusters based on the surface normal vectors of the pixels in the depth map, each cluster including a subset of the pixels in the depth map; generating patches of connected pixels in the image based on the clusters of the depth map, wherein one of the patches of the image corresponds to one of the clusters of the depth map; rectifying a perspective distortion of each respective patch of the patches based on the surface normal vectors of the pixels of the depth map in the corresponding cluster to generate a rectified patch, wherein rectifying the perspective distortion of a first patch of the patches comprises: determining a first angle difference between an optical axis of a camera that captured the image and a surface normal of the first patch; subsequent to determining the first angle difference is greater than a threshold, identifying a portion of the first patch based on a second angle difference between the optical axis of the camera and a surface normal of the portion of the first patch, wherein the second angle difference is not greater than the threshold; and rectifying a perspective of the identified portion of the first patch without rectifying a remaining portion of the first patch; and extracting perspective corrected features from the rectified patches, the perspective corrected features to be used for feature matching. ### Claim 20 (depends on 19) The computer system of claim 19, wherein classifying the pixels into the clusters based on the surface normal vectors of the pixels comprises: classifying the surface normal vectors into three clusters based on the surface normal vectors of the pixels, the three clusters corresponding to three planes orthogonal to each other in a three-dimensional space.