# US 11,030,235 B2 · Method for navigating through a set of images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,030,235 B2 - Document ID: US11030235B2 - Status: Granted - Kind: B2 - Application number: 16/066,719 - Filed: 2016-12-27 - Published: 2019-01-17 - Granted: 2021-06-08 - Earliest claimed: 2016-01-04 - Assignees: Facebook, Inc. - Applicants: Facebook, Inc. - Inventors: Jan Erik Solem; Yubin Kuang; Johan Gyllenspetz; Peter Neubauer; Pau Gargallo Piracés - Web page: https://patentgrep.com/patent/US11030235B2 ## CPC - G06F 16/587: using geographical or spatial information, e.g. location - G01C 21/3647: {Guidance involving output of stored or live camera images or video streams} - G06F 16/24578: {using ranking} - G06F 16/29: Geographical information databases - G06F 16/51: Indexing; Data structures therefor; Storage structures - G06F 16/54: Browsing; Visualisation therefor - G06F 16/5838: {using colour} - G06F 16/5866: {using information manually generated, e.g. tags, keywords, comments, manually generated location and time information} - G06F 16/9537: Spatial or temporal dependent retrieval, e.g. spatiotemporal queries - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) ## Abstract The present invention relates to a method for navigating through a set of images depicting a scenery, said method comprising: receiving information relating to a subset of images within said set of images, said information comprising metadata of each image, said subset of images comprising images being acquired from a geographical position, which is within a bounding area surrounding the geographical position of a current image within the set of images; receiving weight information, said weight information controlling weights to be given to a plurality of different measures of similarities based on different types of metadata; and calculating a plurality of combined measures of similarity for pairs of images in the subset of images, said calculating being performed using said metadata and said weight information. ## Claims ### Claim 1 A method for navigating through a set of images depicting a scenery, said method comprising: receiving information relating to a subset of images within said set of images, said information comprising metadata of each image within said subset of images, said metadata including a geographical position of an image capturing sensor when acquiring the corresponding image, a viewing angle of the image capturing sensor when acquiring the corresponding image, and image property information, wherein said subset of images is selected from the set of images based upon each image in the subset of images being acquired from a geographical position within a bounding area defined in relation to a start geographical position of navigation and a target geographical position of navigation; receiving weight information for determining similarities between images within said subset of images, said weight information controlling weights to be provided for determining a plurality of different measures of similarities between images within said subset of images based on different types of metadata; calculating a plurality of combined measures of similarity between pairs of images in the subset of images, each combined measure of similarity representing a similarity between the images in a pair of images within said subset of images, said calculating being performed using said metadata and said weight information; generating a graph representation, wherein at least one node in the graph representation is associated with an image included in the subset of images and at least one edge in the graph representation is associated with a transition between two images included in the subset of images, and said generating the graph representation is associated with said calculating of the plurality of combined measures of similarity; receiving an indication of a target geographical position associated with a distance in a direction relative to a viewing angle of a current image; and determining a best ranked path from a first image in the subset of images to a second image in the subset of images, wherein the first image has a geographical position within a distance from the start geographical position that is smaller than a set threshold, the second image has a geographical position within a distance from the target geographical position that is smaller than a set threshold, and the path comprises a number of transitions between images from the first image via intermediate images to the second image; said determining comprising: calculating a transition measure of a path from the first image to the second image as a sum of the combined measures of similarity between pairs of images included in the transitions and associated with a starting node representing the first image and a target node representing the second image in the graph representation; and selecting the best ranked path based on a comparison of the transition measures for paths from the first image to the second image. ### Claim 2 (depends on 1) The method of claim 1, further comprising determining at least one best ranked transition image among the images within said subset of images based on said plurality of combined measures of similarity. ### Claim 3 (depends on 1) The method of claim 1, further comprising classifying a transition between two images within said set of images into a class among a predefined, finite number of transition classes, wherein each predefined transition class defines a type of transition between the two images according to a difference in the geographical position and a difference in the viewing angle of the two images. ### Claim 4 (depends on 1) The method of claim 1, wherein said metadata includes a three-dimensional reconstruction index indicating that the image is part of a particular sub-model of three-dimensional structure of objects in the scenery having the three-dimensional reconstruction index. ### Claim 5 (depends on 1) The method of claim 1, wherein the bounding area is updated based on navigation through the set of images. ### Claim 6 (depends on 1) The method of claim 1, wherein the bounding area is determined based on the viewing angle of the current image. ### Claim 7 (depends on 1) The method of claim 1, further comprising sending a request for information relating to the subset of images, wherein said request comprises an indication of the current image. ### Claim 8 A system for navigating through a set of images depicting a scenery, said system comprising: a processor configured to: receive information relating to a subset of images within said set of images, said information comprising metadata of each image within said subset of images, said metadata including a geographical position of an image capturing sensor when acquiring the corresponding image, a viewing angle of the image capturing sensor when acquiring the corresponding image, and image property information, wherein said subset of images is selected from the set of images based upon each image in the subset of images being acquired from a geographical position within a bounding area defined in relation to a start geographical position of navigation and a target geographical position of navigation; receive weight information for determining similarities between images within said subset of images, said weight information controlling weights to be provided for determining a plurality of different measures of similarities between images within said subset of images based on different types of metadata; calculate a plurality of combined measures of similarity between pairs of images in the subset of images, each combined measure of similarity representing a similarity between the images in a pair of images within said subset of images, said calculating being performed using said metadata and said weight information; generate a graph representation, wherein at least one node in the graph representation is associated with an image included in the subset of images and at least one edge in the graph representation is associated with a transition between two images included in the subset of images, and said generating the graph representation is associated with said calculating of the plurality of combined measures of similarity; receive an indication of a target geographical position associated with a distance in a direction relative to a viewing angle of a current image; and determine a best ranked path from a first image in the subset of images to a second image in the subset of images, wherein the first image has a geographical position within a distance from the start geographical position that is smaller than a set threshold, the second image has a geographical position within a distance from the target geographical position that is smaller than a set threshold, and the path comprises a number of transitions between images from the first image via intermediate images to the second image, and wherein the processor is configured to determine said best ranked path including by being configured to: calculate a transition measure of a path from the first image to the second image as a sum of the combined measures of similarity between pairs of images included in the transitions and associated with a starting node representing the first image and a target node representing the second image in the graph representation, and select the best ranked path based on a comparison of the transition measures for paths from the first image to the second image; and a memory coupled to the processor and configured to provide the processor with instructions. ### Claim 9 (depends on 8) The system of claim 8, wherein the processor is further configured to determine at least one best ranked transition image among the images within said subset of images based on said plurality of combined measures of similarity. ### Claim 10 (depends on 8) The system of claim 8, wherein the processor is further configured to classify a transition between two images within said set of images into a class among a predefined, finite number of transition classes, wherein each predefined transition class defines a type of transition between the two images according to a difference in the geographical position and a difference in the viewing angle of the two images. ### Claim 11 (depends on 8) The system of claim 8, wherein said metadata includes a three-dimensional reconstruction index indicating that the image is part of a particular sub-model of three-dimensional structure of objects in the scenery having the three-dimensional reconstruction index. ### Claim 12 (depends on 8) The system of claim 8, wherein the bounding area is updated based on navigation through the set of images. ### Claim 13 (depends on 8) The system of claim 8, wherein the bounding area is determined based on the viewing angle of the current image. ### Claim 14 (depends on 8) The system of claim 8, wherein the processor is further configured to send a request for information relating to the subset of images, wherein said request comprises an indication of the current image. ### Claim 15 A computer program product for navigating through a set of images depicting a scenery, embodied in a non-transitory computer readable medium and comprising computer instructions for: receiving information relating to a subset of images within said set of images, said information comprising metadata of each image within said subset of images, said metadata including a geographical position of an image capturing sensor when acquiring the corresponding image, a viewing angle of the image capturing sensor when acquiring the corresponding image, and image property information, wherein said subset of images is selected from the set of images based upon each image in the subset of images being acquired from a geographical position within a bounding area defined in relation to a start geographical position of navigation and a target geographical position of navigation; receiving weight information for determining similarities between images within said subset of images, said weight information controlling weights to be provided for determining a plurality of different measures of similarities between images within said subset of images based on different types of metadata; calculating a plurality of combined measures of similarity between pairs of images in the subset of images, each combined measure of similarity representing a similarity between the images in a pair of images within said subset of images, said calculating being performed using said metadata and said weight information; generating a graph representation, wherein at least one node in the graph representation is associated with an image included in the subset of images and at least one edge in the graph representation is associated with a transition between two images included in the subset of images, and said generating the graph representation is associated with said calculating of the plurality of combined measures of similarity; receiving an indication of a target geographical position associated with a distance in a direction relative to a viewing angle of a current image; and determining a best ranked path from a first image in the subset of images to a second image in the subset of images, wherein the first image has a geographical position within a distance from the start geographical position that is smaller than a set threshold, the second image has a geographical position within a distance from the target geographical position that is smaller than a set threshold, and the path comprises a number of transitions between images from the first image via intermediate images to the second image; said determining comprising: calculating a transition measure of a path from the first image to the second image as a sum of the combined measures of similarity between pairs of images included in the transitions and associated with a starting node representing the first image and a target node representing the second image in the graph representation; and selecting the best ranked path based on a comparison of the transition measures for paths from the first image to the second image. ### Claim 16 (depends on 15) The computer program product of claim 15, further comprising computer instructions for determining at least one best ranked transition image among the images within said subset of images based on said plurality of combined measures of similarity. ### Claim 17 (depends on 15) The computer program product of claim 15, further comprising computer instructions for classifying a transition between two images within said set of images into a class among a predefined, finite number of transition classes, wherein each predefined transition class defines a type of transition between the two images according to a difference in the geographical position and a difference in the viewing angle of the two images. ### Claim 18 (depends on 15) The computer program product of claim 15, wherein said metadata includes a three-dimensional reconstruction index indicating that the image is part of a particular sub-model of three-dimensional structure of objects in the scenery having the three-dimensional reconstruction index. ### Claim 19 (depends on 15) The computer program product of claim 15, wherein the bounding area is determined based on the viewing angle of the current image and updated based on navigation through the set of images. ### Claim 20 (depends on 15) The computer program product of claim 15, further comprising computer instructions for sending a request for information relating to the subset of images, wherein said request comprises an indication of the current image.