# US 10,769,441 B2 · Cluster based photo navigation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,769,441 B2
- Document ID: US10769441B2
- Status: Granted
- Kind: B2
- Application number: 15/651,573
- Filed: 2017-07-17
- Published: 2018-01-04
- Granted: 2020-09-08
- Earliest claimed: 2017-07-17
- Assignees: Google Inc.
- Applicants: Google Inc.
- Inventors: Guiheng Zhou; Liyong Chen; Hui Lou; Junjie Ke; Hao Chen; Deben Kong; David Robert Gallup
- Web page: https://patentgrep.com/patent/US10769441B2

## CPC

- G06V 20/30: in albums, collections or shared content, e.g. social network photos or video
- G06F 18/2321: using statistics or function optimisation, e.g. modelling of probability density functions
- G06F 18/2413: based on distances to training or reference patterns
- G06V 10/763: {Non-hierarchical techniques, e.g. based on statistics of modelling distributions}
- G06V 10/764: using classification, e.g. of video objects
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/0485: Scrolling or panning
- G06V 10/16: {using multiple overlapping images; Image stitching}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The technology relates to navigating imagery that is organized into clusters based on common patterns exhibited when imagery is captured. For example, a set of captured images which satisfy a predetermined pattern may be determined. The images in the set of set of captured images may be grouped into one or more clusters according to the predetermined pattern. A request to display a first cluster of the one or more clusters may be received and, in response, a first captured image from the requested first cluster may be selected. The selected first captured image may then be displayed.

## Claims

### Claim 1

A method for organizing and navigating image clusters comprising:
accessing, by one or more processing devices, a set of captured images;
detecting, by the one or more processing devices, whether images within the set of captured images satisfy a predetermined pattern;
grouping, by the one or more processing devices, the images in the set of captured images into one or more clusters according to the detected predetermined pattern;
receiving, by the one or more processing devices, a request to display a first cluster of the one or more clusters of captured images;
determining, by the one or more processing devices, from the images within the first cluster, a set of neighboring captured images that are within a predetermined proximity to the first captured image;
assigning, by the one or more processing devices, one or more neighboring images of the first captured image from the set of neighboring captured images;
selecting, by the one or more processing devices in response to the request, a first captured image from the first cluster to display; and
providing the first captured image from the first cluster for display.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
providing, by the one or more processing devices in response to a click or drag event, the one or more neighboring images.

### Claim 3 (depends on 1)

The method of claim 1, wherein the predetermined pattern is a panoramic pattern, and wherein detecting whether the images within the set of captured images satisfies the predetermined pattern comprises:
identifying, by the one or more processing devices, each possible panoramic circle in the set of captured images, wherein each possible panoramic center is associated with a subset of the set of captured images and has a view angle for the subset of captured images; and
removing any of the possible panoramic circles in which (i) a number of the subset of captured images is fewer than a predetermined number, and (ii) the view angle is less than a predetermined angle.

### Claim 4 (depends on 3)

The method of claim 3, wherein identifying each possible panoramic circle includes:
setting an initial panoramic circle around a coordinate for one of the captured images;
revising the initial panoramic circle center coordinate by averaging coordinates from captured images within the initial panoramic circle;
repeating the revising until a threshold number of revisions has occurred or the panoramic circle center coordinate converges to a single point; and
setting a final panoramic circle around the revised panoramic circle center coordinate.

### Claim 5 (depends on 1)

The method of claim 1, wherein the predetermined pattern is an orbit pattern, and wherein detecting whether the images within the set of captured images satisfies the predetermined pattern comprises:
identifying, by the one or more processing devices, for each image in the set of captured images, a ray emanating from the location the image was captured, and in the direction a camera center was pointed when the captured image was taken;
calculating, by the one or more processing devices, intersection points between each ray;
determining, by the one or more processing devices orbit centers based on densities of the intersection points.

### Claim 6 (depends on 5)

The method of claim 5, wherein the method further comprises:
for each identified orbit center:
filtering out images that do not include the orbit center; and
dividing the remaining images into groups in which the remaining images have smooth translation around the orbit center; and
removing groups which do not include at least a predetermined number of images and do not satisfy a threshold angle of coverage.

### Claim 7 (depends on 1)

The method of claim 1, wherein the predetermined pattern is a translation pattern, and wherein detecting whether images within the set of captured images satisfies the predetermined pattern comprises:
identifying, by the one or more processing devices, for each image in the set of captured images, an adjacent initial set of candidate images;
reserving, by the one or more processing devices, for each image, the candidate images in the initial set of candidate images which are located closest to the image, and are within an angle threshold of a viewing angle of the image.

### Claim 8 (depends on 1)

The method of claim 1, wherein the predetermined pattern is a panoramic pattern, and wherein assigning the one or more neighboring images of the first captured image from the set of neighboring captured images for a panoramic pattern includes determining the one or more neighboring images which have the lowest drag target cost, wherein the drag target cost is calculated by:
determining a drag vector direction;
reserving a group of the one or more neighboring images which are in the direction of the drag vector direction;
identifying, by the one or more processing device, an amount of overlap between the first captured image and each of the one or more neighboring images in the group of one or more neighboring images; and
selecting the neighboring images with the highest amount of overlap.

### Claim 9 (depends on 1)

The method of claim 1, wherein the predetermined pattern is an orbit or translation pattern, wherein assigning the one or more neighboring images of the first captured image from the set of neighboring captured images for an orbit and translation pattern includes determining the one or more neighboring images having a target distance closest to a predetermined target distance, and wherein the target distance for each captured image in the set of neighboring images is determined by:
identifying, by the one or more processing devices, a ray emanating from a location from which the image was captured and in a direction the camera was pointed;
determining, by the one or more processing devices, an angle difference between the first captured angle and each captured image in the set of neighboring images;
grouping, by the one or more processing devices, each captured image in the set of neighboring images to the left and right of first captured image based on the angle difference; and
determining, by the one or more processing devices, a target distance.

### Claim 10

A system for organizing and navigating image clusters comprising:
one or more computing devices; and
memory storing instructions, the instructions executable by the one or more computing devices;
wherein the instructions comprise:
accessing, by one or more computing devices, a set of captured images;
detecting, by the one or more computing devices, whether images within the set of captured images satisfy a predetermined pattern, wherein the predetermined pattern is one or more of a translation pattern corresponding to a set of images captured along a linear direction, an orbit pattern corresponding to a set of images captured around a location, or a panoramic pattern corresponding to a set of images captured in a rotational motion around an axis;
grouping, by the one or more computing devices, the images in the set of captured images into one or more clusters according to the detected predetermined pattern;
receiving, by the one or more computing devices, a request to display a first cluster of the one or more clusters of captured images;
determining, by the one or more processing devices, from the images within the first cluster, a set of neighboring captured images that are within a predetermined proximity to the first captured image;
assigning, by the one or more processing devices, one or more neighboring images of the first captured image from the set of neighboring captured images;
selecting, by the one or more computing devices in response to the request, a first captured image from the first cluster to display; and
providing the first captured image from the first cluster for display.

### Claim 11 (depends on 10)

The system of claim 10, further comprising:
providing, by the one or more processing devices in response to a click or drag event, the one or more neighboring images.

### Claim 12 (depends on 10)

The system of claim 10, wherein the predetermined pattern is a panoramic pattern, and wherein detecting whether the images within the set of captured images satisfies the predetermined pattern comprises:
identifying, by the one or more processing devices, each possible panoramic circle in the set of captured images, wherein each possible panoramic center is associated with a subset of the set of captured images and has a view angle for the subset of captured images; and
removing any of the possible panoramic circles in which (i) a number of the subset of captured images is fewer than a predetermined number, and (ii) the view angle is less than a predetermined angle.

### Claim 13 (depends on 12)

The system of claim 12, wherein identifying each possible panoramic circle includes:
setting an initial panoramic circle around a coordinate for one of the captured images;
revising the initial panoramic circle center coordinate by averaging coordinates from captured images within the initial panoramic circle;
repeating the revising until a threshold number of revisions has occurred or the panoramic circle center coordinate converges to a single point; and
setting a final panoramic circle around the revised panoramic circle center coordinate.

### Claim 14 (depends on 10)

The system of claim 10, wherein the predetermined pattern is an orbit pattern, and wherein detecting whether the images within the set of captured images satisfies the predetermined pattern comprises:
identifying, by the one or more processing devices, for each image in the set of captured images, a ray emanating from the location the image was captured, and in the direction a camera center was pointed when the captured image was taken;
calculating, by the one or more processing devices, intersection points between each ray;
determining, by the one or more processing devices orbit centers based on densities of the intersection points.

### Claim 15 (depends on 14)

The system of claim 14, wherein the system further comprises:
for each identified orbit center:
filtering out images that do not include the orbit center; and
dividing the remaining images into groups in which the remaining images have smooth translation around the orbit center; and
removing groups which do not include at least a predetermined number of images and do not satisfy a threshold angle of coverage.

### Claim 16 (depends on 10)

The system of claim 10, wherein the predetermined pattern is a translation pattern, and wherein detecting whether images within the set of captured images satisfies the predetermined pattern comprises:
identifying, by the one or more processing devices, for each image in the set of captured images, an adjacent initial set of candidate images;
reserving, by the one or more processing devices, for each image, the candidate images in the initial set of candidate images which are located closest to the image, and are within an angle threshold of a viewing angle of the image.

### Claim 17

A system for organizing and navigating image clusters comprising:
one or more computing devices; and
memory storing instructions, the instructions executable by the one or more computing devices;
wherein the instructions comprise:
accessing, by one or more computing devices, a set of captured images;
detecting, by the one or more computing devices, whether images within the set of captured images satisfy a predetermined pattern;
grouping, by the one or more computing devices, the images in the set of captured images into one or more clusters according to the detected predetermined pattern;
receiving, by the one or more computing devices, a request to display a first cluster of the one or more clusters of captured images;
selecting, by the one or more computing devices in response to the request, a first captured image from the first cluster to display;
providing the first captured image from the first cluster for display;
determining, by the one or more computing devices, from the images within the first cluster, a set of neighboring captured images that are within a predetermined proximity to the first captured image;
assigning, by the one or more computing devices, one or more neighboring images of the first captured image from the set of neighboring captured images; and
providing, by the one or more computing devices in response to a click or drag event, the one or more neighboring images.

### Claim 18 (depends on 17)

The system of claim 17, wherein the predetermined pattern is one of a panoramic pattern, an orbit pattern, and a translation pattern.