# US 2021/0243362 A1 · Techniques for enhanced image capture using a computer-vision network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0243362 A1
- Document ID: US20210243362A1
- Status: Published application
- Kind: A1
- Application number: 17/163,043
- Filed: 2021-01-29
- Published: 2021-08-05
- Earliest claimed: 2020-01-31
- Assignees: Hover Inc.
- Applicants: Hover Inc.
- Inventors: William Castillo; Brandon Scott; Alrik Firl; David Royston Cutts; Jonathan Mark Igner; Dario Rethage; Domenico Curro
- Web page: https://patentgrep.com/patent/US20210243362A1

## CPC

- H04N 23/61: based on recognised objects
- H04N 23/64: {Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image}
- G06T 7/11: Region-based segmentation
- G06T 7/12: Edge-based segmentation
- G06T 7/174: involving the use of two or more images
- G06T 7/277: involving stochastic approaches, e.g. using Kalman filters
- G06T 7/74: {involving reference images or patches}
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- H04N 23/633: {for displaying additional information relating to control or operation of the camera}
- H04N 23/635: {Region indicators; Field of view indicators}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06T 15/00: Three-dimensional [3D] image rendering
- G06T 2207/20072: Graph-based image processing
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2210/00: Indexing scheme for image generation or computer graphics
- G06V 30/19013: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching (specially adapted for image segmentation G06T7/10; specially adapted for the analysis of motion G06T7/20; specially adapted for image alignment G06T7/30; specially adapted for the calculation of depth from stereo images G06T7/50; specially adapted for position determination G06T7/70)}
- G06V 30/19107: {Clustering techniques}
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Disclosed are techniques for enhancing two-dimensional (2D) image capture of subjects (e.g., a physical structure, such as a residential building) to maximize the feature correspondences available for three-dimensional (3D) model reconstruction. More specifically, disclosed is a computer-vision network configured to provide viewfinder interfaces and analyses to guide the improved capture of an intended subject for specified purposes. Additionally, the computer-vision network can be configured to generate a metric representing a quality of feature correspondences between images of a complete set of images used for reconstructing a 3D model of a physical structure. The computer-vision network can also be configured to generate feedback at or before image capture time to guide improvements to the quality of feature correspondences between a pair of images.

## Claims

### Claim 1

A computer-implemented method, comprising:
capturing a set of pixels representing a scene visible to an image capturing device including a display, the set of pixels including a plurality of border pixels, and each border pixel of the plurality of border pixels being located at or within a defined range of a boundary of the set of pixels;
detecting a physical structure depicted within the set of pixels, the physical structure being represented by a subset of the set of pixels;
generating a segmentation mask associated with the physical structure depicted within the set of pixels, the segmentation mask including one or more segmentation pixels;
determining a pixel value for each border pixel of the plurality of border pixels;
generating an indicator based on the pixel value of one or more border pixels of the plurality of border pixels, and the indicator representing an instruction for framing the physical structure within the display; and
presenting the indicator.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein determining the pixel value for each border pixel further comprises:
detecting that the one or more border pixels of the plurality of border pixels includes a segmentation pixel of the one or more segmentation pixels, and wherein the plurality of border pixels includes:
one or more left edge border pixels located at a left edge of the set of pixels;
one or more or more top edge border pixels located at a top edge of the set of pixels;
one or more right edge border pixels located at a right edge of the set of pixels; and
one or more bottom edge border pixels located at a bottom edge of the set of pixels.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein:
when a left edge border pixel of the one or more left edge border pixels includes a segmentation pixel, the instruction represented by the indicator instructs a user viewing the display to move the image capturing device in a leftward direction;
when a top edge border pixel of the one or more top edge border pixels includes a segmentation pixel, the instruction represented by the indicator instructs the user viewing the display to move the image capturing device in an upward direction;
when a right edge border pixel of the one or more right edge border pixels includes a segmentation pixel, the instruction represented by the indicator instructs the user viewing the display to move the image capturing device in a rightward direction; and
when a bottom edge border pixel of the one or more bottom edge border pixels includes a segmentation pixel, the instruction represented by the indicator instructs the user viewing the display to move the image capturing device in a downward direction.

### Claim 4 (depends on 2)

The computer-implemented method of claim 2, wherein:
when each of a left edge border pixel, a top edge border pixel, a right edge border pixel, and a bottom edge border pixel includes a segmentation pixel, the instruction represented by the indicator instructs a user viewing the display to move backward.

### Claim 5 (depends on 2)

The computer-implemented method of claim 2, wherein when none of the one or more left edge border pixels, the one or more top edge border pixels, the one or more right edge border pixels, and the one or more bottom edge border pixels includes a segmentation pixel, the instruction represented by the indicator instructs a user viewing the display to zoom in to frame the physical structure.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the segmentation mask is a bounding box surrounding the subset of pixels that represent the physical structure.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein presenting the indicator comprises:
displaying the indicator on the display of the image capturing device; or
audibly presenting the indicator to a user operating the image capturing device.

### Claim 8

A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause a data processing apparatus to perform operations including:
capturing a set of pixels representing a scene visible to an image capturing device including a display, the set of pixels including a plurality of border pixels, and each border pixel of the plurality of border pixels being located at or within a defined range of a boundary of the set of pixels;
detecting a physical structure depicted within the set of pixels, the physical structure being represented by a subset of the set of pixels;
generating a segmentation mask associated with the physical structure depicted within the set of pixels, the segmentation mask including one or more segmentation pixels;
determining a pixel value for each border pixel of the plurality of border pixels;
generating an indicator based on the pixel value of one or more border pixels of the plurality of border pixels, and the indicator representing an instruction for framing the physical structure within the display; and
presenting the indicator.

### Claim 9 (depends on 8)

The computer-program product of claim 8, wherein the operation of determining the pixel value for each border pixel further comprises:
detecting that the one or more border pixels of the plurality of border pixels includes a segmentation pixel of the one or more segmentation pixels, and wherein the plurality of border pixels includes:
one or more left edge border pixels located at a left edge of the set of pixels;
one or more or more top edge border pixels located at a top edge of the set of pixels;
one or more right edge border pixels located at a right edge of the set of pixels; and
one or more bottom edge border pixels located at a bottom edge of the set of pixels.

### Claim 10 (depends on 9)

The computer-program product of claim 9, wherein:
when a left edge border pixel of the one or more left edge border pixels includes a segmentation pixel, the instruction represented by the indicator instructs a user viewing the display to move the image capturing device in a leftward direction;
when a top edge border pixel of the one or more top edge border pixels includes a segmentation pixel, the instruction represented by the indicator instructs the user viewing the display to move the image capturing device in an upward direction;
when a right edge border pixel of the one or more right edge border pixels includes a segmentation pixel, the instruction represented by the indicator instructs the user viewing the display to move the image capturing device in a rightward direction; and
when a bottom edge border pixel of the one or more bottom edge border pixels includes a segmentation pixel, the instruction represented by the indicator instructs the user viewing the display to move the image capturing device in a downward direction.

### Claim 11 (depends on 9)

The computer-program product of claim 9, wherein:
when each of a left edge border pixel, a top edge border pixel, a right edge border pixel, and a bottom edge border pixel includes a segmentation pixel, the instruction represented by the indicator instructs a user viewing the display to move backward.

### Claim 12 (depends on 9)

The computer-program product of claim 9, wherein when none of the one or more left edge border pixels, the one or more top edge border pixels, the one or more right edge border pixels, and the one or more bottom edge border pixels includes a segmentation pixel, the instruction represented by the indicator instructs a user viewing the display to zoom in to frame the physical structure.

### Claim 13 (depends on 8)

The computer-program product of claim 8, wherein the segmentation mask is a bounding box surrounding the subset of pixels that represent the physical structure.

### Claim 14 (depends on 8)

The computer-program product of claim 8, wherein presenting the indicator comprises:
displaying the indicator on the display of the image capturing device; or
audibly presenting the indicator to a user operating the image capturing device.

### Claim 15

A computer-implemented method, comprising:
receiving a first set of pixels of a first image frame representing a scene visible to an image capturing device;
detecting a physical structure depicted within the first set of pixels, the physical structure being represented by a subset of the first set of pixels;
generating a first segmentation mask associated with the physical structure depicted within the first set of pixels of the first image frame, the first segmentation mask including one or more first segmentation pixels;
receiving a second set of pixels of a second image frame representing the scene visible to the image capturing device;
detecting the physical structure depicted within the second set of pixels, the physical structure being represented by a subset of the second set of pixels;
generating a second segmentation mask associated with the physical structure depicted within the second set of pixels of the second image frame, the second segmentation mask including one or more second segmentation pixels, and at least one first segmentation pixel being different from at least one second segmentation pixel;
generating an aggregated segmentation mask based on the first segmentation mask and the second segmentation mask, wherein the aggregated segmentation mask is generated to encompass the physical structure depicted in a third image frame captured by the image capturing device; and
generating a feedback signal using the aggregated segmentation mask, the feedback signal corresponding to an instruction to change a position or orientation of the image capturing device to include the physical structure within the third image frame.

### Claim 16 (depends on 15)

The computer-implemented method of claim 15, wherein generating the aggregated segmentation mask further comprises:
aggregating the one or more first segmentation pixels and the one or more second segmentation pixels.

### Claim 17 (depends on 15)

The computer-implemented method of claim 15, wherein generating the aggregated segmentation mask further comprises:
predicting a location of the physical structure in the third image frame using a Kalman filter.

### Claim 18 (depends on 15)

The computer-implemented method of claim 15, further comprising:
detecting that the physical structure depicted in the first set of pixels of the first image frame is occluded by an object, the object separating the depicted physical structure into a first non-contiguous part and a second non-contiguous part;
generating a first partial segmentation mask to encompass the first non-contiguous part of the depicted physical structure;
generating a second partial segmentation mask to encompass the second non-contiguous part of the depicted physical structure; and
selecting one of the first partial segmentation mask and the second partial segmentation mask as the first segmentation mask.

### Claim 19 (depends on 18)

The computer-implemented method of claim 18, wherein selecting the one of the first partial segmentation mask and the second partial segmentation mask further comprises:
selecting a largest of the first segmentation mask and the second partial segmentation mask.

### Claim 20 (depends on 18)

The computer-implemented method of claim 18, wherein selecting the one of the first partial segmentation mask and the second partial segmentation mask further comprises:
selecting one of the first partial segmentation mask and the second partial segmentation mask based on a previous segmentation mask selected in a previous image frame.

### Claim 21 (depends on 18)

The computer-implemented method of claim 18, wherein selecting the one of the first partial segmentation mask and the second partial segmentation mask further comprises:
determining a first location of a first centroid of the first partial segmentation mask;
determining a second location of a second centroid of the second partial segmentation mask; and
selecting one of the first partial segmentation mask and the second partial segmentation mask based on a distance between the first location of the first centroid or the second location of the second centroid and a center of a display of the image capturing device.

### Claim 22

A computer-implemented method, comprising:
capturing a set of two-dimensional (2D) images of a physical structure, each 2D image of the set of 2D images depicting the physical structure from an angle;
generating a three-dimensional (3D) coverage metric for the set of 2D images by:
detecting, for each pair of images within the set of 2D images, one or more feature matches between a first 2D image and a second 2D image of the pair of images, wherein each feature match of the one or more feature matches indicates that a first 3D position associated with a first pixel of the first 2D image matches a second 3D position associated with a second pixel of the second 2D image;
transforming the set of 2D images into a graph based on a result of the detection, the graph including a plurality of nodes and a plurality of edges, each node of the plurality of nodes representing a 2D image of the set of 2D images, and each edge of the plurality of edges connecting two nodes together and representing an existence of at least one feature match between two images associated with the two nodes;
performing a clustering operation on the graph, the clustering operation forming one or more clusters of nodes of the plurality of nodes; and
generating the 3D coverage metric based on a result of performing the clustering operation;
generating, based on the 3D coverage metric, a feedback signal in response to capturing the set of 2D images, the feedback signal indicating an instruction to capture one or more additional 2D images to add to the set of 2D images;
capturing the one or more additional 2D images; and
forming an updated set of 2D images including the set of 2D images and the one or more additional 2D images.

### Claim 23

A computer-implemented method, comprising:
initiating an image capture session using an image capturing device including a display; and
during the image capture session:
capturing a first 2D image of a physical structure from a first pose, the first 2D image including a first pixel associated with a first three-dimensional (3D) position on the physical structure;
capturing a second 2D image depicting the physical structure from a second pose, the second 2D image including a second pixel associated with a second 3D position on the physical structure, and the first pose being different from the second pose;
detecting one or more feature matches between the first 2D image and the second 2D image, wherein a feature match of the one or more feature matches indicates that the first 3D position associated with the first pixel of the first 2D image matches the second 3D position associated with the second pixel of the second 2D image; and
determining a 3D reconstruction condition based on the one or more feature matches between the first 2D image and the second 2D image.