# US 11,308,714 B1 · Artificial intelligence system for identifying and assessing attributes of a property shown in aerial imagery

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,714 B1
- Document ID: US11308714B1
- Status: Granted
- Kind: B1
- Application number: 16/546,918
- Filed: 2019-08-21
- Granted: 2022-04-19
- Earliest claimed: 2018-08-23
- Applicants: Athenium LLC
- Inventors: Christos Marios Christoudias; Ellen Dee Cousins; Ali Alhaj Darwish
- Web page: https://patentgrep.com/patent/US11308714B1

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06V 20/176: {Urban or other man-made structures}
- G06F 17/15: Correlation function computation {including computation of convolution operations (arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00)}
- G06F 18/24: Classification techniques
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06V 10/82: using neural networks
- G06N 3/048: Activation functions
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/00: Machine learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A computer-based method includes receiving, at a computer-based system, an aerial image of a property that includes a first visual indicator on the aerial image that follows and identifies a boundary line for the property; using a building rooftop Deep Fully Convolutional Network (DFCN), configured and trained to predict the presence of building rooftops in aerial imagery, to predict whether any building rooftops are present within the boundary line of the property based on the aerial image; and applying a second visual indicator to the aerial image to identify and outline a building rooftop in the aerial image identified by the building rooftop deep fully convolutional network. In some implementations, other Convolutional Networks (ConvNets) are used to predict other property attributes and characteristics.

## Claims

### Claim 1

A computer-based method comprising:
receiving, at a computer-based system, an aerial image of a property that includes a first visual indicator on the aerial image that follows and identifies a boundary line for the property;
using a building rooftop deep fully convolutional network, configured and trained to predict the presence of building rooftops in aerial imagery, to predict whether any building rooftops are present within the boundary line of the property based on the aerial image;
applying a second visual indicator to the aerial image to identify and outline a building rooftop in the aerial image identified by the building rooftop deep fully convolutional network; and
using an additional deep fully convolutional network, configured and trained to predict the presence of an additional property feature in aerial imagery, to predict whether said additional property feature is present with the boundary line of the property based on the aerial image,
wherein the building rooftop and additional deep convolutional networks have identical configurations but use different weights.

### Claim 2 (depends on 1)

The computer-based method of claim 1, further comprising:
using a skylights deep fully convolutional network, configured and trained to predict the presence of skylights on building rooftops in aerial imagery, to predict whether any skylights are present on the building rooftop that was identified by the second visual indicator;
using a solar panels deep fully convolutional network, configured and trained to predict the presence of solar panels on building rooftops in aerial imagery, to predict whether any solar panels are present on the building rooftop that was identified by the second visual indicator; and/or
using a vegetation overhang deep fully convolutional network, configured and trained to predict the presence of vegetation overhanging building rooftops in aerial imagery, to predict whether any vegetation is overhanging the building rooftop that was identified by the second visual indicator.

### Claim 3 (depends on 2)

The computer-based method of claim 2, further comprising:
applying a third visual indicator to the aerial image to identify and outline skylight(s) identified by the skylights deep fully convolutional network on the building rooftop;
applying a fourth visual indicator to the aerial image to identify and outline solar panel(s) identified by the solar panels deep fully convolutional network on the building rooftop; and/or
applying a fifth visual indicator to the aerial image to identify and outline vegetation overhang identified by the vegetation overhang deep fully convolutional network on the building rooftop.

### Claim 4 (depends on 3)

The computer-based method of claim 3, wherein the skylights deep fully convolutional network, the solar panels deep fully convolutional network, and the vegetation overhang deep fully convolutional network have identical configurations but use different weights.

### Claim 5 (depends on 4)

The computer-based method of claim 4, further comprising:
providing each of the skylights deep fully convolutional network, solar panels deep fully convolutional network, and vegetation overhang deep fully convolutional network,
wherein each of the skylights deep fully convolutional network, solar panels deep fully convolutional network, and vegetation overhang deep fully convolutional network comprises:
an encoder,
a decoder, and
a plurality of skip connections that extend from various points in the encoder to the decoder,
wherein the encoder comprises:
an input,
a normalization layer, and
ten processing segments; and
wherein the decoder comprises:
eight processing segments; and
an output,
wherein every processing segment in the encoder and decoder comprises:
a convolution layer,
a batch normalization layer; and
an exponential linear unit (elu) layer, and
wherein every other processing segment in the encoder, starting with a second processing segment in the encoder, has a max pooling layer.

### Claim 6 (depends on 5)

The computer-based method of claim 5, wherein:
each of the first two convolution layers in the encoder is associated with a feature vector that represents 32 features,
each of the next two convolution layers in the encoder is associated with a feature vector that represents 64 features,
each of the next two convolution layers in the encoder is associated with a feature vector that represents 128 features,
each of the next two convolution layers in the encoder is associated with a feature vector that represents 256 features,
each of the next two convolution layers in the encoder is associated with a feature vector that represents 512 features, and
wherein:
each of the first two convolution layers in the decoder is associated with a feature vector that represents 256 features,
each of the next two convolution layers in the decoder is associated with a feature vector that represents 128 features,
each of the next two convolution layers in the decoder is associated with a feature vector that represents 64 features, and
the last two dilated convolution layers in the decoder are associated with a feature vector that represents 32 features, and
wherein:
a first of the skip connections extends from a point in the encoder just before a first of the max pooling layers to an output of a final processing segment in the decoder,
a second of the skip connections extends from a point in the encoder just before a second of the max pooling layers to a point in the decoder just before a last two of the processing segments in the decoder,
a third of the skip connections extends from a point in the encoder just before a third of the max pooling layers to a point in the decoder just before a last four of the processing segments in the decoder, and
a fourth of the skip connections extends from a point in the encoder just before a last of the max pooling layers to a point in the decoder just after a first two of the processing segments in the decoder.

### Claim 7 (depends on 1)

The computer-based method of claim 1, further comprising applying a version of the aerial image to one or more of a plurality of image classifiers, wherein each image classifier is configured to classify one or more attributes in the aerial image.

### Claim 8 (depends on 1)

The computer-based method of claim 1, further comprising:
classifying, with a vegetation overhang classifier, a degree to which vegetation in the aerial image is overhanging the building rooftop identified by the building rooftop deep fully convolutional network;
classifying, with a roof condition classifier, a condition of the building rooftop identified by the building rooftop deep full convolutional network;
classifying, with a roof material classifier, a material of the building rooftop identified by the building rooftop deep fully convolutional network; and/or
classifying, with a roof type classifier, a roof type for the building rooftop identified by the building rooftop deep fully convolutional network.

### Claim 9 (depends on 8)

The computer-based method of claim 8, wherein classifying the vegetation overhang comprises:
calculating an area of the building rooftop identified and outlined by the second visual indicator in the aerial image;
identifying, with a vegetation overhang deep fully convolutional network, vegetation overhanging the building rooftop identified and outlined by the second visual indicator in the aerial image;
calculating an area of the vegetation overhang identified by the vegetation overhang deep fully convolutional network;
calculating a percentage of the building rooftop covered by the vegetation overhang based on the calculated building rooftop area and the vegetation overhang area; and
applying the calculated percentage to decision tree logic to determine the degree to which vegetation in the aerial image is overhanging the building rooftop identified by the building rooftop deep fully convolutional network.

### Claim 10 (depends on 8)

The computer-based method of claim 8, wherein classifying the roof condition comprises:
using a building rooftop staining deep convolutional neural network, configured and trained to predict whether staining is present on building rooftops in aerial imagery, to identify a probability that staining is present on the building rooftop identified and outlined by the second visual indicator in the aerial image;
using a building rooftop surface deterioration deep convolutional neural network, configured and trained to predict whether surface deterioration is present on building rooftops in aerial imagery, to identify that surface deterioration is present on the building rooftop identified and outlined by the second visual indicator in the aerial image;
using the probability that staining is present on the building rooftop identified and outlined by the second visual indicator in the aerial image, and the probability that surface deterioration is present on the building rooftop identified and outlined by the second visual indicator in the aerial image to calculate a disrepair index for the building rooftop; and
applying the calculated disrepair index to decision tree logic to determine the condition of the building rooftop identified by the building rooftop deep fully convolutional network.

### Claim 11 (depends on 8)

The computer-based method of claim 8, wherein classifying the building rooftop material comprises:
using building rooftop material deep convolutional neural networks, configured and trained to identify probabilities that the building rooftop has different types of roof material from aerial imagery, to identify probabilities that the building rooftop includes a plurality of different materials, respectively; and
selecting, with a computer-based selector, a top one or two, highest probability roof materials.

### Claim 12 (depends on 8)

The computer-based method of claim 8, wherein classifying the building roof type comprises:
using building rooftop type deep convolutional neural networks, configured and trained to identify probabilities that the building rooftop has different possible types from aerial imagery, to identify probabilities that the building rooftop has different types, respectively; and
selecting, with a computer-based selector, a top one or two, highest probability roof types.

### Claim 13 (depends on 1)

The computer-based method of claim 1, further comprising:
using an above-ground pool deep fully convolutional network, configured and trained to predict the presence of above-ground pools in aerial imagery, to predict whether any above-ground pools are present within the boundary line of the property based on the aerial image; and
applying a third visual indicator to the aerial image to identify and outline an above-ground pool in the aerial image identified by the above-ground pool deep fully convolutional network.

### Claim 14 (depends on 13)

The computer-based method of claim 13, further comprising:
using an in-ground pool deep fully convolutional network, configured and trained to predict the presence of in-ground pools in aerial imagery, to predict whether any in-ground pools are present within the boundary line of the property based on the aerial image; and
applying a fourth visual indicator to the aerial image to identify and outline an in-ground pool in the aerial image identified by the in-ground pool deep fully convolutional network.

### Claim 15 (depends on 14)

The computer-based method of claim 14, further comprising:
using a trampoline deep fully convolutional network, configured and trained to predict the presence of trampolines in aerial imagery, to predict whether any trampolines are present within the boundary line of the property based on the aerial image; and
applying a fifth visual indicator to the aerial image to identify and outline a trampoline in the aerial image identified by the trampoline deep fully convolutional network.

### Claim 16 (depends on 15)

The computer-based method of claim 15, wherein the above-ground pool deep fully convolutional network, the in-ground pool deep fully convolutional network, and the trampoline deep fully convolutional network have identical configurations but use different weights.

### Claim 17

A computer-based method comprising:
receiving, at a computer-based system, an aerial image of a property that includes a first visual indicator on the aerial image that follows and identifies a boundary line for the property;
using a building rooftop deep fully convolutional network, configured and trained to predict the presence of building rooftops in aerial imagery, to predict whether any building rooftops are present within the boundary line of the property based on the aerial image;
applying a second visual indicator to the aerial image to identify and outline a building rooftop in the aerial image identified by the building rooftop deep fully convolutional network,
providing an above-ground pool deep fully convolutional network, an in-ground pool deep fully convolutional network, and a trampoline deep fully convolutional network,
wherein each of the above-ground pool deep fully convolutional network, the in-ground pool deep fully convolutional network, and the trampoline deep fully convolutional network comprises:
an encoder,
a decoder, and
a plurality of skip connections that extend from various points in the encoder to the decoder,
wherein the encoder comprises:
an input,
a normalization layer, and
ten processing segments; and
wherein the decoder comprises:
eight processing segments; and
an output,
wherein every processing segment in the encoder and decoder comprises:
a convolution layer,
a batch normalization layer; and
an exponential linear unit (elu) layer, and
wherein every other processing segment in the encoder, starting with a second processing segment in the encoder, has a max pooling layer.

### Claim 18 (depends on 17)

The computer-based method of claim 17, wherein:
each of the first two convolution layers in the encoder is associated with a feature vector that represents 16 features,
each of the next two convolution layers in the encoder is associated with a feature vector that represents 32 features,
each of the next two convolution layers in the encoder is associated with a feature vector that represents 64 features,
each of the next two convolution layers in the encoder is associated with a feature vector that represents 128 features,
each of the next two convolution layers in the encoder is associated with a feature vector that represents 256 features, and
wherein:
each of the first two convolution layers in the decoder is associated with a feature vector that represents 128 features,
each of the next two convolution layers in the decoder is associated with a feature vector that represents 64 features,
each of the next two convolution layers in the decoder is associated with a feature vector that represents 32 features, and
the last two dilated convolution layers in the decoder are associated with a feature vector that represents 16 features, and
wherein:
a first of the skip connections extends from a point in the encoder just before a first of the max pooling layers to an output of a final processing segment in the decoder,
a second of the skip connections extends from a point in the encoder just before a second of the max pooling layers to a point in the decoder just before a last two of the processing segments in the decoder,
a third of the skip connections extends from a point in the encoder just before a third of the max pooling layers to a point in the decoder just before a last four of the processing segments in the decoder, and
a fourth of the skip connections extends from a point in the encoder just before a last of the max pooling layers to a point in the decoder just after a first two of the processing segments in the decoder.

### Claim 19 (depends on 1)

The computer-based method of claim 1, further comprising:
enabling a first human annotator to annotate the received image, at a computer-based user terminal, to specify whether visual inspection by the first human annotator revealed, in the aerial image, any building rooftops, in-ground pools, above-ground pools, trampolines, skylights, solar panels, vegetation overhang, staining, surface deterioration, specific roof materials, and/or specific roof types; and
comparing the annotation(s) made by the first human annotator to any determinations or predictions made by the computer-based system.

### Claim 20 (depends on 19)

The computer-based method of claim 19, further comprising:
if the comparison reveals a discrepancy between the annotations made by the first human annotator and the determinations or predictions made by the computer-based system, presenting the aerial image to a second human annotator who is different than the first human annotator.

### Claim 21 (depends on 1)

The computer-based method of claim 1, further comprising:
masking and resizing the aerial image prior to using the building rooftop deep fully convolutional network to predict whether any building rooftops are present within the boundary line of the property based on the aerial image.

### Claim 22 (depends on 21)

The computer-based method of claim 21, wherein masking and resizing is performed for all property and rooftop segmentation deep fully convolutional network models.

### Claim 23 (depends on 1)

The computer-based method of claim 1, wherein using the building rooftop deep fully convolutional network produces a probability image version of the aerial image, wherein the probability image is a pixel-by-pixel representation of whether the aerial image shows a building rooftop.

### Claim 24 (depends on 23)

The computer-based method of claim 23, wherein all property and rooftop segmentation deep fully convolutional network models produce a probability image version of the areal image.

### Claim 25 (depends on 23)

The computer-based method of claim 23, further comprising:
thresholding and resizing the probability image; and
applying connected components analysis and an area filter to produce a final output that is a version of the aerial image that includes the second visual indicator identifying and outlining the building rooftop in the aerial image.

### Claim 26 (depends on 25)

The computer-based method of claim 25, wherein all property and rooftop segmentation deep fully convolutional network models threshold and resize the probability image and apply connection components and area filtering to produce a final output.

### Claim 27

A computer-based method comprising:
receiving, at a computer-based system, an aerial image of a property that includes a first visual indicator on the aerial image that follows and identifies a boundary line for the property;
using a building rooftop deep fully convolutional network, configured and trained to predict the presence of building rooftops in aerial imagery, to predict whether any building rooftops are present within the boundary line of the property based on the aerial image;
applying a second visual indicator to the aerial image to identify and outline a building rooftop in the aerial image identified by the building rooftop deep fully convolutional network,
wherein the building rooftop deep fully convolutional network comprises:
an encoder,
a decoder, and
a plurality of skip connections that extend from various points in the encoder to the decoder,
wherein the encoder comprises:
an input,
a normalization layer, and
ten processing segments; and
wherein the decoder comprises:
eight processing segments; and
an output,
wherein every processing segment in the encoder and decoder comprises:
a dilated convolution layer,
a batch normalization layer; and
an exponential linear unit (elu) layer, and
wherein every other processing segment in the encoder, starting with a second processing segment in the encoder, has a max pooling layer.

### Claim 28 (depends on 27)

The computer-based method of claim 27, wherein:
each of the first two dilated convolution layers in the encoder is associated with a feature vector that represents 64 features and a 2×2 dilation kernel,
each of the next two dilated convolution layers in the encoder is associated with a feature vector that represents 128 features and a 4×4 dilation kernel,
each of the next two dilated convolution layers in the encoder is associated with a feature vector that represents 256 features and an 8×8 dilation kernel,
each of the next two dilated convolution layers in the encoder is associated with a feature vector that represents 512 features and a 16×16 dilation kernel,
each of the next two dilated convolution layers in the encoder is associated with a feature vector that represents 1024 features using a 32×32 dilation kernel, and
wherein:
each of the first two dilated convolution layers in the decoder is associated with a feature vector that represents 512 features and a 16×16 dilation kernel,
each of the next two dilated convolution layers in the decoder is associated with a feature vector that represents 256 features and a 8×8 dilation kernel,
each of the next two dilated convolution layers in the decoder is associated with a feature vector that represents 128 features and a 4×4 dilation kernel, and
the last two dilated convolution layers in the decoder are associated with a feature vector that represents 64 features and a 2×2 dilation kernel, and
wherein:
a first of the skip connections extends from a point in the encoder just before a first of the max pooling layers to an output of a final processing segment in the decoder,
a second of the skip connections extends from a point in the encoder just before a second of the max pooling layers to a point in the decoder just before a last two of the processing segments in the decoder,
a third of the skip connections extends from a point in the encoder just before a third of the max pooling layers to a point in the decoder just before a last four of the processing segments in the decoder, and
a fourth of the skip connections extends from a point in the encoder just before a last of the max pooling layers to a point in the decoder just after a first two of the processing segments in the decoder.

### Claim 29

A computer-based system comprising:
a network interface connection for receiving a plurality of aerial images from a data source over a network, wherein each image has a first visual indicator on the aerial image that follows and identifies a boundary line for the property;
a building rooftop deep fully convolutional network, configured and trained to predict the presence of building rooftops in aerial imagery, and apply a second visual indicator to the aerial image to identify and outline a building rooftop in the aerial image identified by the building rooftop deep fully convolutional network, and
an additional deep fully convolutional network, configured and trained to predict the presence of an additional property feature in aerial imagery, and predict whether said additional property feature is present with the boundary line of the property based on the aerial image,
wherein the building rooftop and additional deep convolutional networks have identical configurations but use different weights.