# US 10,504,007 B2 · Determination of population density using convoluted neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,504,007 B2
- Document ID: US10504007B2
- Status: Granted
- Kind: B2
- Application number: 15/795,741
- Filed: 2017-10-27
- Published: 2019-05-02
- Granted: 2019-12-10
- Earliest claimed: 2017-10-27
- Assignees: Facebook, Inc.
- Applicants: Facebook, Inc.
- Inventors: Tobias Gerard Tiecke; Amy Zhang; Xianming Liu; Andreas Gros
- Web page: https://patentgrep.com/patent/US10504007B2

## CPC

- G06V 10/82: using neural networks
- G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling}
- G06F 18/241: relating to the classification model, e.g. parametric or non-parametric approaches
- G06F 18/2414: {Smoothing the distance, e.g. radial basis function networks [RBFN]}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06V 10/30: Noise filtering
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7753: {Incorporation of unlabelled data, e.g. multiple instance learning [MIL]}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

In one embodiment, a method includes receiving an image on a computing device. The computing device may further execute a weakly-supervised classification algorithm to determine whether a target feature is present in the received image. As an example, the weakly-supervised classification algorithm may determine whether a building is depicted in the received image. In response to determining that a target feature is present, the method further includes using a weakly-supervised segmentation algorithm of the convoluted neural network to segment the received image for the target feature. Based on a determined footprint size of the target feature, a distribution of statistical information over the target feature in the image can be calculated.

## Claims

### Claim 1

A method comprising:
receiving an image;
executing a weakly-supervised classification algorithm to determine whether a target feature is present in the received image;
in response to determining that a target feature is present, using a weakly-supervised segmentation algorithm of the convoluted neural network to segment the received image for the target feature and determine a footprint size of the target feature; and
calculating a distribution of statistical information over the target feature based on the determined footprint size of the target feature.

### Claim 2 (depends on 1)

The method of claim 1, wherein calculating the distribution of statistical information is based at least in part on a property that scales with the size of the target feature within an image.

### Claim 3 (depends on 1)

The method of claim 1, wherein using the convoluted neural network to remove noise or haze comprises using an adaptive learnable transformer, wherein the adaptive learnable transformer is trained to remove noise or haze from pixels corresponding to the target feature.

### Claim 4 (depends on 3)

The method of claim 3, wherein training the adaptive learnable transformer is based on image-level labeled data and the weakly-supervised classification algorithm.

### Claim 5 (depends on 1)

The method of claim 1, wherein the weakly-supervised classification algorithm is trained using image-level labeled data, without pixel-level labeled data.

### Claim 6 (depends on 1)

The method of claim 1, wherein determining whether a target feature is present in the received image comprises:
for each pixel in the received image, determining, using the weakly-supervised classification algorithm, a per-pixel probability that the pixel corresponds to the target feature; and
determining an average of the per-pixel probabilities for the pixels in the received image.

### Claim 7 (depends on 1)

The method of claim 1, wherein the weakly-supervised classification algorithm further comprises a feedback loop to suppress irrelevant neuron activations of the convoluted neural network.

### Claim 8 (depends on 1)

The method of claim 1, wherein the weakly-supervised segmentation algorithm is trained using image-level labeled data, without pixel-level labeled data.

### Claim 9 (depends on 1)

The method of claim 1, wherein the weakly-supervised segmentation algorithm is trained to minimize a loss function
f
w
=
arg
⁢
⁢
min
⁢
∑
i
⁢
1
2
⁢

y
i
-
f
w
⁡
(
x
i
)

2
+
λ
⁢

w

2
wherein fw is the transformation from input x to output ŷ parametered with w.

### Claim 10 (depends on 1)

The method of claim 1, wherein the convoluted neural network comprises a plurality of layers.

### Claim 11 (depends on 10)

The method of claim 10, wherein each layer comprises a plurality of neurons.

### Claim 12 (depends on 11)

The method of claim 11, wherein for a particular layer 1 with input x1 and target output y1, the convoluted neural network optimizes a target function min ½∥yl−fw(xl)∥2+γ∥xl∥1, wherein:
fw is the transformation from input x to output ŷ parametered with w; and
for a particular neuron nil:
if
⁢
⁢
∂

y
l
-
f
w
⁡
(
x
l
)

2
∂
x
i
l
>
γ
⁢
⁢
then
⁢
⁢
x
i
l
⁢
⁢
may
⁢
⁢
be
⁢
⁢
positively
⁢
⁢
activated
;
if
⁢
⁢
∂

y
l
-
f
w
⁡
(
x
l
)

2
∂
x
i
l
<
-
γ
⁢
⁢
then
⁢
⁢
x
i
l
⁢
⁢
may
⁢
⁢
be
⁢
⁢
negatively
⁢
⁢
activated
;
else, neuron xil is deactivated.

### Claim 13 (depends on 10)

The method of claim 10, wherein the convoluted neural network comprises one or more stacks, wherein each stack comprises a feedback unit comprising one or more layers.

### Claim 14 (depends on 13)

The method of claim 13, wherein each feedback unit allows neurons with positive gradients to be activated.

### Claim 15 (depends on 13)

The method of claim 13, wherein the one or more layers comprise:
a feedback layer;
a rectified linear unit layer;
a batch normalization layer; and
a convolution layer.

### Claim 16 (depends on 15)

The method of claim 15, wherein the feedback layer allows neurons with a positive gradient
∂

y
-
y
^

2
∂
x
l
>
γ
to be activated.

### Claim 17 (depends on 16)

The method of claim 16, wherein γ=0.

### Claim 18 (depends on 1)

The method of claim 1, wherein the weakly-supervised classification algorithm and the weakly-supervised segmentation algorithm are trained using a feedback network following the algorithm:
Input: Sample {xi, yi}, and neural network initialization fw
1
Forward: Inferring estimation ŷi = fw (xi);
2
Feedback: Estimating Neuron Activations θ
3
for each layer l do
4
if
⁢
⁢
∂

y
1
-
fw
⁡
(
x
1
)

2
∂
x
⁢
1
I
->
γ
⁢
⁢
then
5
Active neuron xil: Set θil = 1;
6
else
7
Suppress neuron xil: Set θil = 0;
8
end
9
Backward: Perform backpropagation and update parameter using
SGD.

### Claim 19

One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
receive an image;
execute a weakly-supervised classification algorithm to determine whether a target feature is present in the received image;
in response to determining that a target feature is present, use a weakly-supervised segmentation algorithm of the convoluted neural network to segment the received image for the target feature and determine a footprint size of the target feature; and
calculate a distribution of statistical information over the target feature based on the determined footprint size of the target feature.

### Claim 20

A system comprising: one or more processors; and a memory coupled to the processors comprising instructions executable by the processors, the processors being operable when executing the instructions to:
receive an image;
execute a weakly-supervised classification algorithm to determine whether a target feature is present in the received image;
in response to determining that a target feature is present, use a weakly-supervised segmentation algorithm of the convoluted neural network to segment the received image for the target feature and determine a footprint size of the target feature; and
calculate a distribution of statistical information over the target feature based on the determined footprint size of the target feature.