# US 2021/0279907 A1 · Methods and systems for sensing obstacles in an indoor environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279907 A1
- Document ID: US20210279907A1
- Status: Published application
- Kind: A1
- Application number: 16/809,901
- Filed: 2020-03-05
- Published: 2021-09-09
- Earliest claimed: 2020-03-05
- Assignees: Xerox Corporation
- Applicants: Xerox Corporation
- Inventors: Fritz Francis Ebner; Abhisek Dey; Karthik Subramanian
- Web page: https://patentgrep.com/patent/US20210279907A1

## CPC

- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 7/50: Depth or shape recovery
- H04W 4/026: {using orientation information, e.g. compass}
- H04W 4/029: Location-based management or tracking services
- H04W 4/33: for indoor environments, e.g. buildings
- G06T 2207/10024: Color image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

A system of determining a distance between a mobile electronic device and an obstacle in an environment includes an electronic device, and a computer-readable storage medium. The computer-readable storage medium includes one or more programming instructions that, when executed, cause the electronic device to obtain from a camera of a mobile electronic device one or more images of one or more obstacles in a surrounding environment, and apply a convolution neural network to one or more of the obtained images to generate a one-dimensional array of depth perception estimates. The array includes one or more angle-distance pairs. an angle value of each angle-distance pair represents an angle of one or more of the obstacles relative to the camera. A distance value of each angle-distance pairs represents an estimated distance between the camera and one or more of the obstacles at the corresponding angle of the angle-distance pair.

## Claims

### Claim 1

A method of determining a distance between a mobile electronic device and an obstacle in an environment, the method comprising:
by an electronic device:
obtaining from a camera of a mobile electronic device one or more images of one or more obstacles in a surrounding environment, and
applying a convolution neural network to one or more of the obtained images to generate a one-dimensional array of depth perception estimates, wherein the array comprises one or more angle-distance pairs, wherein an angle value of each angle-distance pair represents an angle of one or more of the obstacles relative to the camera, wherein a distance value of each angle-distance pairs represents an estimated distance between the camera and one or more of the obstacles at the corresponding angle of the angle-distance pair.

### Claim 2 (depends on 1)

The method of claim 1, wherein the convolution neural network is trained on a loss function represented by:
L
Primary
=
1
n
⁢
∑
i
=
1
n
⁢
e

y
i
-
y
i
⁡
(
true
)

where n is the array;
Yi is an output of the convolution neural network for measurement i,
Ytrue is an actual distance for measurement i.

### Claim 3 (depends on 2)

The method of claim 2, wherein the array has a length of 224.

### Claim 4 (depends on 1)

The method of claim 1, wherein each angle value represents an index of the array.

### Claim 5 (depends on 1)

The method of claim 1, wherein the camera comprises a monocular RGB (Red, Green, Blue) camera.

### Claim 6 (depends on 1)

The method of claim 1, wherein the camera comprises a rear-facing camera.

### Claim 7 (depends on 1)

The method of claim 1, wherein the convolution neural network comprises:
one or more trainable convolution layers;
one or more global average pooling layers; and
a fully-connected layer.

### Claim 8 (depends on 7)

The method of claim 7, wherein:
the one or more of the trainable convolution layers are arranged in a sequence,
one or more of the trainable convolution layers is configured to accept as input an output of a previous trainable convolution layer in the sequence and filter it with its own kernel.

### Claim 9 (depends on 1)

The method of claim 1, further comprising fine tuning the convolution neural network using the following function:
L
Secondary
=
1
n
⁢
∑
i
=
1
n
⁢

y
i
-
y
i
⁡
(
true
)

where n is a matrix of depth perception estimates having a length of 224;
Yi is an output of the convolution neural network for measurement i
Ytrue is an actual distance for measurement i

### Claim 10 (depends on 1)

The method of claim 1, further comprising applying a second convolution neural network to one or more of the obtained images to generate a confidence metric associated with one or more of the depth perception estimates, wherein the confidence metric represents an indication of how accurate the associated depth perception estimate is.

### Claim 11 (depends on 10)

The method of claim 10, wherein the second convolution neural network comprises:
one or more trainable convolution layers;
one or more global maximum pooling layers; and
a confidence layer.

### Claim 12 (depends on 10)

The method of claim 10, further comprising not updating the convolution neural network to incorporate any of the depth perception estimates that are associated with a confidence metric that is below a threshold value or outside of a threshold range of values.

### Claim 13

A system of determining a distance between a mobile electronic device and an obstacle in an environment, the system comprising:
an electronic device; and
a computer-readable storage medium comprising one or more programming instructions that, when executed, cause the electronic device to:
obtain from a camera of a mobile electronic device one or more images of one or more obstacles in a surrounding environment, and
apply a convolution neural network to one or more of the obtained images to generate a one-dimensional array of depth perception estimates, wherein the array comprises one or more angle-distance pairs, wherein an angle value of each angle-distance pair represents an angle of one or more of the obstacles relative to the camera, wherein a distance value of each angle-distance pairs represents an estimated distance between the camera and one or more of the obstacles at the corresponding angle of the angle-distance pair.

### Claim 14 (depends on 13)

The system of claim 13, wherein the convolution neural network is trained on a loss function represented by:
L
Primary
=
1
n
⁢
∑
i
=
1
n
⁢
e

y
i
-
y
i
⁡
(
true
)

where n is the array;
Yi is an output of the convolution neural network for measurement i,
Ytrue is an actual distance for measurement i.

### Claim 15 (depends on 14)

The system of claim 14, wherein the array has a length of 224.

### Claim 16 (depends on 13)

The system of claim 13, wherein each angle value represents an index of the array.

### Claim 17 (depends on 13)

The system of claim 13, wherein the camera comprises a monocular RGB (Red, Green, Blue) camera.

### Claim 18 (depends on 13)

The system of claim 13, wherein the camera comprises a rear-facing camera.

### Claim 19 (depends on 13)

The system of claim 13, wherein the convolution neural network comprises:
one or more trainable convolution layers;
one or more global average pooling layers; and
a fully-connected layer.

### Claim 20 (depends on 19)

The system of claim 19, wherein:
the one or more of the trainable convolution layers are arranged in a sequence,
one or more of the trainable convolution layers is configured to accept as input an output of a previous trainable convolution layer in the sequence and filter it with its own kernel.

### Claim 21 (depends on 13)

The system of claim 13, wherein the computer-readable storage medium further comprises one or more programming instructions that, when executed, cause the electronic device to fine tune the convolution neural network using the following function:
L
Secondary
=
1
n
⁢
∑
i
=
1
n
⁢

y
i
-
y
i
⁡
(
true
)

where n is a matrix of depth perception estimates having a length of 224;
Yi is an output of the convolution neural network for measurement i
Ytrue is an actual distance for measurement i

### Claim 22 (depends on 13)

The system of claim 13, wherein the computer-readable storage medium further comprises one or more programming instructions that, when executed, cause the electronic device to apply a second convolution neural network to one or more of the obtained images to generate a confidence metric associated with one or more of the depth perception estimates, wherein confidence metric represents an indication of how accurate the associated depth perception estimate is.

### Claim 23 (depends on 22)

The system of claim 22, wherein the second convolution neural network comprises:
one or more trainable convolution layers;
one or more global maximum pooling layers; and
a confidence layer.

### Claim 24 (depends on 22)

The system of claim 22, wherein the computer-readable storage medium further comprises one or more programming instructions that, when executed, cause the electronic device to not update the convolution neural network to incorporate any of the depth perception estimates that are associated with a confidence metric that is below a threshold value or outside of a threshold range of values.