# US 10,872,239 B2 · Entrance detection from street-level imagery

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,872,239 B2
- Document ID: US10872239B2
- Status: Granted
- Kind: B2
- Application number: 16/399,507
- Filed: 2019-04-30
- Published: 2019-08-22
- Granted: 2020-12-22
- Earliest claimed: 2014-06-08
- Assignees: Uber Technologies, Inc.
- Applicants: Uber Technologies, Inc.
- Inventors: Jingchen Liu; Vasudev Parameswaran; Thommen Korah; Varsha Hedau; Radek Grzeszczuk; Yanxi Liu
- Web page: https://patentgrep.com/patent/US10872239B2

## CPC

- G06V 20/38: {Outdoor scenes}
- G06F 18/24: Classification techniques
- G06F 18/2415: based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
- G06T 7/90: Determination of colour characteristics
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Architecture that detects entrances on building facades. In a first stage, scene geometry is exploited and the multi-dimensional problem is reduced down to a one-dimensional (1D) problem. Entrance hypotheses are generated by considering pairs of locations along lines exhibiting strong gradients in the transverse direction. In a second stage, a rich set of discriminative image features for entrances is explored according to constructed designs, specifically focusing on properties such as symmetry and color consistency, for example. Classifiers (e.g., random forest) are utilized to perform automatic feature selection and entrance classification. In another stage, a joint model is formulated in three dimensions (3D) for entrances on a given facade, which enables the exploitation of physical constraints between different entrances on the same facade in a systematic manner to prune false positives, and thereby select an optimum set of entrances on a given facade.

## Claims

### Claim 1

A computer-implemented method comprising:
accessing images of a façade of a building;
extracting one or more entrance candidates from the images by detecting vertical edgelets in the images using edge detection, a vertical edgelet including an edge pixel and one or more neighboring pixels along a vertical axis; and
selecting one or more true candidates from the one or more entrance candidates based on known constraints of entrances.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
determining a frontal view of the façade using a map of the façade of the building;
generating a foreground mask that masks out portions of the images outside the frontal view of the façade; and
applying the foreground mask to the images.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, further comprising:
determining a ground line using the map; and
generating the foreground mask, the foreground mask masking out portions of the images below the ground line.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein extracting one or more entrance candidates from the images further comprises:
separating entrances from non-entrances using a classifier, the classifier performing automatic feature selection and entrance classification to remove non-entrances.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein features used in the classifier include degree of entrance symmetry and entrance color.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein extracting one or more entrance candidates from the images further comprises:
generating a distribution of all detected vertical edgelets;
extracting a number of top ranked local peaks from the distribution to determine one or more vertical boundaries; and
determining one or more entrance candidates based on the one or more vertical boundaries.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, further comprising:
detecting horizontal edgelets in the images using edge detection;
generating a second distribution of all detected horizontal edgelets;
extracting a number of top ranked local peaks from the second distribution to determine one or more horizontal boundaries; and
selecting one or more entrance candidates, each entrance candidate having a pair of vertical boundaries within a predefined ratio of length to a horizontal boundary of the one or more horizontal boundaries.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein selecting one or more true candidates from the entrance candidate further comprises:
projecting the entrance candidates into multi-dimensional space to resolve conflicts based on the known physical constraints using a multi-dimensional reasoning component, the known physical constraints encoded within a multi-dimensional reasoning component.

### Claim 9

A system for detecting entrances in images of structural facades, comprising:
at least one processor configured to execute computer-executable instructions in a computer readable hardware storage memory, the computer-executable instructions, when executed by the processor, cause the processor to:
access images of a façade of a building;
extract one or more entrance candidates from the images by detecting vertical edgelets in the images using edge detection, a vertical edgelet including an edge pixel and one or more neighboring pixels along a vertical axis; and
selecting one or more true candidates from the one or more entrance candidates based on known constraints of entrances.

### Claim 10 (depends on 9)

The system of claim 9, wherein the computer readable hardware storage memory comprises further computer-executable instructions, that when executed by the processor, cause the processor to:
determining a frontal view of the façade using a map of the façade of the building;
generating a foreground mask that masks out portions of the images outside the frontal view of the façade; and
applying the foreground mask to the images.

### Claim 11 (depends on 10)

The system of claim 10, wherein the computer readable hardware storage memory comprises further computer-executable instructions, that when executed by the processor, cause the processor to:
determine a ground line using the map; and
generate the foreground mask, the foreground mask masking out portions of the images below the ground line.

### Claim 12 (depends on 9)

The system of claim 9, wherein the computer readable hardware storage memory comprises further computer-executable instructions for the extraction of the one or more entrance candidates from the images, that when executed by the processor, cause the processor to:
separate entrances from non-entrances using a classifier, the classifier performing automatic feature selection and entrance classification to remove non-entrances.

### Claim 13 (depends on 12)

The system of claim 12, wherein features used in the classifier include degree of entrance symmetry and entrance color.

### Claim 14 (depends on 9)

The system of claim 9, wherein the computer readable hardware storage memory comprises further computer-executable instructions for the extraction of the one or more entrance candidates from the images, that when executed by the processor, cause the processor to:
generate a distribution of all detected vertical edgelets;
extract a number of top ranked local peaks from the distribution to determine one or more vertical boundaries; and
determine one or more entrance candidates based on the one or more vertical boundaries.

### Claim 15 (depends on 14)

The system of claim 14, wherein the computer readable hardware storage memory comprises further computer-executable instructions, that when executed by the processor, cause the processor to:
detect horizontal edgelets in the images using edge detection;
generate a second distribution of all detected horizontal edgelets;
extract a number of top ranked local peaks from the second distribution to determine one or more horizontal boundaries; and
select one or more entrance candidates, each entrance candidate having a pair of vertical boundaries within a predefined ratio of length to a horizontal boundary of the one or more horizontal boundaries.

### Claim 16

A computer-readable hardware storage medium comprising computer-executable instructions for detecting entrances in images of structural facades that when executed by a processor, cause the processor to:
access images of a façade of a building;
extract one or more entrance candidates from the images by detecting vertical edgelets in the images using edge detection, a vertical edgelet including an edge pixel and one or more neighboring pixels along a vertical axis; and
select one or more true candidates from the one or more entrance candidates based on known constraints of entrances.

### Claim 17 (depends on 16)

The computer-readable hardware storage medium of claim 16, comprising further computer-executable instructions, that when executed by the processor, cause the processor to:
determine a frontal view of the façade using a map of the façade of the building;
generate a foreground mask that masks out portions of the images outside the frontal view of the façade; and
apply the foreground mask to the images.

### Claim 18 (depends on 17)

The computer-readable hardware storage medium of claim 17, comprising further computer-executable instructions, that when executed by the processor, cause the processor to:
determine a ground line using the map; and
generate the foreground mask, the foreground mask masking out portions of the images below the ground line.

### Claim 19 (depends on 16)

The computer-readable hardware storage medium of claim 16, comprising further computer-executable instructions for the extraction of the one or more entrance candidates from the images, that when executed by the processor, cause the processor to:
separate entrances from non-entrances using a classifier, the classifier performing automatic feature selection and entrance classification to remove non-entrances.

### Claim 20 (depends on 16)

The computer-readable hardware storage medium of claim 16, comprising further computer-executable instructions for the extraction of the one or more entrance candidates from the images, that when executed by the processor, cause the processor to:
generate a distribution of all detected vertical edgelets;
extract a number of top ranked local peaks from the distribution to determine one or more vertical boundaries; and
determine one or more entrance candidates based on the one or more vertical boundaries.