# US 11,999,316 B2 · Systems and methods of vehicle surveillance

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,999,316 B2
- Document ID: US11999316B2
- Status: Granted
- Kind: B2
- Application number: 17/578,704
- Filed: 2022-01-19
- Published: 2023-07-20
- Granted: 2024-06-04
- Earliest claimed: 2022-01-19
- Assignees: Nio Technology (Anhui) Co., Ltd
- Applicants: NIO Technology (Anhui) Co., Ltd.
- Inventors: Imama Ali
- Web page: https://patentgrep.com/patent/US11999316B2

## CPC

- B60R 25/1012: {Zone surveillance means, e.g. parking lots, truck depots}
- B60R 25/102: a signal being sent to a remote location, e.g. a radio signal being transmitted to a police station, a security company or the owner
- B60R 25/305: {using a camera}
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06T 7/11: Region-based segmentation
- G06T 7/194: involving foreground-background segmentation
- G06T 7/50: Depth or shape recovery
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G08B 13/19647: {Systems specially adapted for intrusion detection in or around a vehicle}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06T 2207/30232: Surveillance
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T 2207/30264: Parking
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS

## Abstract

Disclosed systems and methods include receiving an image, processing the image with a depth estimation module to generate a first depth map, generating a foreground mask of the image, inpainting regions of the image, generating a second depth map based on the inpainted regions of the image, comparing histograms of the first and second depth maps, and generating proximity warnings based on the comparison of the histograms.

## Claims

### Claim 1

A method of surveilling a vehicle, the method comprising:
receiving, from an image sensor, an image;
processing the received image to generate a first depth map;
processing, with a background modeling algorithm, the received image to generate one or more of a foreground mask and a background mask;
generating, with an image inpainting network, an inpainted image comprising the received image with an inpainted region associated with the one or more of the foreground mask and the background mask;
processing the inpainted image to generate a second depth map;
computing a histogram of the first depth map and a histogram of the second depth map;
comparing the histogram of the first depth map with the histogram of the second depth map to determine one or more of a size and a relative depth range of one or more foreground objects; and
generating an alert comprising data associated with the determined one or more of the size and the relative depth range of the one or more foreground objects.

### Claim 2 (depends on 1)

The method of claim 1, wherein the image is generated with a fish-eye lens.

### Claim 3 (depends on 1)

The method of claim 1, wherein the image sensor is externally mounted to the vehicle.

### Claim 4 (depends on 1)

The method of claim 1, wherein processing the image to generate the first depth map and processing the image with the background modeling algorithm to generate the one or more of the foreground mask and the background mask are performed in parallel.

### Claim 5 (depends on 1)

The method of claim 1, wherein the histogram of the first depth map is computed based on a region of the first depth map corresponding to the one or more of the foreground mask and the background mask.

### Claim 6 (depends on 1)

The method of claim 1, further comprising using a mean filter to recover missing information from the histogram.

### Claim 7 (depends on 1)

The method of claim 1, wherein the histogram of the second depth map is computed based on a region of the first depth map corresponding to the one or more of the foreground mask and the background mask.

### Claim 8 (depends on 1)

The method of claim 1, wherein the one or more of the size and the relative depth range of the one or more foreground objects is determined based on a comparison of the first depth map and the second depth map.

### Claim 9 (depends on 1)

The method of claim 1, wherein the inpainted image reflects a background of the received image with the one or more foreground objects replaced with a generated background.

### Claim 10 (depends on 1)

The method of claim 1, further comprising transmitting the alert to a user device.

### Claim 11 (depends on 1)

The method of claim 1, further comprising displaying the alert.

### Claim 12 (depends on 1)

The method of claim 1, further comprising, prior to generating the alert, determining the one or more of the size and the relative depth range exceeds a threshold.

### Claim 13 (depends on 1)

The method of claim 1, further comprising repeating the method on an interval.

### Claim 14 (depends on 1)

The method of claim 1, further comprising, in response to determining the one or more of the size and the relative depth range of the one or more foreground objects, activating one or more sensors.

### Claim 15 (depends on 1)

The method of claim 1, further comprising, prior to computing the histogram of the first depth map, applying the one or more of the foreground mask and the background mask to the first depth map.

### Claim 16 (depends on 15)

The method of claim 15, further comprising processing, with the background modeling algorithm, the inpainted image to generate a second one or more of a foreground mask and a background mask.

### Claim 17 (depends on 16)

The method of claim 16, further comprising, prior to computing the histogram of the second depth map, applying the second one or more of the foreground mask and the background mask to the second depth map.

### Claim 18

A user device comprising:
a processor; and
a computer-readable storage medium storing computer-readable instructions which, when executed by the processor, cause the processor to execute a method of surveilling a vehicle, the method comprising:
receiving, from an image sensor, an image;
processing the received image to generate a first depth map;
processing, with a background modeling algorithm, the received image to generate one or more of a foreground mask and a background mask;
generating, with an image inpainting network, an inpainted image comprising the received image with an inpainted region associated with the one or more of the foreground mask and the background mask;
processing the inpainted image to generate a second depth map;
computing a histogram of the first depth map and a histogram of the second depth map;
based on the histogram of the first depth map and the histogram of the second depth map, determining one or more of a size and a relative depth range of one or more foreground objects; and
generating an alert comprising data associated with the determined one or more of the size and the relative depth range of the one or more foreground objects.

### Claim 19 (depends on 18)

The user device of claim 18, wherein the image is generated with a fish-eye lens.

### Claim 20

A computer program product comprising:
a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code configured, when executed by a processor, to execute a method of surveilling a vehicle, the method comprising:
receiving, from an image sensor, an image;
processing the received image to generate a first depth map;
processing, with a background modeling algorithm, the received image to generate one or more of a foreground mask and a background mask;
generating, with an image inpainting network, an inpainted image comprising the received image with an inpainted region associated with the one or more of the foreground mask and the background mask;
processing the inpainted image to generate a second depth map;
computing a histogram of the first depth map and a histogram of the second depth map;
based on the histogram of the first depth map and the histogram of the second depth map, determining one or more of a size and a relative depth range of one or more foreground objects; and
generating an alert comprising data associated with the determined one or more of the size and the relative depth range of the one or more foreground objects.