# US 11,443,525 B1 · Image and video compression for remote vehicle assistance

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,525 B1
- Document ID: US11443525B1
- Status: Granted
- Kind: B1
- Application number: 16/747,380
- Filed: 2020-01-20
- Granted: 2022-09-13
- Earliest claimed: 2014-04-10
- Assignees: Waymo LLC
- Applicants: Waymo LLC
- Inventors: Vadim Furman; Andrew Hughes Chatham; Abhijit Ogale; Dmitri Dolgov
- Web page: https://patentgrep.com/patent/US11443525B1

## CPC

- H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635)
- B60R 11/04: Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
- G01C 21/30: Map- or contour-matching
- G01C 21/3896: {Transmission of map data from central databases}
- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- H04N 19/136: Incoming video signal characteristics or properties
- H04N 19/167: Position within a video image, e.g. region of interest [ROI]
- H04N 19/17: the unit being an image region, e.g. an object
- H04N 19/85: using pre-processing or post-processing specially adapted for video compression
- B60R 2011/0026: {Windows, e.g. windscreen}
- B60R 2011/004: {outside the vehicle}
- B60R 2011/008: {Adjustable or movable supports}
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A vehicle may receive one or more images of an environment of the vehicle. The vehicle may also receive a map of the environment. The vehicle may also match at least one feature in the one or more images with corresponding one or more features in the map. The vehicle may also identify a given area in the one or more images that corresponds to a portion of the map that is within a threshold distance to the one or more features. The vehicle may also compress the one or more images to include a lower amount of details in areas of the one or more images other than the given area. The vehicle may also provide the compressed images to a remote system, and responsively receive operation instructions from the remote system.

## Claims

### Claim 1

A method comprising:
obtaining, by a vehicle, from one or more sensors of the vehicle, sensor data indicative of an environment of the vehicle;
detecting, based on the sensor data, an object in the environment;
determining, by the vehicle based on classification data stored in a data storage of the vehicle, a classification of the detected object, wherein the determined classification of the detected object has an associated confidence level;
comparing, by the vehicle, the associated confidence level to a confidence threshold;
based at least on the comparison of the associated confidence level to the confidence threshold, providing, by the vehicle, at least a portion of the sensor data for receipt by a remote system;
based on the provision, receiving, from the remote system, navigation instructions for navigating the vehicle in the environment; and
operating the vehicle based on at least the navigation instructions from the remote system.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
based on at least the detection, providing a request for navigation assistance for receipt by the remote system, wherein receiving the navigation instructions from the remote system is further based on the provided request.

### Claim 3 (depends on 2)

The method of claim 2,
wherein providing the request is also based on the comparison of the associated confidence level to the confidence threshold.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
determining, by the vehicle, that the detected object relates to an unusual event, wherein providing at least the portion of the sensor data for receipt by the remote system is also based on the determination that the detected object relates to the unusual event.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
generating, by a computing system of the vehicle, given navigation instructions for navigating the vehicle in the environment,
wherein operating the vehicle comprises updating the given navigation instructions based on the navigation instructions received from the remote system.

### Claim 6 (depends on 1)

The method of claim 1, wherein detecting the object is further based on the object being in a region of the environment associated with the navigation instructions.

### Claim 7 (depends on 1)

The method of claim 1, wherein the navigation instructions received from the remote system are based on at least user input from a human operator at the remote system.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
compressing the sensor data, wherein compressing the sensor data comprises reducing a given portion of the sensor data associated with a given region of the environment other than a region that includes the detected object, and
wherein providing the at least portion of the sensor data comprises providing the compressed sensor data for receipt by the remote system.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
accessing a stored representation of the environment of the vehicle;
matching a feature of the environment indicated in the sensor data to a corresponding feature indicated in the stored representation; and
based on the matching, selecting, from the sensor data, the at least portion of the sensor data provided for receipt by the remote system.

### Claim 10 (depends on 9)

The method of claim 9, wherein selecting the at least portion of the sensor data is based on at least a distance between the matched feature and a region of the environment associated with the at least portion of the sensor data.

### Claim 11 (depends on 9)

The method of claim 9, wherein the matched feature relates to a road, and wherein the vehicle is located on the road.

### Claim 12

A vehicle comprising:
one or more sensors that provide sensor data indicative of an environment of the vehicle;
one or more processors; and
data storage storing instructions executable by the one or more processors to cause the vehicle to perform operations comprising:
obtaining, from the one or more sensors, sensor data indicative of the environment;
detecting, based on the sensor data, an object in the environment;
determining, based on classification data stored in the data storage, a classification of the detected object, wherein the determined classification of the detected object has an associated confidence level;
comparing the associated confidence level to a confidence threshold;
based at least on the comparison of the associated confidence level to the confidence threshold, transmitting at least a portion of the sensor data for receipt by a remote system;
based on the transmitting, receiving, from the remote system, navigation instructions for navigating the vehicle in the environment; and
operating the vehicle based on at least the navigation instructions from the remote system.

### Claim 13 (depends on 12)

The vehicle of claim 12, wherein the operations further comprise:
accessing a stored representation of the environment of the vehicle; and
based on the stored representation, selecting, from the sensor data, the at least portion of the sensor data provided for receipt by the remote system.

### Claim 14 (depends on 12)

The vehicle of claim 12, wherein the one or more sensors of the vehicle comprise a camera.

### Claim 15 (depends on 14)

The vehicle of claim 14, wherein the sensor data comprises one or more images captured using the camera.

### Claim 16 (depends on 14)

The method vehicle of claim 14, wherein the sensor data comprises a video stream based on at least output from the camera.

### Claim 17 (depends on 12)

The vehicle of claim 12, wherein the operations further comprise:
based on at least the detection, transmitting a request for navigation assistance for receipt by the remote system, wherein the navigation instructions from the remote system are further based on the transmitted request.

### Claim 18

A system comprising:
one or more processors;
data storage storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
receiving, from one or more sensors of a vehicle, sensor data indicative of an environment of the vehicle;
detecting, based on the sensor data, an object in the environment;
determining, based on classification data stored in the data storage, a classification of the detected object, wherein the determined classification of the detected object has an associated confidence level;
comparing the associated confidence level to a confidence threshold;
based at least on the comparison of the associated confidence level to the confidence threshold, transmitting at least a portion of the sensor data for receipt by a remote system;
based on the transmitting, receiving, from the remote system, navigation instructions for navigating the vehicle in the environment; and
operating the vehicle based on at least the navigation instructions from the remote system.

### Claim 19 (depends on 18)

The system of claim 18, wherein the sensor data comprises one or more images captured by one or more cameras of the vehicle.

### Claim 20 (depends on 19)

The system of claim 19, wherein the navigation instructions are based on user input received at the remote system in response to display of at least a portion of the one or more images at the remote system.