# US 2023/0274491 A1 · Hybrid depth maps

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274491 A1
- Document ID: US20230274491A1
- Status: Published application
- Kind: A1
- Application number: 18/001,659
- Filed: 2021-09-01
- Published: 2023-08-31
- Earliest claimed: 2021-09-01
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Eric Turner; Adarsh Prakash Murthy Kowdle; Bicheng Luo; Juan David Hincapie Ramos
- Web page: https://patentgrep.com/patent/US20230274491A1

## CPC

- G01C 21/3804: {Creation or updating of map data}
- G06T 15/04: Texture mapping
- G06T 7/50: Depth or shape recovery
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method including receiving (S605) a request for a depth map, generating (S625) a hybrid depth map based on a device depth map (110) and downloaded depth information (105), and responding (S630) to the request for the depth map with the hybrid depth map (415). The device depth map (110) can be depth data captured on a user device (515) using sensors and/or software. The downloaded depth information (105) can be associated with depth data, map data, image data, and/or the like stored on a remote (to the user device) server (505).

## Claims

### Claim 1

A method comprising:
receiving a request for a depth map;
generating a hybrid depth map based on a device depth map and downloaded depth information; and
responding to the request for the depth map with the hybrid depth map.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating the hybrid depth map includes
retrieving the downloaded depth information from a network device, and
transforming a format of the downloaded depth information to match a format of the device depth map.

### Claim 3 (depends on 1)

The method of claim 1, wherein generating the hybrid depth map includes
rendering the device depth map as geometric data,
rendering the downloaded depth information as geometric data, and
blending the device depth map geometric data with the downloaded depth information geometric data.

### Claim 4 (depends on 3)

The method of claim 3, wherein the geometric data is a surface element (surfel).

### Claim 5 (depends on 1)

The method of claim 1, wherein
the downloaded depth information is in a latitude/longitude coordinate system format,
the device depth map is in a cartesian coordinate system format, and
generating the hybrid depth map includes transforming the downloaded depth information format to the cartesian coordinate system format.

### Claim 6 (depends on 1)

The method of claim 1, further comprising triggering a downloading of the downloaded depth information based on a location of a mobile device.

### Claim 7 (depends on 1)

The method of claim 1, further comprising triggering a downloading of the downloaded depth information based on a threshold distance of a mobile device from a location associated with downloaded depth information on the mobile device.

### Claim 8 (depends on 1)

The method of claim 1, further comprising filtering the downloaded depth information based on a distance of a datapoint in the downloaded depth information from a location of a mobile device.

### Claim 9 (depends on 1)

The method of claim 1, wherein the device depth map is generated at least one of using hardware sensors of a mobile device and using software executing on the mobile device.

### Claim 10 (depends on 1)

The method of claim 1, wherein
the device depth map is generated using data that is captured at substantially the same time as a corresponding image is being captured by a mobile device, and
the image is being rendered on a display of the mobile device directly following the capture of the image.

### Claim 11 (depends on 1)

The method of claim 1, wherein
the downloaded depth information is captured by a different device than a device generating the device depth map,
the downloaded depth information includes data that was captured earlier in time as compared to a time the device depth map is generated,
the downloaded depth information includes data that can be converted to depth data by the device generating the device depth map, and
the downloaded depth information includes distance values in a range that is outside a range of depth values associated with the device depth map.

### Claim 12 (depends on 1)

The method of claim 1, wherein
the hybrid depth map includes distance values in a first range limited in distance by capabilities associated with a device generating the device depth map, and
the hybrid depth map includes distance values in at least one second range outside the first range.

### Claim 13

A non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by at least one processor, are configured to cause a computing system to:
receive a request for a depth map;
generate a hybrid depth map based on a device depth map and downloaded depth information; and
respond to the request for the depth map with the hybrid depth map.

### Claim 14 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, wherein generating the hybrid depth map includes
retrieving the downloaded depth information from a network device, and
transforming a format of the downloaded depth information to match a format of the device depth map.

### Claim 15

(canceled)

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the geometric data is a surface element (surfel).

### Claim 17

(canceled)

### Claim 18 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, further comprising triggering a downloading of the downloaded depth information based on a location of a mobile device.

### Claim 19

(canceled)

### Claim 20 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, further comprising filtering the downloaded depth information based on a distance of a datapoint in the downloaded depth information from a location of a mobile device.

### Claim 21

(canceled)

### Claim 22 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, wherein
the device depth map is generated using data that is captured at substantially the same time as a corresponding image is being captured by a mobile device, and
the image is being rendered on a display of the mobile device directly following the capture of the image.

### Claim 23

(canceled)

### Claim 24 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, wherein
the hybrid depth map includes distance values in a first range limited in distance by capabilities associated with a device generating the device depth map, and
the hybrid depth map includes distance values in at least one second range outside the first range.

### Claim 25

An apparatus comprising:
at least one processor; and
at least one memory including computer program code;
the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to:
receive a request for a depth map;
generate a hybrid depth map based on a device depth map and downloaded depth information; and
respond to the request for the depth map with the hybrid depth map.

### Claim 26

(canceled)

### Claim 27 (depends on 25)

The apparatus of claim 25, wherein generating the hybrid depth map includes
rendering the device depth map as geometric data,
rendering the downloaded depth information as geometric data, and
blending the device depth map geometric data with the downloaded depth information geometric data.

### Claim 28

(canceled)

### Claim 29 (depends on 25)

The apparatus of claim 25, wherein
the downloaded depth information is in a latitude/longitude coordinate system format,
the device depth map is in a cartesian coordinate system format, and
generating the hybrid depth map includes transforming the downloaded depth information format to the cartesian coordinate system format.

### Claim 30

(canceled)

### Claim 31 (depends on 25)

The apparatus of claim 25, further comprising triggering a downloading of the downloaded depth information based on a threshold distance of a mobile device from a location a location associated with downloaded depth information on the mobile device.

### Claim 32

(canceled)

### Claim 33 (depends on 25)

The apparatus of claim 25, wherein the device depth map is generated at least one of using hardware sensors of a mobile device and using software executing on the mobile device.

### Claim 34

(canceled)

### Claim 35 (depends on 25)

The apparatus of claim 25, wherein
the downloaded depth information is captured by a different device than a device generating the device depth map,
the downloaded depth information includes data that was captured earlier in time as compared to a time the device depth map is generated,
the downloaded depth information includes data that can be converted to depth data by the device generating the device depth map, and
the downloaded depth information includes distance values in a range that is outside a range of depth values associated with the device depth map.

### Claim 36

(canceled)
