# US 2018/0324399 A1 · Creating a three-dimensional image via a wide-angle camera sensor

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0324399 A1
- Document ID: US20180324399A1
- Status: Published application
- Kind: A1
- Application number: 15/588,537
- Filed: 2017-05-05
- Published: 2018-11-08
- Earliest claimed: 2017-05-05
- Assignees: Motorola Mobility LLC
- Applicants: Motorola Mobility LLC
- Inventors: KURT E. SPEARS; SEAN C. KELLY; YUXIN WANG
- Web page: https://patentgrep.com/patent/US20180324399A1

## CPC

- H04N 13/156: Mixing image signals
- G06T 7/557: from light fields, e.g. from plenoptic cameras
- H04N 13/128: Adjusting depth or disparity
- H04N 13/178: Metadata, e.g. disparity information
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- H04N 13/232: using fly-eye lenses, e.g. arrangements of circular lenses
- H04N 13/254: in combination with electromagnetic radiation sources for illuminating objects
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- H04N 13/296: Synchronisation thereof; Control thereof
- H04N 23/65: Control of camera operation in relation to power supply
- H04N 23/672: {based on the phase difference signals}
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- G06T 2207/10024: Color image
- G06T 2207/10052: Images from lightfield camera
- G06T 2207/20221: Image fusion; Image merging
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method, system, and computer program product for generating a three-dimensional image from a captured two-dimensional image. The method includes capturing, via a wide-angle camera sensor of an image capturing device, a two-dimensional image of a scene. The method further includes determining depth information of the scene. The method further includes generating, from the depth information, a depth map that depicts a depth of the scene in a third dimension. The method further includes generating a wide-angle three-dimensional image by merging the depth map and the two-dimensional image.

## Claims

### Claim 1

A method comprising:
capturing, via a wide-angle camera sensor of an image capturing device, a two-dimensional image of a scene;
determining depth information of the scene;
generating, from the depth information, a depth map that depicts a depth of the scene in a third dimension; and
generating a wide-angle three-dimensional image by merging the depth map and the two-dimensional image.

### Claim 2 (depends on 1)

The method of claim 1, wherein:
the wide-angle camera sensor includes a two-dimensional light-field image sensor having a plurality of micro-lenses, each micro-lens having a color filter that covers a super-pixel, which includes an array of sub-pixels arranged in a quadrilateral pattern; and
determining the depth information further comprises:
determining, from the two-dimensional image, color values captured by each sub-pixel within each super-pixel;
determining an amount of captured light in each sub-pixel within each super-pixel;
comparing, in at least one direction, a phase of the captured light in each sub-pixel relative to a phase of light captured in sub-pixels in a plurality of proximate super-pixels;
determining a depth at each super-pixel of the scene based on a phase difference between each super-pixel in the at least one direction; and
storing the depth at each super-pixel as the depth information of the scene.

### Claim 3 (depends on 2)

The method of claim 2, wherein determining the depth at each super-pixel further comprises averaging a phase value of the plurality of proximate super-pixels.

### Claim 4 (depends on 1)

The method of claim 1, wherein:
the wide-angle camera sensor is a first wide-angle camera sensor that is attached to a first side of the image capturing device and aligned in a first direction;
a second wide-angle camera sensor is attached to a second side of the image capturing device, aligned in a second direction that is opposite the first direction; and
the method further comprises:
analyzing conditions of the scene by the first and second wide-angle camera sensors;
determining, based on the analysis of the conditions, a preferred capture mode for capturing the scene by the first and second wide-angle camera sensors;
applying the preferred capture mode to the first and second wide-angle camera sensors;
synchronizing the capture of the two-dimensional image by the first wide-angle camera sensor with a capture of a second two-dimensional image by the second wide-angle camera sensor;
generating a second wide-angle three-dimensional image by merging the depth map and the second two-dimensional image; and
combining the wide-angle three-dimensional image associated with the first wide-angle camera sensor with the second wide-angle three-dimensional image associated with the second wide-angle camera sensor to create a surround view three-dimensional image that captures the scene in circumferential view along an azimuth axis.

### Claim 5 (depends on 1)

The method of claim 1, wherein:
determining the depth information is performed by a depth detection module that includes at least one emitter module and a time-of-flight receiver module; and
the determining of the depth information further comprises:
modulating, via the at least one emitter module, a field of light within the scene;
receiving, via the time-of-flight receiver module, a reflection of the field of light;
detecting, for each pixel in the scene, a phase shift in the reflection relative to the field of light;
calculating, based on the detected phase shift, a depth of the scene at each pixel; and
storing the calculated depth at each pixel as the depth information.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
synchronizing, via a control module, the capture of the two-dimensional image by the at least one wide-angle camera sensor and the projection of the field of light by the at least one emitter module.

### Claim 7 (depends on 1)

The method of claim 1, wherein:
determining the depth information is performed by a depth detection module, which includes a receiver module and an emitter module that is arranged with the receiver module along a baseline; and
the determining of the depth information further comprises:
projecting, via the emitter module, a structured light pattern within the scene;
capturing, via the receiver module, image data of the scene having the structured light pattern superimposed thereon;
analyzing a distortion of the structured light pattern at each pixel in the image data;
calculating a depth of the scene at each pixel based on a corresponding distortion of the structured light pattern in the image data; and
storing the calculated depth at each pixel of the image data as the depth information.

### Claim 8 (depends on 1)

The method of claim 1, wherein:
determining the depth information is performed by a depth detection module, which includes a receiver module, a first emitter module that is arranged with the receiver module along a first baseline, and a second emitter module that is arranged with the receiver module along a second baseline that is different from the first baseline; and
the determining of the depth information further comprises:
projecting, via the first emitter module, a first structured light pattern within the scene during a first time interval;
capturing, via the receiver module, first image data of the scene having the first structured light pattern superimposed thereon;
projecting, via the second emitter module, a second structured light pattern within the scene during a second time interval, the second time interval being different from the first time interval;
capturing, via the receiver module, second image data of the scene having the second structured light pattern superimposed thereon;
analyzing a distortion of the first structured light pattern in the first image data;
calculating a first depth of the scene at each pixel based on the distortion of the first structured light pattern;
analyzing a distortion of the second structured light pattern in the second image data;
calculating a second depth of the scene at each pixel based on the distortion of the second structured light pattern;
merging the first depth and the second depth to determine a depth at each pixel of the scene; and
storing the determined depth at each pixel as the depth information.

### Claim 9

An image capturing device, comprising:
a wide-angle camera sensor that captures a two-dimensional image of a scene;
a depth detection module that determines depth information of the scene; and
a processor that:
generates, from the depth information, a depth map that depicts a depth of the scene in a third dimension; and
generates a wide-angle three-dimensional image by merging the depth map and the two-dimensional image.

### Claim 10 (depends on 9)

The image capturing device of claim 9, wherein:
the depth detection module is a sub-component within the wide-angle camera sensor;
the wide-angle camera sensor includes a two-dimensional light-field image sensor having a plurality of micro-lenses, each micro-lens having a color filter that covers a super-pixel, which includes an array of sub-pixels arranged in a quadrilateral pattern; and
to capture the depth information:
the wide-angle camera sensor determines, from the two-dimensional image, color values captured by each sub-pixel within each super-pixel; and
the processor:
detects an amount of captured light in each sub-pixel within each super-pixel;
compares, in at least one direction, a phase of the captured light in each sub-pixel relative to a phase of light captured in sub-pixels in a plurality of proximate super-pixels;
determines a depth at each super-pixel of the scene based on a phase difference between each super-pixel in the at least one direction; and
stores the depth at each super-pixel as the depth information of the scene.

### Claim 11 (depends on 10)

The image capturing device of claim 10, wherein in determining the depth at each super-pixel, the processor averages a phase value of the plurality of proximate super-pixels.

### Claim 12 (depends on 9)

The image capturing device of claim 9, wherein:
the wide-angle camera sensor is a first wide-angle camera sensor that is attached to a first side of the image capturing device and aligned in a first direction, and which analyzes a first set of conditions of the scene;
the image capturing device further comprises a second wide-angle camera sensor attached to a second side of the image capturing device, aligned in a second direction that is opposite the first direction, and which analyzes second set of conditions of the scene;
the processor:
determine based on the analysis of the conditions, a preferred capture mode for capturing the scene by the first and second wide-angle camera sensors; and
apply the preferred capture mode to the first and second wide-angle camera sensors;
the image capturing device further comprises a control module that synchronizes the capture of the two-dimensional image by the first wide-angle camera sensor with a capture of a second two-dimensional image by the second wide-angle camera sensor; and
the processor:
generates a second wide-angle three-dimensional image by merging the depth map and the second two-dimensional image; and
creates a surround view three-dimensional image that captures the scene in circumferential view along an azimuth axis by combining a first wide-angle three-dimensional image associated with the first wide-angle camera sensor with a second wide-angle three-dimensional image associated with the second wide-angle camera sensor.

### Claim 13 (depends on 9)

The image capturing device of claim 9, wherein:
the depth detection module includes:
at least one emitter module that modulates a field of light within the scene; and
a time-of-flight receiver module that receives a reflection of the field of light and detects a phase of light at each pixel in the image data; and
the processor:
detects, for each pixel in the scene, a phase shift in the reflection relative to the field of light;
calculates, based on the detected phase shift, a depth of the scene at each pixel; and
stores the calculated depth at each pixel as the depth information.

### Claim 14 (depends on 13)

The image capturing device of claim 13, further comprising:
a control model that synchronizes the capture of the two-dimensional image by the at least one wide-angle camera sensor and the projection of the field of light by the at least one emitter module.

### Claim 15 (depends on 9)

The image capturing device of claim 9, wherein:
the depth detection module includes:
an emitter module that projects a structured light pattern within the scene;
a receiver module that is arranged with the emitter module along a baseline and
which captures image data of the scene having the structured light pattern superimposed thereon; and
the processor:
analyzes a distortion of the structured light pattern at each pixel in the image data;
calculates a depth of the scene at each pixel based on a corresponding distortion of the structured light pattern in the image data; and
stores the calculated depth at each pixel of the image data as the depth information.

### Claim 16 (depends on 9)

The image capturing device of claim 9, wherein:
the depth detection module includes:
a first emitter module that projects a first structured light pattern within the scene during a first time interval;
a second emitter module that projects a second structured light pattern within the scene during a second time interval that is different from the first time interval; and
a receiver module that is arranged with: (a) with the first emitter baseline along a first baseline and (b) the second emitter module along a second baseline that is different from the first baseline, and that:
captures, during the first time interval, first image data of the scene having the first structured light pattern superimposed thereon; and
captures, during the second time interval, second image data of the scene having the second structured light pattern superimposed thereon; and
the processor:
analyzes a distortion of the first structured light pattern in the first image data;
calculates a first depth of the scene at each pixel based on the distortion of the first structured light pattern in the first image data;
analyzes a distortion of the second structured light pattern in the second image data;
calculates a second depth of the scene at each pixel based on a corresponding distortion of the second structured light pattern in the second image data;
merges the first depth and the second depth to determine a depth at each pixel of the scene; and
stores the determined depth at each pixel as the depth information.

### Claim 17

A computer program product comprising:
a computer readable storage device; and
program code on the computer readable storage device that, when executed by a processor associated with an image capturing device, enables the image capturing device to provide the functionality of:
capturing, via a wide-angle camera sensor of the image capturing device, a two-dimensional image of a scene;
determining depth information of the scene;
generating, from the depth information, a depth map that depicts a depth of the scene in a third dimension; and
generating a wide-angle three-dimensional image by merging the depth map and the two-dimensional image.

### Claim 18 (depends on 17)

The computer program product of claim 17, wherein:
the wide-angle camera sensor includes a two-dimensional light-field image sensor having a plurality of micro-lenses, each micro-lens having a color filter that covers a super-pixel, which includes an array of sub-pixels arranged in a quadrilateral pattern; and
the program code for capturing the depth information further comprises program code for:
determining, from the two-dimensional image, color values captured by each sub-pixel within each super-pixel;
determining an amount of captured light in each sub-pixel within each super-pixel;
comparing, in at least one direction, a phase of the captured light in each sub-pixel relative to a phase of light captured in sub-pixels in a plurality of proximate super-pixels;
determining a depth at each super-pixel of the scene based on a phase difference between each super-pixel in the at least one direction; and
storing the depth at each super-pixel as the depth information of the scene.
