# US 2023/0410341 A1 · Passive and single-viewpoint 3d imaging system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410341 A1
- Document ID: US20230410341A1
- Status: Published application
- Kind: A1
- Application number: 18/307,387
- Filed: 2023-04-26
- Published: 2023-12-21
- Earliest claimed: 2019-05-01
- Assignees: William Marsh Rice University; Carnegie Mellon University
- Applicants: William Marsh Rice University; Carnegie Mellon University
- Inventors: Yicheng Wu; Vivek Boominathan; Huaijin Chen; Aswin C. Sankaranarayanan; Ashok Veeraraghavan
- Web page: https://patentgrep.com/patent/US20230410341A1

## CPC

- G06T 7/55: from multiple images
- G06T 7/571: from focus
- G06T 7/529: from texture
- H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
- H04N 13/122: Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence)
- H04N 13/128: Adjusting depth or disparity
- H04N 13/229: using lenticular lenses, e.g. arrangements of cylindrical lenses
- G06T 2200/08: involving all processing steps from image acquisition to 3D model generation
- G06T 2207/20084: Artificial neural networks [ANN]
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method for a passive single-viewpoint 3D imaging system comprises capturing an image from a camera having one or more phase masks. The method further includes using a reconstruction algorithm, for estimation of a 3D or depth image.

## Claims

### Claim 1

A method for a passive single-viewpoint 3D imaging system, comprising:
capturing an image from a camera, the camera having one or more phase masks;
using a reconstruction algorithm, for estimation of a 3D or depth image,
wherein the reconstruction algorithm comprises:
building a differentiable forward model with an optical layer and a reconstruction network,
wherein in the optical layer, a physics-based model simulates a coded image captured on a sensor with a phase mask, and
wherein in the reconstruction network, a neural network produces a depth map from the coded image.

### Claim 2 (depends on 1)

The method of claim 1, wherein the phase masks are separately fabricated and inserted between a camera lens and an image sensor.

### Claim 3 (depends on 1)

The method of claim 1 wherein the phase masks are integrated into a camera lens assembly and fabricated as a single optimized modified optical assembly.

### Claim 4 (depends on 1)

The method of claim 1, wherein the phase masks are integrated a the camera lens assembly and fabricated as either single or multiple optimized modified optical assembly.

### Claim 5 (depends on 1)

The method of claim 1, wherein a plurality of parameters of the neural network are trained using a red green blue depth (RGBD) dataset to optimize a height map of the phase mask and parameters in the neural network.

### Claim 6 (depends on 5)

The method in claim 5, wherein the training process comprises:
calibrating a plurality of point spread functions (PSFs) at different depths;
simulating the coded image with the calibrated PSFs to fine-tune the neural network; and
recovering a corresponding depth map by capturing and passing an image through the neural network, wherein the neural network is a U-Net.

### Claim 7 (depends on 1)

The method of claim 1, wherein the reconstruction algorithm is based on optimization of an energy function and wherein the reconstruction algorithm returns both the 3D or depth image and a 2D intensity image.

### Claim 8 (depends on 6)

The method of claim 6, wherein the optical layer takes in an all-in-focus image and the corresponding depth map as inputs and outputs a physically accurate coded intensity image.

### Claim 9 (depends on 5)

The method of claim 5, further comprising optimizing, both parameters of the optical layer and the reconstruction network, based on a loss defined between an estimated depth and a ground truth depth.

### Claim 10 (depends on 1)

The method of claim 1, further comprising fabricating an optimized phase mask using photolithography method.

### Claim 11 (depends on 10)

The method of claim 10, further comprising printing the optimized phase mask on a glass substrate and cutting the glass substrate to a smaller size to fit into an aperture plane of the camera.

### Claim 12

A system for a passive and single-viewpoint 3D imaging system, comprising:
a camera; and
an optimized phase mask placed in an aperture plane of the camera,
wherein the optimized phase mask and a reconstruction algorithm allow the system to estimate a 3D or depth image,
wherein the reconstruction algorithm comprises a computer processor configured to:
build a differentiable forward model with an optical layer comprising a physics-based model to simulate coded image captured on a sensor with the phase mask and a neural network to produce a depth map from the coded image;
train a framework with a synthetic red green blue depth (RGBD) dataset to optimize a height map of the phase mask and parameters in the neural network;
calibrate a plurality of point spread functions (PSFs) at different depths;
simulate the coded image with the calibrated PSFs to fine-tune the neural network; and
recover a corresponding depth map by capturing and passing an image through the neural network.

### Claim 13 (depends on 12)

The system of claim 12, wherein the optimized phase mask is fabricated using a 3D printer using photolithography.

### Claim 14 (depends on 12)

The system of claim 12, wherein the optical layer takes in an all-in-focus image and the corresponding depth map as inputs and outputs a physically accurate coded intensity image.

### Claim 15 (depends on 12)

The system of claim 12, wherein both parameters of the optical layer and the reconstruction network are optimized based on a loss defined between an estimated depth and a ground truth depth.

### Claim 16 (depends on 12)

The system of claim 12, wherein the optimized phase mask is printed on a glass substrate and the glass substrate cut to a smaller size to fit into the aperture plane of the camera.

### Claim 17

A non-transitory computer readable medium storing instructions, the instructions executable by a computer processor and comprising functionality for:
building a differentiable forward model with an optical layer and a reconstruction network,
wherein in the optical layer, a physics-based model simulates a coded image captured on a sensor with a phase mask, and
wherein in the reconstruction network, a neural network produces a depth map from the coded image;
training a framework with a red green blue depth (RGBD) dataset to optimize a height map of the phase mask and parameters in the neural network;
fabricating an optimal phase mask using photolithography method and placing in an aperture plane of a camera;
calibrating a plurality of point spread functions (PSFs) at different depths;
simulating the coded image with the calibrated PSFs to fine-tune the neural network; and
recovering a corresponding depth map by capturing and passing an image through the neural network.

### Claim 18 (depends on 17)

The non-transitory computer readable medium of claim 17, the instructions further comprising functionality for optimizing, both parameters of the optical layer and the reconstruction network, based on a loss defined between an estimated depth and a ground truth depth.
