# US 10,119,809 B2 · Simulating multi-camera imaging systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,119,809 B2
- Document ID: US10119809B2
- Status: Granted
- Kind: B2
- Application number: 14/749,786
- Filed: 2015-06-25
- Published: 2016-08-18
- Granted: 2018-11-06
- Earliest claimed: 2015-02-16
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Ramkumar Narayanswamy; Ginni Grover; Ram C. Nalla
- Web page: https://patentgrep.com/patent/US10119809B2

## CPC

- G01B 11/245: using a plurality of fixed, simultaneously operating transducers ({G01B11/2408 - G01B11/2425, } G01B11/255 take precedence)
- G06T 15/00: Three-dimensional [3D] image rendering
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Camera simulation is extended to multi-camera systems. One aspect relates combine computer-graphics, single-camera simulation, multi-camera image signal processor (ISP) are combined to simulate and optimize a multi-camera system. Scene modeling is performed using synthetic computer generated scenes. The computer-graphics images are generated, taking into account the camera-intrinsic and camera-extrinsic parameters.

## Claims

### Claim 1

A method comprising:
generating a 2D projection of a virtual scene using a computer;
converting the 2D projection into an optical image using an image propagation method via the computer;
processing the virtual scene to form a de-magnified image of the virtual scene on a sensor plane and imparting lens aberration and diffraction effects to the virtual scene on the sensor plane to produce a raw image via the computer;
processing the raw image using a camera image signal processing pipe to create virtual images that correspond to images from different real cameras;
processing the virtual images in a multi-camera image signal processing pipe;
developing a three-dimensional model of the virtual scene and then to render the virtual scene for each of a plurality of virtual cameras of a virtual multi-camera system using ray tracing; and
providing field of view and parallax for each of the plurality of virtual cameras of said virtual multi-camera system, according to position of each of said plurality of virtual cameras.

### Claim 2 (depends on 1)

The method of claim 1 further including simulating camera module imperfections for a plurality of camera modules, to produce computer generated images with the imperfections for said plurality of camera modules; and
using a multi-camera image signal processor to extract depth and disparity for said computer generated images from each of said plurality of camera modules.

### Claim 3 (depends on 2)

The method of claim 2 including simulating the lens aberration and diffraction effects on the virtual scene in a frequency domain and then converting to a spatial domain.

### Claim 4 (depends on 2)

The method of claim 2 including modeling a lensing effect as a blurring operation using a multiplication in a frequency domain.

### Claim 5 (depends on 4)

The method of claim 4 including computing blur by windowing a small section of the virtual scene.

### Claim 6 (depends on 2)

The method of claim 2 including applying photon capture and electronic effects of a sensor for each of said plurality of camera modules.

### Claim 7 (depends on 2)

The method of claim 2 including generating images that account for camera-intrinsic and extrinsic parameters.

### Claim 8 (depends on 2)

The method of claim 2 including simulating said virtual multi-camera system by accounting for locations of said plurality of camera modules.

### Claim 9 (depends on 2)

The method of claim 2 including modeling the lens aberration and diffraction effects.

### Claim 10 (depends on 8)

The method of claim 8 including optimizing for the position of each of said plurality of virtual cameras.

### Claim 11

One or more non-transitory computer readable media storing instructions executed by a processor to perform a sequence comprising:
generating a 2D projection of a virtual scene using a computer;
converting the 2D projection into an optical image using an image propagation method via the computer;
processing the virtual scene to form a de-magnified image of the virtual scene on a sensor plane and imparting lens aberration and diffraction effects to the virtual scene on the sensor plane to produce a raw image via the computer;
processing the raw image using a camera image signal processing pipe to create virtual images that correspond to images from different real cameras;
processing the virtual images in a multi-camera image signal processing pipe;
developing a three-dimensional model of the virtual scene and then to render the virtual scene for each of a plurality of virtual cameras of a virtual multi-camera system using ray tracing; and
providing field of view and parallax for each of the plurality of virtual cameras of said virtual multi-camera system, according to position of each of said plurality of virtual cameras.

### Claim 12 (depends on 11)

The media of claim 11, said sequence further including:
simulating camera module imperfections for a plurality of camera modules, to produce computer generated images with the imperfections for said plurality of camera modules; and
using a multi-camera image signal processor to extract depth and disparity for said computer generated images from each of said plurality of camera modules.

### Claim 13 (depends on 12)

The media of claim 12, said sequence including simulating the lens aberration and diffraction effects on the virtual scene in a frequency domain and then converting to a spatial domain.

### Claim 14 (depends on 12)

The media of claim 12, said sequence including modeling a lensing effect as a blurring operation using a multiplication in a frequency domain.

### Claim 15 (depends on 14)

The media of claim 14, said sequence including computing blur by windowing a small section of the virtual scene.

### Claim 16 (depends on 12)

The media of claim 12, said sequence including applying photon capture and electronic effects of a sensor for each of said plurality of camera modules.

### Claim 17 (depends on 12)

The media of claim 12, said sequence including generating images that account for camera-intrinsic and extrinsic parameters.

### Claim 18 (depends on 12)

The media of claim 12, said sequence including simulating said virtual multi-camera system by accounting for locations of said plurality of camera modules.

### Claim 19 (depends on 12)

The media of claim 12, said sequence including modeling the lens aberration and diffraction effects.

### Claim 20 (depends on 18)

The media of claim 18, said sequence including optimizing for the position of each of said plurality of virtual cameras.

### Claim 21

An apparatus comprising:
a hardware processor to generate a 2D projection of a virtual scene using a computer, convert the 2D projection into an optical image using an image propagation method via the computer, process the virtual scene to form a de-magnified image of the virtual scene on a sensor plane and
impart lens aberration and diffraction effects to the virtual scene on the sensor plane to produce a raw image via the computer, process the raw image using a camera image signal processing pipe to create virtual images that correspond to images from different real cameras, develop a three-dimensional model of the virtual scene and then to render the virtual scene for each of a plurality of virtual cameras of a virtual multi-camera system using ray tracing, and provide field of view and parallax for each of the plurality of virtual cameras of said virtual multi-camera system, according to position of each of said plurality of virtual cameras; and
a storage coupled to said processor.

### Claim 22 (depends on 21)

The apparatus of claim 21 said processor to simulate camera module imperfections for a plurality of camera modules, to produce computer generated images with the imperfections for said plurality of camera modules, and use a multi-camera image signal processor to extract depth and disparity for said computer generated images from each of said plurality of camera modules.

### Claim 23 (depends on 22)

The apparatus of claim 22, said processor to simulate the lens aberration and diffraction effects on the virtual scene in the frequency domain and then converting to the spatial domain.

### Claim 24 (depends on 22)

The apparatus of claim 22, said processor to model a lensing effect as a blurring operation using a multiplication in the frequency domain.

### Claim 25 (depends on 24)

The apparatus of claim 24, said processor to compute blur by windowing a small section of the virtual scene.

### Claim 26 (depends on 22)

The apparatus of claim 22, said processor to apply photon capture and electronic effects of a sensor for each of said plurality of camera modules.

### Claim 27 (depends on 22)

The apparatus of claim 22, said processor to generate images that account for camera-intrinsic and extrinsic parameters.

### Claim 28 (depends on 22)

The apparatus of claim 22, said processor to simulate said virtual multi-camera system by accounting for the locations of said plurality of camera modules.

### Claim 29 (depends on 24)

The apparatus of claim 24, said processor to include an inverse fast fourier transform to inverse transform results of said multiplication.

### Claim 30 (depends on 22)

The apparatus of claim 22, said apparatus to include a display communicatively coupled to the processor.

### Claim 31 (depends on 22)

The apparatus of claim 22, said apparatus to include a battery coupled to the processor.