# US 2017/0374345 A1 · Method and apparatus for creating a pair of stereoscopic images using least one lightfield camera

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0374345 A1
- Document ID: US20170374345A1
- Status: Published application
- Kind: A1
- Application number: 15/631,009
- Filed: 2017-06-23
- Published: 2017-12-28
- Earliest claimed: 2016-06-23
- Assignees: InterDigital CE Patent Holdings, SAS
- Applicants: Thomson Licensing
- Inventors: Jurgen STAUDER; Benoit VANDAME; Cedric THEBAULT
- Web page: https://patentgrep.com/patent/US20170374345A1

## CPC

- H04N 13/128: Adjusting depth or disparity
- G06T 3/4015: Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
- G06T 7/85: {Stereo camera calibration}
- G06T 15/06: Ray-tracing
- G06T 15/205: {Image-based rendering}
- H04N 13/178: Metadata, e.g. disparity information
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 13/246: Calibration of cameras
- H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
- H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
- G06T 2207/10052: Images from lightfield camera
- G06T 2207/20228: Disparity calculation for image-based rendering
- G06T 2207/30244: Camera pose
- G06T 2215/16: Using real world measurements to influence rendering
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method for creating a pair of stereoscopic images is described. The method includes using at least one lightfield camera which receives respective required camera parameters for a left view and a right view. The required camera parameters define a theoretical stereo image pair to acquire respective actual camera parameters for respective sub aperture images that are generated based on the captured image by the lightfield cameras, to determine a best matching sub aperture image for the left view by comparing the required camera parameter for the left view with the actual camera parameters for the respective sub aperture images, to determine a best matching sub aperture image for right view by comparing the required camera parameter for right view with the actual camera parameters for the respective sub aperture images, and to associate the best matching sub aperture image for left view and the best matching sub aperture image for right view as a pair of stereoscopic images.

## Claims

### Claim 1

A method for creating a pair of stereoscopic images using at least one lightfield camera, comprising:
receiving respective required camera parameter for a left view and a right view, wherein the required camera parameter defines a theoretical stereo image pair;
acquiring respective actual camera parameters for respective sub aperture images that are generated based on an image captured by the lightfield camera;
determining a best matching sub aperture image for the left view by comparing the required camera parameter for the left view with the actual camera parameters for the respective sub aperture images and a best matching sub aperture image for the right view by comparing the required camera parameter for the right view with the actual camera parameters for the respective sub aperture images; and
associating the best matching sub aperture image for the left view and the best matching sub aperture image for the right view as a pair of stereoscopic images.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the actual camera parameter for the sub aperture image is estimated by:
receiving an image of an object within a scene in the image captured by the lightfield camera and acquiring scene-world-coordinates of respective feature points of the object;
generating plurality of sub aperture images from the received image of the object;
estimating the actual camera parameters based on the scene-world-coordinates of respective feature points of the object and corresponding pixel coordinates of the scene-world-coordinates of respective feature points in the sub aperture image.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the actual camera parameter and the required camera parameter comprises rotation matrix R and translation vector t, respectively.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the actual camera parameter of the determined best matching sub aperture image for left view has the closest value to the required camera parameter for left view and the actual camera parameter of the determined best matching sub aperture image for right view has the closest value to the required camera parameter for right view.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the lightfield camera including a lightfield camera for left view and a lightfield camera for right view, the determining further including:
determining the best matching sub aperture image for left view by comparing the required camera parameter for left view with the respective actual camera parameters for respective sub aperture images generated based on the image captured by the lightfield camera for left view; and
determining the best matching sub aperture image for right view by comparing the required camera parameter for right view with the respective actual camera parameters for respective sub aperture images generated based on the image captured by the lightfield camera for right view.

### Claim 6 (depends on 4)

The method according to claim 4, the associating further comprising:
calculating a transfer matrix DLrem that is able to transfer the actual camera parameter of a remaining sub aperture image for left view into the actual camera parameter of the best matching sub aperture image for left view, wherein the DLrem is calculated for respective sub aperture images for left view;
calculating a transfer matrix DRrem that is able to transfer the actual camera parameter of a remaining sub aperture image for right view into the actual camera parameter of the best matching sub aperture image for right view, wherein the DRrem is calculated for respective sub aperture images for right view; and
associating the remaining sub aperture image for left view to the remaining sub aperture image for right view of which DRrem has the closest value to the DLrem,
wherein the remaining sub aperture images are sub aperture images other than the best matching sub aperture images.

### Claim 7 (depends on 1)

The method according to claim 1, further comprising:
sorting a remaining sub aperture image that is other than the best matching sub aperture image either to remaining sub aperture image for left view or remaining sub aperture image for right view, the sorting comprising:
calculating a DLrem that is able to transfer the actual camera parameter of the remaining sub aperture image into the actual camera parameter of the best matching sub aperture image for left view;
calculating a DRrem that is able to transfer the actual camera parameter of the remaining sub aperture image into the actual camera parameter of the best matching sub aperture image for right view; and
comparing the DLrem and DRrem and sorting the remaining sub aperture image to the remaining sub aperture image for left view if DLrem is smaller than DRrem, otherwise sorting to the remaining sub aperture image for right view.

### Claim 8 (depends on 7)

The method according to claim 7, further comprising;
comparing the DLrem and the DRrem and if DLrem is equivalent to DRrem, sorting neither to the remaining sub aperture image for left view nor to the remaining sub aperture image for right view.

### Claim 9 (depends on 7)

The method according to claim 7, further comprising;
associating the remaining sub aperture image sorted to the remaining sub aperture image for left view to the remaining sub aperture image sorted to the remaining sub aperture image for right view of which DRrem has the closest value to the DLrem of the remaining sub aperture image sorted to the remaining sub aperture image for left view.

### Claim 10

A device for creating a pair of stereoscopic images comprising a processor configured to:
receive respective required camera parameter for a left view and a right view, wherein the required camera parameter defines a theoretical stereo image pair;
acquire respective actual camera parameters for respective sub aperture images that are generated based on an image captured by a lightfield camera;
determine a best matching sub aperture image for the left view by comparing the required camera parameter for the left view with the actual camera parameters for the respective sub aperture images and a best matching sub aperture image for the right view by comparing the required camera parameter for the right view with the actual camera parameters for the respective sub aperture images; and
associate the best matching sub aperture image for the left view and the best matching sub aperture image for the right view as a pair of stereoscopic images.

### Claim 11 (depends on 10)

The device according to claim 10, wherein the actual camera parameter for the sub aperture image is estimated by:
receiving an image of an object within a scene captured by the lightfield camera and acquiring scene-world-coordinates of respective feature points of the object;
generating plurality of sub aperture images from the received image of the object;
estimating the actual camera parameter based on the scene-world-coordinates of respective feature points of the object and corresponding pixel coordinates of the scene-world-coordinates of respective feature points in the sub aperture image.

### Claim 12 (depends on 1)

Computer program product downloadable from a communication network and/or recorded on a medium readable by computer and/or executable by a processor, comprising program code instructions for implementing a method according to claim 1.

### Claim 13 (depends on 1)

Non-transitory computer-readable medium comprising a computer program product recorded thereon and capable of being run by a processor, including program code instructions for implementing a method according to claim 1.
