# US 2023/0410320 A1 · Method for processing a light field image delivering a super-rays representation of a light field image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410320 A1
- Document ID: US20230410320A1
- Status: Published application
- Kind: A1
- Application number: 18/240,856
- Filed: 2023-08-31
- Published: 2023-12-21
- Earliest claimed: 2016-12-01
- Assignees: InterDigital CE Patent Holdings, SAS
- Applicants: InterDigital CE Patent Holdings, SAS
- Inventors: Matthieu Hog; Neus Sabater; Christine Guillemot
- Web page: https://patentgrep.com/patent/US20230410320A1

## CPC

- G06T 7/174: involving the use of two or more images
- G06T 7/11: Region-based segmentation
- G06T 2207/10052: Images from lightfield camera
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An apparatus and method for processing a light field comprising a plurality of views of a 3D scene includes obtaining a super-ray representation for the light field, wherein the super-ray representation comprises a plurality of super-rays, wherein each super-ray groups pixels corresponding to a set of light rays having a similar color from each of the plurality of views, and wherein the set of light rays for each super-ray originates from one area of the 3D scene; and processing the light field by processing the super-ray representation for the light field, wherein processing the light field comprises at least one of editing the light field, segmenting the light field, and refocusing the light field.

## Claims

### Claim 1

A method for processing a light field comprising a plurality of views of a 3D scene, the method comprising:
obtaining a super-ray representation for the light field, wherein the super-ray representation comprises a plurality of super-rays, wherein each super-ray groups pixels corresponding to a set of light rays having a similar color from each of the plurality of views, and wherein the set of light rays for each super-ray originates from one area of the 3D scene; and
processing the light field by processing the super-ray representation for the light field, wherein processing the light field comprises at least one of editing the light field, segmenting the light field, and refocusing the light field.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
projecting all light rays associated with a first super-ray of the plurality of super-rays on a first view of the plurality of views; and
identifying as a first centroid a spatial coordinate associated with an average spatial position of the spatial positions of the light rays associated with the first super-ray as projected on the first view.

### Claim 3 (depends on 1)

The method of claim 1, further comprising computing an average color for all pixels associated with a first super-ray and assigning the average color to the first super-ray in the super-ray representation of the light field.

### Claim 4 (depends on 1)

The method of claim 1, wherein the light field comprises L neighborhoods, and wherein the method further comprises generating L super-rays, one super-ray for each of the L neighborhoods.

### Claim 5 (depends on 1)

The method of claim 1, wherein a first light ray of the plurality of super-rays is associated with a first pixel, wherein the first light ray has a first depth, and wherein each ray of the first super-ray is re-projected at the first depth on a first view of the plurality of views.

### Claim 6 (depends on 1)

The method of claim 1, wherein the plurality of views comprises at least M views, the method further comprising:
selecting a first pixel in a first view of the plurality of views;
generating a first super-ray for the first pixel comprising de-projecting the first pixel to identify a corresponding first point in the 3D scene, and, for each Nth view of the plurality of views from a second view through an Mth view, where N and M are integers, the generating further comprises:
re-projecting the corresponding first point in the 3D scene to an Nth view of the plurality of views to identify a corresponding Nth pixel in the Nth view;
identifying an Nth set of pixels having a similar color to the corresponding Nth pixel, wherein each pixel of the Nth set of pixels is within an Nth neighborhood of the corresponding Nth pixel; and
grouping the Nth set of pixels into the first super-ray.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
projecting all light rays associated with the first super-ray on the first view; and
identifying as a first centroid of the first neighborhood a spatial coordinate associated with an average spatial position of the spatial positions of the light rays associated with the first super-ray as projected on the first view.

### Claim 8 (depends on 6)

The method of claim 6, further comprising:
selecting a second pixel in the first view of the plurality of views;
generating a second super-ray for the first pixel comprising de-projecting the second pixel to identify a corresponding second point in the 3D scene, and, for each Nth view of the plurality of views from the second view through the Mth view, the generating further comprises:
re-projecting the corresponding second point in the 3D scene to an Nth view of the plurality of views to identify a corresponding Nth pixel in the Nth view;
identifying an Nth set of pixels having a similar color to the corresponding Nth pixel, wherein each pixel of the Nth set of pixels is within an Nth neighborhood of the corresponding Nth pixel; and
grouping the Nth set of pixels into the second super-ray; and
processing the light field by processing at least the first super-ray and the second super-ray.

### Claim 9 (depends on 6)

The method of claim 6, wherein de-projecting and re-projecting are performed according to intrinsic and extrinsic calibration parameters.

### Claim 10

An electronic device configured to process a light field image comprising a plurality of views of a 3D scene, the electronic device comprising at least one processor and at least one memory having stored instructions operative, when executed by the at least one processor, to cause the electronic device to:
obtain a super-ray representation for the light field, wherein the super-ray representation comprises a plurality of super-rays, wherein each super-ray groups pixels corresponding to a set of light rays having a similar color from each of the plurality of views, and wherein the set of light rays for each super-ray originates from one area of the 3D scene; and
process the light field by processing the super-ray representation for the light field, wherein processing the light field comprises at least one of editing the light field, segmenting the light field, and refocusing the light field.

### Claim 11 (depends on 10)

The electronic device of claim 10, wherein the instructions are further operative to cause the electronic device to:
project all light rays associated with a first super-ray of the plurality of super-rays on a first view of the plurality of views; and
identify as a first centroid a spatial coordinate associated with an average spatial position of the spatial positions of the light rays associated with the first super-ray as projected on the first view.

### Claim 12 (depends on 10)

The electronic device of claim 10, wherein the instructions are further operative to cause the electronic device to compute an average color for all pixels associated with a first super-ray and assign the average color to the first super-ray in the super-ray representation of the light field.

### Claim 13 (depends on 10)

The electronic device of claim 10, wherein the light field comprises L neighborhoods, and wherein the method further comprises generating L super-rays, one super-ray for each of the L neighborhoods.

### Claim 14 (depends on 10)

The electronic device of claim 10, wherein a first light ray of the plurality of super-rays is associated with a first pixel, wherein the first light ray has a first depth, and wherein each ray of the first super-ray is re-projected at the first depth on a first view of the plurality of views.

### Claim 15 (depends on 10)

The electronic device of claim 10, wherein the plurality of views comprises at least M views, wherein the instructions are further operative to cause the electronic device to:
select a first pixel in a first view of the plurality of views;
generate a first super-ray for the first pixel comprising de-projecting the first pixel to identify a corresponding first point in the 3D scene, and, for each Nth view of the plurality of views from the second view through the Mth view, where N and M are integers, the generating further comprises:
re-projecting the corresponding first point in the 3D scene to an Nth view of the plurality of views to identify a corresponding Nth pixel in the Nth view;
identifying an Nth set of pixels having a similar color to the corresponding Nth pixel, wherein each pixel of the Nth set of pixels is within an Nth neighborhood of the corresponding Nth pixel; and
grouping the Nth set of pixels into the first super-ray.

### Claim 16 (depends on 15)

The electronic device of claim 15, wherein the instructions are further operative to cause the electronic device to:
project all light rays associated with the first super-ray of the plurality of super-rays on the first view of the plurality of views; and
identify as a first centroid a spatial coordinate associated with an average spatial position of the spatial positions of the light rays associated with the first super-ray as projected on the first view.

### Claim 17 (depends on 15)

The electronic device of claim 15, wherein the instructions are further operative to cause the electronic device to:
select a second pixel in the first view of the plurality of views;
generate a second super-ray for the first pixel comprising de-projecting the second pixel to identify a corresponding second point in the 3D scene, and, for each Nth view of the plurality of views from the second view through the Mth view, where N and M are integers, the generating further comprises:
re-projecting the corresponding second point in the 3D scene to an Nth view of the plurality of views to identify a corresponding Nth pixel in the Nth view;
identifying an Nth set of pixels having a similar color to the corresponding Nth pixel, wherein each pixel of the Nth set of pixels is within an Nth neighborhood of the corresponding Nth pixel; and
grouping the Nth set of pixels into the second super-ray; and
process the light field by processing at least the first super-ray and the second super-ray.

### Claim 18 (depends on 15)

The electronic device of claim 15, wherein de-projecting and re-projecting are performed according to intrinsic and extrinsic calibration parameters.
