# US 10,311,649 B2 · Systems and method for performing depth based image editing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,311,649 B2
- Document ID: US10311649B2
- Status: Granted
- Kind: B2
- Application number: 15/694,700
- Filed: 2017-09-01
- Published: 2017-12-21
- Granted: 2019-06-04
- Earliest claimed: 2012-02-21
- Assignees: FotoNation Limited
- Applicants: FotoNation Limited
- Inventors: Andrew Kenneth John McMahon; Kartik Venkataraman; Robert Mullis
- Web page: https://patentgrep.com/patent/US10311649B2

## CPC

- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/041: Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G06T 2200/21: involving computational photography
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and methods for the manipulation of captured light fields and captured light field image data in accordance with embodiments of the invention are disclosed. In one embodiment of the invention, a system for manipulating captured light field image data includes a processor, a display, a user input device, and a memory, wherein a depth map includes depth information for one or more pixels in the image data, and wherein an image manipulation application configures the processor to display a first synthesized image, receive user input data identifying a region within the first synthesized image, determine boundary data for the identified region using the depth map, receive user input data identifying at least one action, and perform the received action using the boundary data and the captured light field image data.

## Claims

### Claim 1

An array camera system for capturing and manipulating captured image data, comprising:
an array camera comprising a plurality of cameras, where each camera includes separate optics, and a plurality of sensor elements, and each camera is configured to independently capture an image of a scene;
a processor;
a display connected to the processor and configured to display images;
a memory connected to the processor;
software connected to the processor and directs the processor to:
capture images;
store the captured images in the memory; and
generate a depth map having depth information associated with the captured images of the scene; and
an image manipulation application within the memory that directs the processor to:
select a collection of pixels within at least one captured image based upon depth information, wherein the collection of pixels is selected by identifying a boundary of the collection of pixels based upon color and intensity values, and the associated depth information, where the boundary is determined by separately clustering each given pixel based upon depth, color, and intensity;
modify the pixels of the selected collection of pixels of the at least one captured image;
copy the modified pixels of the selected collection of pixels; and
paste the modified pixels of the selected collection of pixels into another image.

### Claim 2 (depends on 1)

The system of claim 1, wherein the image manipulation application directs the processor to modify at least one of: brightness, luminosity, color values and transparency values of the pixels of the selected collection of pixels of the at least one captured image.

### Claim 3 (depends on 1)

The system of claim 1, wherein the collection of pixels is an object.

### Claim 4 (depends on 1)

The system of claim 1, wherein the collection of pixels excludes the pixels of an object.

### Claim 5 (depends on 1)

The system of claim of claim 1, wherein the image manipulation application directs the processor to modify a focus of the selected collection of pixels of at least one captured image by:
identifying a focal plane of the selected collection of pixels using a depth of field; and
refocusing the pixels of the selected collection of pixels using the depth of field, the identified focal plane, and blur modification data, where the blur modification data affects bokeh of the selected collection of pixels.

### Claim 6 (depends on 5)

The system of claim 5, wherein the depth of field is determined by at least one of: utilizing the associated depth information of the at least one captured image, setting the depth of field to a default value, and using a user-provided depth of field through a user input device connected to the processor and configured to provide user generated input data.

### Claim 7 (depends on 1)

The system of claim 1, wherein the image manipulation application further directs the processor to:
display the captured image using the display; and
identify the selected collection of pixels within the displayed image.

### Claim 8 (depends on 1)

The system of claim 1 further comprising a user input device connected to the processor and configured to provide user generated input data, wherein identifying the collection of pixels within the displayed image comprises receiving user input data from the user input device identifying at least one pixel of the selected collection of pixels within the captured image.

### Claim 9 (depends on 1)

The system of claim 1 further comprising an eyeball tracking device, wherein the associated depth of the selected collection of pixels is determined based upon where a user is focusing and dynamically adjusted as the user focuses at different parts of the captured image.

### Claim 10 (depends on 1)

The system of claim 1, further comprising:
a user input device connected to the processor and configured to generate user input data in response to user input;
wherein the image manipulation application further configures the processor to receive user input data.

### Claim 11 (depends on 10)

The system of claim 10, wherein the input device is a touchscreen device.

### Claim 12 (depends on 10)

The system of claim 10, wherein the received input data comprises the selected collection of pixels.

### Claim 13 (depends on 10)

The system of claim 10, wherein the received input data comprises modification of the pixels of the selected collection of pixels.

### Claim 14 (depends on 1)

The system of claim 1, wherein the captured image and associated depth information are stored in memory in an image data file.

### Claim 15 (depends on 14)

The system of claim 14, wherein the image data file also includes at least one auxiliary map selected from a group consisting of a confidence map, an edge map, and a missing pixel map.

### Claim 16 (depends on 1)

The system of claim 1, wherein the image into which the selected collection of pixels having modified pixels is copied is another image of the scene.

### Claim 17

A method for manipulating captured image data using an array camera having a processor, comprising:
capturing, using the processor directed by software, a plurality of images of a scene;
generating, using the processor directed by the software, a depth map having depth information associated with the captured images of the scene;
selecting, using the processor directed by an image manipulation application, a collection of pixels within at least one captured image, based upon the depth information, wherein the collection of pixels is selected by identifying a boundary of the collection of pixels based upon color and intensity values, and the associated depth information, where the boundary is determined by separately clustering each given pixel based upon depth, color, and intensity;
modifying, using the processor directed by an image manipulation application, the pixels of the selected collection of pixels of the at least one captured image;
copying, using the processor directed by an image manipulation application, the modified pixels of the selected collection of pixels; and
pasting, using the processor directed by an image manipulation application, the modified pixels of the selected collection of pixels into another image.

### Claim 18 (depends on 17)

The method of claim 17 wherein modifying, using the processor directed by an image manipulation application, the pixels of the selected collection of pixels comprises modifying at least one of: brightness, luminosity, color values or transparency values of the pixels.

### Claim 19 (depends on 17)

The method of claim 17, wherein the collection of pixels is an object.

### Claim 20 (depends on 17)

The method of claim 17, wherein the collection of pixels excludes an object.

### Claim 21 (depends on 17)

The method of claim of claim 17 further comprising:
identifying, using the processor directed by an image manipulation application, a focal plane of the selected collection of pixels using a depth of field; and
refocusing, using the processor directed by an image manipulation application, the pixels of the selected collection of pixels using the depth of field, the identified focal plane, and blur modification data, where the blur modification data affects bokeh of the selected collection of pixels.

### Claim 22 (depends on 21)

The method of claim 21, wherein the depth of field is determined by at least one of: utilizing the associated depth information of the at least one captured image, setting the depth of field to a default value, and using a user-provided depth of field through a user input device connected to the processor and configured to provide user generated input data.

### Claim 23 (depends on 17)

The method of claim 17, further comprising:
displaying, using the processor directed by an image manipulation application, the captured image using the display; and
identifying, using the processor directed by an image manipulation application, the selected collection of pixels within the displayed image.

### Claim 24 (depends on 23)

The method of claim 23, wherein identifying the collection of pixels within the displayed image comprises receiving user input data from a user input device identifying at least one pixel of the object within the captured image, wherein the user input device is connected to the processor and configured to provide user generated input data.

### Claim 25 (depends on 23)

The method of claim 23, wherein the associated depth of the selected collection of pixels is determined, using an eyeball tracking device, based upon where a user is focusing and dynamically adjusted as the user focuses at different parts of the captured image.

### Claim 26 (depends on 17)

The method of claim 17, further comprising receiving user input data via a user input device connected to the processor and configured to generate user input data in response to user input.

### Claim 27 (depends on 26)

The method of claim 26, wherein the input device is a touchscreen device.

### Claim 28 (depends on 26)

The method of claim 26, wherein the received input data comprises a selected collection of pixels.

### Claim 29 (depends on 26)

The method of claim 26, wherein the received input data comprises modification of the pixels of the selected collection of pixels.

### Claim 30 (depends on 17)

The method of claim 17 further comprising storing the captured image and associated depth information in memory in an image data file.

### Claim 31 (depends on 30)

The method of claim 30, wherein the image data file also includes at least one auxiliary map selected from a group consisting of a confidence map, an edge map, and a missing pixel map.

### Claim 32 (depends on 17)

The method of claim 17, wherein the other image is an image of the scene.