# US 2015/0312553 A1 · Capturing and relighting images using multiple devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0312553 A1
- Document ID: US20150312553A1
- Status: Published application
- Kind: A1
- Application number: 14/634,659
- Filed: 2015-02-27
- Published: 2015-10-29
- Earliest claimed: 2012-12-04
- Assignees: Lytro, Inc.
- Applicants: Lytro, Inc.
- Inventors: Yi-Ren Ng; Chia-Kai Liang; Kurt Barton Akeley; Bennett Wilburn
- Web page: https://patentgrep.com/patent/US20150312553A1

## CPC

- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 5/144: {Movement detection (for video coding H04N19/503; analysis of motion in general G06T7/20)}
- H04N 13/167: Synchronising or controlling image signals
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- H04N 23/56: provided with illuminating means
- H04N 23/66: Remote control of cameras or camera parts, e.g. by remote control devices
- H04N 23/74: by influencing the scene brightness using illuminating means
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A system and method are provided for coordinating image capture using multiple devices, including for example multiple image capture devices (cameras), multiple lighting devices (flash), and/or the like. In at least one embodiment, the system of the present invention is configured to collect image information from multiple image capture devices, such as cameras, and/or to collect multiple images having different lighting configurations. The collected image data can be processed to generate various effects, such as relighting, parallax, refocusing, and/or three-dimensional effects, and/or to introduce interactivity into the image presentation. In at least one embodiment, the system of the present invention is implemented using any combination of any number of image capture device(s) and/or flash (lighting) device(s), which may be equipped to communicate with one another via any suitable means, such as wirelessly. A communication protocol is established for enabling such communication, such as WiFi, Bluetooth, or the like.

## Claims

### Claim 1

A method for capturing multiple images using multiple devices, comprising:
at a controller, determining a pattern for capture of multiple images using multiple devices;
establishing communication, via a communication medium, between the controller and at least one image capture device;
establishing communication, via the communication medium, between the controller and at least one illumination device;
transmitting at least one message, via the communication medium, from the controller to the at least one image capture device and to the at least one illumination device to cause the at least one image capture device to capture a plurality of images of a subject, the images having a plurality of illumination configurations;
storing the captured plurality of images at a storage device; and
generating output from at least one of the stored images.

### Claim 2 (depends on 1)

The method of claim 1, wherein the communication medium comprises a wireless communication medium.

### Claim 3 (depends on 1)

The method of claim 1, wherein establishing communication between the controller and at least one image capture device comprises establishing communication between the controller and a plurality of image capture devices.

### Claim 4 (depends on 1)

The method of claim 1, wherein establishing communication between the controller and at least one illumination device comprises establishing communication between the controller and a plurality of illumination devices.

### Claim 5 (depends on 1)

The method of claim 1, wherein:
each image capture device comprises a camera; and
each illumination device comprises a flash unit.

### Claim 6 (depends on 1)

The method of claim 1, wherein transmitting at least one message comprises:
transmitting at least one message to cause a movable illumination device to successively illuminate the subject from each of a plurality of different positions; and
transmitting at least one message to cause at least one image capture device to capture at least one image in synchrony with each illumination of the subject.

### Claim 7 (depends on 6)

The method of claim 6, wherein transmitting at least one message further comprises transmitting at least one message to cause the movable illumination device to automatically move from one of the positions to another of the positions.

### Claim 8 (depends on 6)

The method of claim 6, wherein transmitting at least one message further comprises transmitting at least one message to cause the movable illumination device to display output instructing a user to move the illumination device from one of the positions to another of the positions.

### Claim 9 (depends on 1)

The method of claim 1, wherein transmitting at least one message comprises:
transmitting at least one message to cause a plurality of illumination devices at different positions with respect to the subject to successively illuminate the subject at each of the positions; and
transmitting at least one message to cause at least one image capture device to capture at least one image in synchrony with each illumination of the subject.

### Claim 10 (depends on 1)

The method of claim 1, wherein:
transmitting at least one message to the at least one image capture device comprises transmitting at least one message indicating at least one of a time and position for image capture; and
transmitting at least one message to the at least one illumination device comprises transmitting at least one message indicating at least one of a time and position for illumination of the subject.

### Claim 11 (depends on 1)

The method of claim 1, further comprising generating output to instruct a user to move at least one of an image capture device and an illumination device to a specified position.

### Claim 12 (depends on 11)

The method of claim 11, wherein generating output to instruct a user comprises generating at least one of:
visual output;
audio output; and
haptic output.

### Claim 13 (depends on 11)

The method of claim 11, further comprising:
detecting user movement of at least one of an image capture device and an illumination device;
comparing the detected movement with instructions in the generated output; and
responsive to the comparison, outputting feedback to the user.

### Claim 14 (depends on 1)

The method of claim 1, further comprising:
detecting a new position of an illumination device caused by user movement of the illumination device; and
responsive to the new position being within a predefined tolerance of a predefined position for an image capture event:
transmitting a message to cause the illumination device to illuminate the subject; and
transmitting a message to cause the at least one image capture device to capture an image of the subject in synchrony with the illumination of the subject.

### Claim 15 (depends on 1)

The method of claim 1, further comprising:
detecting a new position of an image capture device caused by user movement of the image capture device; and
responsive to the new position being within a predefined tolerance of a predefined position for an image capture event:
transmitting a message to cause an illumination device to illuminate the subject; and
transmitting a message to cause the at least one image capture device to capture an image of the subject in synchrony with the illumination of the subject.

### Claim 16 (depends on 1)

The method of claim 1, further comprising:
detecting user activation of an illumination device; and
transmitting a message to cause the at least one image capture device to capture an image of the subject in synchrony with the illumination of the subject.

### Claim 17 (depends on 16)

The method of claim 16, further comprising determining a position of the user-activated illumination device.

### Claim 18 (depends on 1)

The method of claim 1, wherein determining the pattern for capture of multiple images comprises:
detecting a series of user movements of at least one of the illumination devices, defining a movement path;
identifying a plurality of positions along the movement path as being suitable for illumination of the subject; and
storing a representation of the positions;
and wherein transmitting the at least one message to cause the at least one image capture device to capture a plurality of images of a subject comprises transmitting at least one message to cause image capture when the illumination device is at one of the identified positions.

### Claim 19 (depends on 1)

The method of claim 1, wherein determining the pattern for capture of multiple images comprises:
detecting a series of user movements of at least one of the image capture devices, defining a movement path;
identifying a plurality of positions along the movement path as being suitable for image capture;
storing a representation of the positions; and
and wherein transmitting the at least one message to cause the at least one image capture device to capture a plurality of images of a subject comprises transmitting at least one message to cause image capture when the image capture device is at one of the identified positions.

### Claim 20 (depends on 1)

The method of claim 1, wherein the controller is a separate device from the at least one image capture device and the at least one illumination device.

### Claim 21 (depends on 1)

The method of claim 1, wherein each of the controller, the at least one image capture device, and the at least one illumination device comprises at least one selected from the group consisting of:
a smartphone;
a tablet computer; and
a mobile computing device.

### Claim 22 (depends on 1)

The method of claim 1, wherein storing the captured plurality of images at a storage device comprises transmitting the captured image to a remote server for storage at a storage device associated with the remote server.

### Claim 23 (depends on 1)

The method of claim 1, wherein transmitting at least one message comprises transmitting at least one message to cause the at least one image capture device to capture a plurality of light-field images.

### Claim 24 (depends on 1)

The method of claim 1, further comprising:
determining a position of the at least one image capture device and the at least one illumination device during an image capture; and
storing the determined position.

### Claim 25 (depends on 24)

The method of claim 24, wherein storing the determined position comprises storing the determined position as metadata to accompany at least one stored image.

### Claim 26 (depends on 1)

The method of claim 1, wherein generating output from at least one of the stored images comprises:
generating an image by combining at least two of the stored images; and
outputting the generated image.

### Claim 27 (depends on 26)

The method of claim 26, wherein combining at least two of the stored images comprises generating a relighted image from a weighted combination of at least two of the stored images.

### Claim 28 (depends on 1)

The method of claim 1, wherein generating output from at least one of the stored images comprises:
receiving user input to specify a weighting associated with at least two of the stored images;
generating an image by combining at least two of the stored images according to the specified weighting; and
outputting the generated image.

### Claim 29 (depends on 1)

The method of claim 1, wherein generating output from at least one of the stored images comprises:
receiving user input to specify a position for at least one virtual light source;
determining, based on the specified position for at least one virtual light source, a weighting associated with at least two of the stored images; and
generating a relighted image from a combination of at least two of the stored images, wherein the combination is weighted according to the determined weighting.

### Claim 30 (depends on 29)

The method of claim 29, wherein receiving user input to specify a position for at least one virtual light source comprises receiving user input interacting with a displayed distribution graph representing weightings and positions for virtual light sources.

### Claim 31 (depends on 29)

The method of claim 29, wherein receiving user input to specify a position for at least one virtual light source comprises receiving user input dragging points on a displayed distribution graph representing weightings and positions for virtual light sources.

### Claim 32 (depends on 29)

The method of claim 29, wherein receiving user input to specify a position for at least one virtual light source comprises receiving user input to scroll at least one displayed visual element, wherein the position for at least one virtual light source is specified based on the scroll position of the at least one displayed visual element.

### Claim 33 (depends on 1)

The method of claim 1, wherein generating output from at least one of the stored images comprises:
generating a light-field picture by combining at least two of the stored images; and
outputting an image generated from the light-field picture.

### Claim 34 (depends on 1)

The method of claim 1, further comprising, prior to establishing communication between the controller and the at least one image capture device, establishing a relationship between the at least one image capture device and the at least one illumination device.

### Claim 35 (depends on 34)

The method of claim 34, wherein establishing a relationship between the at least one image capture device and the at least one illumination device comprises pairing the at least one image capture device with the at least one illumination device.

### Claim 36 (depends on 34)

The method of claim 34, wherein establishing a relationship between the at least one image capture device and the at least one illumination device comprises:
receiving user input requesting use of at least one illumination device associated with another user;
transmitting a first signal requesting permission from the other use to establish a relationship with the at least one illumination device;
responsive to receiving a second signal indicating permission has been granted to establish a relationship with the at least one illumination device, pairing the at least one image capture device with the at least one illumination device.

### Claim 37

A method for capturing multiple images using multiple devices, comprising:
at an image capture device, determining a pattern for capture of multiple images using multiple devices;
establishing communication, via the communication medium, between the image capture device and at least one illumination device;
transmitting at least one message, via the communication medium, from the image capture device to the at least one illumination device to cause the at least one illumination device to successively illuminate a subject at a plurality of illumination configurations;
capturing a plurality of images of the subject in synchrony with the successive illuminations;
storing the captured plurality of images at a storage device; and
outputting at least one of the stored images.

### Claim 38

A method of interactive display of a relighted image, comprising:
retrieving, from a storage device, at least two images representing a scene, the at least two images having different lighting characteristics;
receiving user input to specify a position for at least one virtual light source;
determining, based on the specified position for at least one virtual light source, a weighting associated with at least two of the retrieved images; and
generating a relighted image from a combination of at least two of the retrieved images, wherein the combination is weighted according to the determined weighting.

### Claim 39 (depends on 38)

The method of claim 38, wherein receiving user input to specify a position for at least one virtual light source comprises receiving user input interacting with a displayed graph representing weightings and positions for virtual light sources.

### Claim 40

A non-transitory computer-readable medium for capturing multiple images using multiple devices, comprising instructions stored thereon, that when executed on a processor, perform the steps of:
determining a pattern for capture of multiple images using multiple devices;
establishing communication, via a communication medium, between the controller and at least one image capture device;
establishing communication, via the communication medium, between the controller and at least one illumination device;
transmitting at least one message, via the communication medium, from the controller to the at least one image capture device and to the at least one illumination device to cause the at least one image capture device to capture a plurality of images of a subject, the images having a plurality of illumination configurations;
causing a storage device to store the captured plurality of images; and
generating output from at least one of the stored images.

### Claim 41 (depends on 40)

The non-transitory computer-readable medium of claim 40, wherein transmitting at least one message comprises:
transmitting at least one message to cause a movable illumination device to successively illuminate the subject from each of a plurality of different positions; and
transmitting at least one message to cause at least one image capture device to capture at least one image in synchrony with each illumination of the subject.

### Claim 42 (depends on 40)

The non-transitory computer-readable medium of claim 40, wherein transmitting at least one message comprises:
transmitting at least one message to cause a plurality of illumination devices at different positions with respect to the subject to successively illuminate the subject at each of the positions; and
transmitting at least one message to cause at least one image capture device to capture at least one image in synchrony with each illumination of the subject.

### Claim 43 (depends on 40)

The non-transitory computer-readable medium of claim 40, further comprising instructions stored thereon, that when executed on a processor, perform the step of:
causing an output device to generate output to instruct a user to move at least one of an image capture device and an illumination device to a specified position.

### Claim 44 (depends on 40)

The non-transitory computer-readable medium of claim 40, further comprising instructions stored thereon, that when executed on a processor, perform the steps of:
detecting a new position of an illumination device caused by user movement of the illumination device; and
responsive to the new position being within a predefined tolerance of a predefined position for an image capture event:
transmitting a message to cause the illumination device to illuminate the subject; and
transmitting a message to cause the at least one image capture device to capture an image of the subject in synchrony with the illumination of the subject.

### Claim 45 (depends on 40)

The non-transitory computer-readable medium of claim 40, further comprising instructions stored thereon, that when executed on a processor, perform the steps of:
detecting a new position of an image capture device caused by user movement of the image capture device; and
responsive to the new position being within a predefined tolerance of a predefined position for an image capture event:
transmitting a message to cause an illumination device to illuminate the subject; and
transmitting a message to cause the at least one image capture device to capture an image of the subject in synchrony with the illumination of the subject.

### Claim 46 (depends on 40)

The non-transitory computer-readable medium of claim 40, wherein transmitting at least one message comprises transmitting at least one message to cause the at least one image capture device to capture a plurality of light-field images.

### Claim 47 (depends on 40)

The non-transitory computer-readable medium of claim 40, wherein generating output from at least one of the stored images comprises:
generating an image by combining at least two of the stored images; and
causing an output device to output the generated image.

### Claim 48 (depends on 47)

The non-transitory computer-readable medium of claim 47, wherein combining at least two of the stored images comprises generating a relighted image from a weighted combination of at least two of the stored images.

### Claim 49 (depends on 40)

The non-transitory computer-readable medium of claim 40, wherein generating output from at least one of the stored images comprises:
receiving, from an input device, user input to specify a position for at least one virtual light source;
determining, based on the specified position for at least one virtual light source, a weighting associated with at least two of the stored images; and
generating a relighted image from a combination of at least two of the stored images, wherein the combination is weighted according to the determined weighting.

### Claim 50 (depends on 40)

The non-transitory computer-readable medium of claim 40, wherein generating output from at least one of the stored images comprises:
generating a light-field picture by combining at least two of the stored images; and
causing at least one output device to output an image generated from the light-field picture.

### Claim 51

A system for capturing multiple images using multiple devices, comprising:
a controller, configured to:
determine a pattern for capture of multiple images using multiple devices;
establish communication, via a communication medium, between the controller and at least one image capture device;
establish communication, via the communication medium, between the controller and at least one illumination device; and
transmit at least one message, via the communication medium, to the at least one image capture device and to the at least one illumination device to cause the at least one image capture device to capture a plurality of images of a subject, the images having a plurality of illumination configurations;
a storage device, communicatively coupled to the controller, configured to store the captured plurality of images; and
an output device, communicatively coupled to the storage device, configured to generate output from at least one of the stored images.

### Claim 52 (depends on 51)

The system of claim 51, wherein the communication medium comprises a wireless communication medium.

### Claim 53 (depends on 51)

The system of claim 51, wherein the controller is configured to establish communication between the controller and a plurality of image capture devices.

### Claim 54 (depends on 51)

The system of claim 51, wherein the controller is configured to establish communication between the controller and a plurality of illumination devices.

### Claim 55 (depends on 51)

The system of claim 51, wherein:
each image capture device comprises a camera; and
each illumination device comprises a flash unit.

### Claim 56 (depends on 51)

The system of claim 51, wherein the controller is configured to:
transmit at least one message to cause a movable illumination device to successively illuminate the subject from each of a plurality of different positions; and
transmit at least one message to cause at least one image capture device to capture at least one image in synchrony with each illumination of the subject.

### Claim 57 (depends on 51)

The system of claim 51, wherein the controller is configured to:
transmit at least one message to cause a plurality of illumination devices at different positions with respect to the subject to successively illuminate the subject at each of the positions; and
transmit at least one message to cause at least one image capture device to capture at least one image in synchrony with each illumination of the subject.

### Claim 58 (depends on 51)

The system of claim 51, wherein the controller is configured to cause at least one output device to generate output to instruct a user to move at least one of an image capture device and an illumination device to a specified position.

### Claim 59 (depends on 51)

The system of claim 51, wherein the controller is configured to:
detect a new position of an illumination device caused by user movement of the illumination device; and
responsive to the new position being within a predefined tolerance of a predefined position for an image capture event:
transmit a message to cause the illumination device to illuminate the subject; and
transmit a message to cause the at least one image capture device to capture an image of the subject in synchrony with the illumination of the subject.

### Claim 60 (depends on 51)

The system of claim 51, wherein the controller is configured to:
detect a new position of an image capture device caused by user movement of the image capture device; and
responsive to the new position being within a predefined tolerance of a predefined position for an image capture event:
transmit a message to cause an illumination device to illuminate the subject; and
transmit a message to cause the at least one image capture device to capture an image of the subject in synchrony with the illumination of the subject.

### Claim 61 (depends on 51)

The system of claim 51, wherein the controller is configured to transmit at least one message by transmitting at least one message to cause the at least one image capture device to capture a plurality of light-field images.

### Claim 62 (depends on 51)

The system of claim 51, further comprising an image combining module, communicatively coupled to the storage device, configured to combine at least two of the stored images to generate an image, and wherein the output device is configured to output the generated image.

### Claim 63 (depends on 62)

The system of claim 62, wherein the image combining module is configured to combine at least two of the stored images by generating a relighted image from a weighted combination of at least two of the stored images.

### Claim 64 (depends on 51)

The system of claim 51, further comprising:
at least one input device, communicatively coupled to the controller, configured to be receptive to user input to specify a position for at least one virtual light source;
wherein:
the image combining module is configured to determine, based on the specified position for at least one virtual light source, a weighting associated with at least two of the stored images, and to generate a relighted image from a combination of at least two of the stored images, wherein the combination is weighted according to the determined weighting;
and wherein the output device is configured to output the generated relighted image.

### Claim 65 (depends on 51)

The system of claim 51, further comprising an image combining module, communicatively coupled to the storage device, configured to generate a light-field picture by combining at least two of the stored images;
and wherein the output device is configured to output an image generated from the light-field picture.
