# US 2024/0257309 A1 · Aperture fusion with separate devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0257309 A1
- Document ID: US20240257309A1
- Status: Published application
- Kind: A1
- Application number: 18/109,609
- Filed: 2023-02-14
- Published: 2024-08-01
- Earliest claimed: 2023-01-31
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Wesley James HOLLAND
- Web page: https://patentgrep.com/patent/US20240257309A1

## CPC

- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 7/30: Determination of transform parameters for the alignment of images, i.e. image registration
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- H04N 5/2628: {Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation}
- H04N 5/265: Mixing
- H04N 7/185: {from a mobile camera, e.g. for remote control}
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30244: Camera pose
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Systems and techniques for combining images captured by two or more image sensors are disclosed. For example, a method can include obtaining a first image of a scene from a first image sensor of a first device. The method can include obtaining a second image including at least a portion of the scene from a second image sensor of a second device. The second image is transmitted over a communications link. The method can include determining a localization between the first device and the second device based on a relative pose between the first device and the second device. The method can include normalizing one or more image properties between the first image and the second image. The method can include generating, based on the localization and normalizing the one or more image properties, a third image based on the first image and the second image.

## Claims

### Claim 1

An apparatus for generating images based on two or more images, comprising:
a first device comprising:
a first image sensor;
a memory; and
one or more processors coupled to the memory and configured to:
obtain a first image of a scene from the first image sensor;
obtain a second image of the scene from a second image sensor of a second device, wherein the second image is transmitted over a communications link between the first device and the second device;
determine a localization between the first device and the second device based on a relative pose between the first device and the second device;
normalize one or more image properties between the first image and the second image; and
generate, based on the localization between the first device and the second device and normalizing the one or more image properties between the first image and the second image, a third image based on the first image and the second image.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein:
the first device comprises a first housing;
the second device comprises second housing, different from the first housing; and
the first device and the second device are configured such that the relative pose between the first device and the second device can be changed.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein the first device comprises a mobile device and the second device comprises a head mounted display.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the second image depicts at least a portion of the first device; and wherein, to generate the third image, the one or more processors are configured to remove the at least a portion of the first device from the second image.

### Claim 5 (depends on 4)

The apparatus of claim 4, wherein, to determine the localization between the first device and the second device, the one or more processors are configured to detect one or more features associated with the first device in the second image, wherein the at least a portion of the first device is not depicted in the first image.

### Claim 6 (depends on 1)

The apparatus of claim 1, the one or more processors configured to:
obtain a fourth image of a second scene from the first image sensor;
obtain a fifth image of the second scene from the second image sensor, wherein the fifth image is transmitted over the communications link between the first device and the second device;
determine an additional localization between the first device and the second device, wherein the additional localization is based on an additional relative pose between the first device and the second device, wherein the additional relative pose is different from the relative pose;
normalize one or more image properties between the fourth image and the fifth image; and
generate, based on the additional localization between the first device and the second device and normalizing the one or more image properties between the fourth image and the fifth image, a sixth image based on the fourth image and the fifth image.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein the communications link is a wireless communications link.

### Claim 8 (depends on 1)

The apparatus of claim 1, wherein, to normalize the one or more image properties between the first image and the second image, the one or more processors are configured to adjust first one or more image properties of the first image or second one or more image properties of the second image to produce the third image having either the first one or more image properties or the second one or more image properties.

### Claim 9 (depends on 1)

The apparatus of claim 1, wherein the one or more image properties comprises at least one or more of resolution, brightness, white balance, color balance, focus, depth of field, field of view, or distortion;
wherein, to generate the third image, the one or more processors are configured to remove an object from the second image, wherein the object is absent from the first image.

### Claim 10 (depends on 1)

The apparatus of claim 1, the one or more processors configured to:
obtain an input associated with a change in zoom;
obtain a fourth image of the scene from the first image sensor;
obtain a fifth image including at least a portion of the scene from the second image sensor;
determine an additional localization between the first device and the second device based on an additional relative pose between the first device and the second device;
normalize one or more image properties between the fourth image and the fifth image; and
generate, based on the localization between the first device and the second device, the input associated with the change in zoom, and normalizing the one or more image properties between the fourth image and the fifth image, a sixth image based on the fourth image and the fifth image.

### Claim 11 (depends on 10)

The apparatus of claim 10, wherein a first field of view associated with the third image is different from a second field of view associated with the sixth image.

### Claim 12 (depends on 1)

The apparatus of claim 1, wherein the localization comprises at least one or more of SLAM maps, sensor measurements, inertial sensor measurements, images, or feature vectors.

### Claim 13 (depends on 1)

The apparatus of claim 1, wherein the one or more processors are configured to:
determine a misalignment between the first image of the scene and the second image of the scene; and
based on the misalignment, generate an indicator for indicating an adjustment between the first image sensor and the second image sensor to correct the misalignment.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the first image comprises a first field of view (FOV) and the second image comprises a second FOV, different from the first FOV.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein, to determine the misalignment between the first image of the scene and the second image of the scene, the one or more processors are configured to determine an amount of overlap between the first FOV and the second FOV.

### Claim 16

A method for generating a combined image based on two or more images, comprising:
obtaining a first image of a scene from a first image sensor of a first device;
obtaining a second image including at least a portion of the scene from a second image sensor of a second device, wherein the second image is transmitted over a communications link between the first device and the second device;
determining a localization between the first device and the second device based on a relative pose between the first device and the second device;
normalizing one or more image properties between the first image and the second image; and
generating, based on the localization between the first device and the second device and normalizing the one or more image properties between the first image and the second image, a third image based on the first image and the second image.

### Claim 17 (depends on 16)

The method of claim 16, wherein the first device comprises a first housing;
the second device comprises second housing, different from the first housing; and
the first device and the second device are configured such that the relative pose between the first device and the second device can be changed.

### Claim 18 (depends on 16)

The method of claim 16, wherein the first device comprises a mobile device and the second device comprises a head mounted display.

### Claim 19 (depends on 16)

The method of claim 16, wherein the second image depicts at least a portion of and wherein generating the third image comprises removing the at least a portion of the first device from the second image.

### Claim 20 (depends on 19)

The method of claim 19, further comprising detecting one or more features associated with the first device in the second image, wherein the first device is absent from the first image.

### Claim 21 (depends on 16)

The method of claim 16, wherein:
the second image depicts least one or more of a human body part, an electronic device, or a mechanical structure; and
generating the third image comprises removing the at least one or more of the human body part, the electronic device, or the mechanical structure from the second image.

### Claim 22 (depends on 16)

The method of claim 16, further comprising:
obtaining a fourth image of a second scene from the first image sensor;
obtaining a fifth image of the second scene from the second image sensor, wherein the fourth image is transmitted over the communications link between the first device and the second device;
determining an additional localization between the first device and the second device, wherein the additional localization is based on an additional relative pose between the first device and the second device, wherein the additional relative pose is different from the relative pose;
normalizing one or more image properties between the fourth image and the fifth image; and
generating, based on the additional localization between the first device and the second device and normalizing the one or more image properties between the fourth image and the fifth image, a sixth image based on the fourth image and the fifth image.

### Claim 23 (depends on 22)

The method of claim 22, wherein a FOV associated with the third image is different from a FOV associated with the sixth image.

### Claim 24 (depends on 16)

The method of claim 16, wherein the communications link is a wireless communications link.

### Claim 25 (depends on 16)

The method of claim 16, wherein normalizing the one or more image properties between the first image and the second image comprises adjusting first one or more image properties of the first image or second one or more image properties of the second image to produce the third image having either the first one or more image properties or the second one or more image properties.

### Claim 26 (depends on 16)

The method of claim 16, wherein the one or more image properties comprises at least one or more of resolution, brightness, white balance, color balance, focus, depth of field, FOV, or distortion.

### Claim 27 (depends on 16)

The method of claim 16, wherein:
the first image comprises a first perspective of the scene and the second image comprises a second perspective of the scene, different from the first perspective of the scene; and
generating the combined image comprises adjusting pixels of the second image to the first perspective or adjusting pixels of the first image to the second perspective.

### Claim 28 (depends on 16)

The method of claim 16, wherein the second device is a head-mounted display and the first device is a mobile device.

### Claim 29 (depends on 16)

The method of claim 16, wherein the localization comprises at least one or more of SLAM maps, sensor measurements, inertial sensor measurements, images, or feature vectors.

### Claim 30 (depends on 16)

The method of claim 16, further comprising:
determining a misalignment between the first image of the scene and the second image of the scene; and
based on the misalignment, generating an indicator for indicating an adjustment between the first image sensor and the second image sensor to correct the misalignment.
