# US 2017/0257566 A1 · Apparatus and method to record a 360 degree image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0257566 A1
- Document ID: US20170257566A1
- Status: Published application
- Kind: A1
- Application number: 15/604,544
- Filed: 2017-05-24
- Published: 2017-09-07
- Earliest claimed: 2015-10-30
- Applicants: Essential Products, Inc.
- Inventors: David John Evans, V; Xinrui Jiang; Andrew E. Rubin; Matthew Hershenson; Xiaoyu Miao
- Web page: https://patentgrep.com/patent/US20170257566A1

## CPC

- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- G06T 3/047: Fisheye or wide-angle transformations
- G06T 3/12: Panospheric to cylindrical image transformations
- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/80: Geometric correction
- H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
- H04N 23/51: Housings
- H04N 23/55: Optical parts specially adapted for electronic image sensors; Mounting thereof
- H04N 23/62: Control of parameters via user interfaces
- H04N 23/631: {Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters}
- H04N 23/69: Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06T 2207/20221: Image fusion; Image merging
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Certain aspects of the technology disclosed herein combine wide angle lenses with normal lenses to create an undistorted 360° view of the surrounding environment. The normal lenses record an image of the surrounding environment in the areas where the wide angle lenses provide a distorted view. The wide angle lenses and the normal lenses can be part of a stand-alone imaging device, can be accessories to a mobile device, or can be integrated into a mobile device. Various ways to integrate the wide lenses and/or the normal lenses into the camera are disclosed herein.

## Claims

### Claim 1

A method comprising:
determining a distortion associated with a wide angle lens by recording with the wide angle lens a distorted image of a first scene, and recording with a normal lens an image of the first scene, wherein the wide angle lens and the normal lens are oriented in substantially the same direction;
recording a distorted image of a second scene with the wide angle lens, and an image of the second scene with the normal lens, by orienting the wide angle lens and the normal lens in different directions; and
creating a wide field of view image by rectifying the distorted image of the second scene recorded by the wide angle lens based on the distortion associated with the wide angle lens, and combining the rectified image and the image of the second scene into the wide field of view image.

### Claim 2 (depends on 1)

The method of claim 1, wherein said determining the distortion comprises determining a transformation between the distorted image of the first scene and the image of the first scene, wherein applying the transformation to the distorted image of the first scene produces substantially the image of the first scene.

### Claim 3 (depends on 1)

The method of claim 1, said recording the wide field of view image comprising orienting the wide angle lens in a first direction, and orienting the normal lens in a second direction substantially 90° away from the first direction.

### Claim 4 (depends on 1)

The method of claim 1, said combining the rectified image and the image of the second scene into the wide field of view image comprising determining a correspondence region between the rectified image and the image of the second scene.

### Claim 5 (depends on 4)

The method of claim 4, said determining the correspondence region comprising:
creating an overlap region between the rectified image and the image of the second scene by establishing a correspondence between a first pixel in the rectified image and a first pixel in the image of the second scene;
calculating a difference between the first pixel in the rectified image and the first pixel in the image of the second scene;
reducing the overlap region until the overlap region is empty; and
determining the correspondence region to be the overlap region associated with the smallest difference between the first pixel in the rectified image and the first pixel in the image of the second scene.

### Claim 6 (depends on 5)

The method of claim 5, said calculating the difference comprising calculating a least square difference between the first pixel in the rectified image and the first pixel in the image of the second scene.

### Claim 7 (depends on 4)

The method of claim 4, comprising blending pixels in the correspondence region between the rectified image and the image of the second scene.

### Claim 8 (depends on 7)

The method of claim 7, said blending pixels comprising:
modulating a pixel in the correspondence region associated with the rectified image by a first weight;
modulating the pixel in the correspondence region associated with the image of the second scene by a second weight; and
adding the modulated first pixel in the correspondence region and the modulated first pixel in the image of the second scene to produce a pixel in the wide field of view image.

### Claim 9 (depends on 8)

The method of claim 8, wherein the first weight and the second weight add up to a constant number.

### Claim 10

A method comprising:
obtaining a plurality of mapping functions corresponding to a plurality of wide angle lenses, wherein a mapping function in the plurality of mapping functions comprises a transformation, wherein when the transformation is applied to a distorted image associated with a wide angle lens in the plurality of wide angle lenses the transformation produces an undistorted image;
receiving a first plurality of images corresponding to a first plurality of cameras, and a second plurality of images corresponding to a second plurality of cameras, wherein at least one image in the first plurality of images and the second plurality of images comprises the distorted image;
determining a pixel correspondence between the first image in the first plurality of images and the second image in the second plurality of images, based on an overlap area associated with the first image and the second image; and
combining the first plurality of undistorted images and the second plurality of undistorted images to obtain an undistorted 360° image.

### Claim 11 (depends on 10)

The method of claim 10, wherein said obtaining the plurality of mapping functions comprises:
recording a first image using the wide angle lens;
recording a second image using a normal lens, wherein the normal lens is disposed the same way as the wide angle lens;
calculating the mapping function; and
storing the mapping function in a memory.

### Claim 12 (depends on 10)

The method of claim 10, wherein said obtaining the plurality of mapping functions comprises retrieving the plurality of mapping functions from a memory.

### Claim 13 (depends on 10)

The method of claim 10, wherein the said determining the pixel correspondence comprises:
applying the mapping function to each distorted image, to obtain a first plurality of undistorted images, and second plurality of undistorted images, the first plurality of undistorted images corresponding to the first plurality of cameras, and the second plurality of undistorted images corresponding to the second plurality of cameras;
iteratively overlapping each image in the first plurality of undistorted images and a corresponding image in the second plurality of undistorted images to obtain a plurality of overlap areas, wherein the overlap area in the plurality of overlap areas is associated with each iteration;
iteratively calculating a plurality of match scores corresponding to a plurality of overlap areas between each image in the first plurality of undistorted images and the corresponding image in the second plurality of undistorted images; and
determining the pixel correspondence based on a best overlap area in the plurality of overlap areas, wherein the best overlap area is associated with the highest match score in the plurality of match scores.

### Claim 14 (depends on 10)

The method of claim 10, wherein said combining the first plurality of undistorted images and the second plurality of undistorted images comprises:
obtaining the overlap area associated with the first image in the first plurality of images, and the second image and the second plurality of images;
determining a first portion of the first image in the first plurality of image, the first portion outside the overlap area;
determining a second portion of the second image in the second plurality of images, the second portion outside the overlap area;
determining an overlap portion, said determining comprising:
for each pixel in the first image associated with the overlap area, weighing the pixel by a first weight, wherein the first weight comprises a number between 0 and 1 inclusive, and wherein the first weight decreases with an increasing distance from an edge associated with the overlap area;
for each corresponding pixel in the second image associated with the overlap area, weighing the pixel by a second weight, wherein the second weight increases as the first weight decreases;
adding the two pixels together to obtain a pixel in the overlap portion; and
tiling the first portion, the second portion, and the overlap portion to obtain a 360° image.

### Claim 15

A system comprising:
a wide angle lens;
a normal lens;
a processor;
a storage medium storing computer-executable instructions that, when executed by the processor, cause the system to perform a computer-implemented operation, the instructions comprising:
instructions for determining a distortion associated with the wide angle lens by recording with the wide angle lens a distorted image of a first scene, and recording with the normal lens an image of the first scene, wherein the wide angle lens and the normal lens are oriented in substantially the same direction;
instructions for recording a distorted image of a second scene with the wide angle lens and an image of the second scene with the normal lens, wherein the wide angle lens and the normal lens are oriented in different directions;
instructions for rectifying the distorted image of the second scene recorded by the wide angle lens based on the distortion associated with the wide angle lens; and
instructions for combining the rectified image and the image of the second scene into a wide field of view image.

### Claim 16 (depends on 15)

The system of claim 15, comprising a movable mechanism to change a direction associated with at least one of the wide angle lens and the normal lens.

### Claim 17 (depends on 15)

The system of claim 15, the normal lens associated with a camera of a mobile device.

### Claim 18 (depends on 15)

The system of claim 15, the wide angle lens associated with a mobile device accessory.

### Claim 19 (depends on 15)

The system of claim 15, the normal lens and the wide angle lens to record images synchronously.

### Claim 20 (depends on 15)

The system of claim 15, the wide angle lens oriented in a direction substantially 90° away from a direction of the normal lens.