# US 2018/0293805 A1 · Methods and apparatuses for determining positions of multi-directional image capture apparatuses

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0293805 A1
- Document ID: US20180293805A1
- Status: Published application
- Kind: A1
- Application number: 15/948,012
- Filed: 2018-04-09
- Published: 2018-10-11
- Earliest claimed: 2017-04-11
- Assignees: Nokia Technologies Oy
- Applicants: Nokia Technologies Oy
- Inventors: Tinghuai WANG; Yu YOU; Lixin FAN
- Web page: https://patentgrep.com/patent/US20180293805A1

## CPC

- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G01B 11/245: using a plurality of fixed, simultaneously operating transducers ({G01B11/2408 - G01B11/2425, } G01B11/255 take precedence)
- G01B 21/22: for measuring angles or tapers; for testing the alignment of axes
- G03B 37/04: with cameras or projectors providing touching or overlapping fields of view
- G06T 7/579: from motion
- G06T 7/73: using feature-based methods
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 23/58: Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06T 2207/10004: Still image; Photographic image
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/30244: Camera pose
- H04N 5/2224: {related to virtual studio applications}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G03B: APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR (optical parts of such apparatus G02B; photosensitive materials or processes for photographic purposes G03C; apparatus for processing exposed photographic materials G03D)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method comprising performing image re-projection on each of a plurality of first images of a scene, thereby to generate a plurality of re-projected second images of the scene, wherein each first image of the scene is captured by a respective camera of a first multi-directional image capture apparatus and each second image of the scene is associated with a respective virtual camera; processing the plurality of second images based on a previously generated virtual three dimensional model of the scene, thereby to generate respective positions of the virtual cameras associated with the second images; and determining a position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras.

## Claims

### Claim 1

A method comprising:
performing image re-projection on a plurality of first images of a scene to generate a plurality of re-projected second images of the scene, wherein each first image of the scene is captured by a respective camera of a first multi-directional image capture apparatus and each second image of the scene is associated with a respective virtual camera;
processing the plurality of re-projected second images based on a previously generated virtual three dimensional model of the scene to generate respective positions of the virtual cameras associated with the second images; and
determining a position of the first multi-directional image capture apparatus based on one or more of the generated respective positions of the virtual cameras.

### Claim 2 (depends on 1)

The method of claim 1, further comprising updating the previously generated virtual three dimensional model with the determined position of the first multi-directional image capture apparatus.

### Claim 3 (depends on 1)

The method of claim 1, wherein the previously generated virtual three dimensional model comprises information describing features of the scene.

### Claim 4 (depends on 1)

The method of claim 1, wherein the previously generated three dimensional model is generated based at least in part on images previously captured by the first multi-directional image capture apparatus; or images previously captured by a set of one or more multi-directional image capture apparatuses which does not include the first multi-directional image capture apparatus.

### Claim 5 (depends on 1)

The method of claim 1, wherein processing the plurality of re-projected second images based on the previously generated virtual three dimensional model comprises:
determining point correspondences between the re-projected second images and the previously generated virtual three dimensional model; and
performing a Direct Linear Transform based on the determined point correspondences to determine positions of the virtual cameras.

### Claim 6 (depends on 1)

The method of claim 1, wherein determining the position of the first multi-directional image capture apparatus based on the generated respective positions of the virtual cameras comprises:
determining a position of a camera of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras; and
determining the position of the first multi-directional image capture apparatus based on the determined position of the camera.

### Claim 7 (depends on 1)

The method of claim 1, wherein determining the position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras comprises:
determining inliers in the generated respective positions of the virtual cameras; and
determining the position of the first multi-directional image capture apparatus based on the inliers.

### Claim 8 (depends on 1)

The method of claim 1, wherein determining the position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras further comprises:
determining a virtual camera based on re-projection error associated therewith; and
determining the position of the first multi-directional image capture apparatus based only on the position of the virtual camera determined based on the re-projection error associated therewith.

### Claim 9 (depends on 1)

The method of claim 1, wherein the processing of the plurality of re-projected second images further generates respective orientations of the virtual cameras, and the method further comprises:
determining an orientation of the first multi-directional image capture apparatus based on one or more of the generated respective orientations of the virtual cameras.

### Claim 10

Apparatus comprising:
at least one processor; and
at least one memory including computer program code, which when executed by the at least one processor, causes the apparatus to:
perform image re-projection on a plurality of first images of a scene to generate a plurality of re-projected second images of the scene, wherein each first image of the scene is captured by a respective camera of a first multi-directional image capture apparatus and each second image of the scene is associated with a respective virtual camera;
process the plurality of re-projected second images based on a previously generated virtual three dimensional model of the scene to generate respective positions of the virtual cameras associated with the second images; and
determine a position of the first multi-directional image capture apparatus based on one or more of the generated respective positions of the virtual cameras.

### Claim 11 (depends on 10)

The apparatus of claim 10, wherein the computer program code, which when executed by the at least one processor, further causes the apparatus to:
update the previously generated virtual three dimensional model with the determined position of the first multi-directional image capture apparatus.

### Claim 12 (depends on 10)

The apparatus of claim 10, wherein the previously generated virtual three dimensional model comprises information describing features of the scene.

### Claim 13 (depends on 10)

The apparatus of claim 10, wherein the previously generated three dimensional model is generated based at least in part on images previously captured by the first multi-directional image capture apparatus; or
images previously captured by a set of one or more multi-directional image capture apparatuses which does not include the first multi-directional image capture apparatus.

### Claim 14 (depends on 10)

The apparatus of claim 10, wherein to process the plurality of second images based on the virtual three dimensional model, the apparatus is further caused to:
determine point correspondences between the re-projected second images and the previously generated virtual three dimensional model; and
perform a Direct Linear Transform based on the determined point correspondences to determine positions of the virtual cameras.

### Claim 15 (depends on 10)

The apparatus of claim 10, wherein to determine the position of the first multi-directional image capture apparatus based on the generated positions of the virtual cameras, the apparatus is further caused to:
determine a position of a camera of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras; and
determine the position of the first multi-directional image capture apparatus based on the determined position of the camera.

### Claim 16 (depends on 10)

The apparatus of claim 10, wherein to determine the position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras, the apparatus is further caused to:
determine inliers in the generated respective positions of the virtual cameras; and
determine the position of the first multi-directional image capture apparatus based on the inliers.

### Claim 17 (depends on 10)

The apparatus of claim 10, wherein to determine the position of the first multi-directional image capture apparatus based on one or more of the generated positions of the virtual cameras, the apparatus is further caused to:
determine a virtual camera based on re-projection error associated therewith; and
determine the position of the multi-directional image capture apparatus based only on the position of the virtual camera determined based on the re-projection error associated therewith.

### Claim 18 (depends on 10)

The apparatus of claim 10, wherein the processing of the plurality of second images further generates respective orientations of the virtual cameras, and the computer program code, which when executed by the at least one processor, further causes the apparatus to:
determine an orientation of the first multi-directional image capture apparatus based on one or more of the generated respective orientations of the virtual cameras.

### Claim 19 (depends on 18)

The apparatus of claim 18, wherein the position of the multi-directional image capture apparatus is determined based on the one or more generated positions of the virtual cameras and the one or more generated orientations of the virtual cameras.

### Claim 20

A computer-readable medium having computer-readable code stored thereon, the computer readable code, when executed by at least one processor, causes an apparatus to:
performing image re-projection on a plurality of first images of a scene to generate a plurality of re-projected second images of the scene, wherein each first image of the scene is captured by a respective camera of a first multi-directional image capture apparatus and each second image of the scene is associated with a respective virtual camera;
processing the plurality of re-projected second images based on a previously generated virtual three dimensional model of the scene to generate respective positions of the virtual cameras associated with the second images; and
determining a position of the first multi-directional image capture apparatus based on one or more of the generated respective positions of the virtual cameras.
