# US 10,482,571 B2 · Dual fisheye, hemispherical image projection and stitching method, device and computer-readable medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,482,571 B2
- Document ID: US10482571B2
- Status: Granted
- Kind: B2
- Application number: 15/697,606
- Filed: 2017-09-07
- Published: 2018-04-26
- Granted: 2019-11-19
- Earliest claimed: 2016-10-20
- Assignees: Ricoh Company, Ltd.
- Applicants: Hong Yi; Weitao Gong; Haihua Yu; Wei Wang
- Inventors: Hong Yi; Weitao Gong; Haihua Yu; Wei Wang
- Web page: https://patentgrep.com/patent/US10482571B2

## CPC

- G06T 3/047: Fisheye or wide-angle transformations
- G06T 3/12: Panospheric to cylindrical image transformations
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An image processing method include projecting first and second images which are captured by two fisheye lenses onto first and second hemispherical projection models, respectively; determining a set of waiting-for-stitching surface region pairs on the first and second hemispherical projection models; extracting feature points from the two fisheye images and determining feature points matching each other to serving as pixels matching each other; determining a set of surface region pairs including the pixels matching each other on the first and second hemispherical projection models as the set of waiting-for-stitching surface region pairs; generating a customized projection model on the grounds of the set of waiting-for-stitching surface region pairs by letting a spherical distance between vertexes of each pair in the set of waiting-for-stitching surface region pairs be zero; and generating a third image by projecting the first and second images onto the customized projection model.

## Claims

### Claim 1

An image processing method comprising:
projecting a first image and a second image onto a first hemispherical projection model and a second hemispherical projection model, respectively;
determining a set of waiting-for-stitching surface region pairs on the first hemispherical projection model and the second hemispherical projection model;
preliminarily generating a customized spherical projection model by stitching the first hemispherical projection model and the second hemispherical projection model along the set of waiting-for-stitching surface region pairs; and
projecting the first image and the second image that are acquired as a real time video stream directly onto the preliminarily customized spherical projection model so as to create a third image without conducting image stitching,
wherein, the first image and the second image are fisheye images captured in opposite imaging directions from a same position.

### Claim 2 (depends on 1)

The image processing method according to claim 1, wherein, the projecting a first image and a second image onto a first hemispherical projection model and a second hemispherical projection model, respectively includes
generating a first texture map and a second texture map based on the first image and the second image, respectively; and
projecting the first texture map and the second texture map onto the first hemispherical projection model and the second hemispherical projection model so as to obtain a first projective texture map and a second projective texture map, respectively.

### Claim 3 (depends on 2)

The image processing method according to claim 2, wherein, the determining a set of waiting-for-stitching surface region pairs on the first hemispherical projection model and the second hemispherical projection model includes
extracting, in a first direction, first feature points and second feature points from the first projective texture map and the second projective texture map, respectively;
selecting, in a second direction orthogonal to the first direction, third feature points among the second feature points extracted from the second projective texture map, which respectively match the first feature points extracted from the first projective texture map, so as to acquire pairs of the first feature points and the third feature points;
determining pairs of surface regions, where the pairs of the first feature points and the third feature points are located, on the first hemispherical projection model and the second hemispherical projection model as the set of waiting-for-stitching surface region pairs.

### Claim 4 (depends on 3)

The image processing method according to claim 3, wherein, the generating a customized projection model based on the set of waiting-for-stitching surface region pairs includes
letting a spherical distance between vertexes of each pair in the set of waiting-for-stitching surface region pairs be zero so as to create the customized projection model; and
setting marks for respective surface regions in the customized projection model.

### Claim 5 (depends on 4)

The image processing method according to claim 4, wherein, the projecting the first image and the second image onto the customized projection model so as to create a third image includes
projecting the first texture map and the second texture map onto the customized projection model; and
processing, based on the marks, respective surface regions in the first texture map and the second texture map projected, so as to create the third image.

### Claim 6 (depends on 1)

A non-transitory computer-readable medium having computer-executable instructions for execution by a processing system, wherein, the computer-executable instructions, when executed, cause the processing system to carry out the image processing method according to claim 1.

### Claim 7

An image processing device comprising:
a projector configured to project a first image and a second image onto a first hemispherical projection model and a second hemispherical projection model, respectively; and
a processor configured to:
determine a set of waiting-for-stitching surface region pairs on the first hemispherical projection model and the second hemispherical projection model; and
preliminarily generate a customized spherical projection model by stitching the first hemispherical projection model and the second hemispherical projection model along the set of waiting-for-stitching surface region pairs,
wherein, the projector is further configured to project the first image and the second image that are acquired as a real time video stream directly onto the preliminarily customized spherical projection model so as to create a third image without conducting image stitching, and the first image and the second image are fisheye images captured in opposite imaging directions from a same position.

### Claim 8 (depends on 7)

The image processing device according to claim 7, wherein, the projector is configured to
generate a first texture map and a second texture map based on the first image and the second image, respectively; and
project the first texture map and the second texture map onto the first hemispherical projection model and the second hemispherical projection model so as to obtain a first projective texture map and a second projective texture map, respectively.

### Claim 9 (depends on 8)

The image processing device according to claim 8, wherein, the processor is configured to
extract, in a first direction, first feature points and second feature points from the first projective texture map and the second projective texture map, respectively;
select, in a second direction orthogonal to the first direction, third feature points among the second feature points extracted from the second projective texture map, which respectively match the first feature points extracted from the first projective texture map, so as to acquire pairs of the first feature points and the third feature points;
determine pairs of surface regions, where the pairs of the first feature points and the third feature points are located, on the first hemispherical projection model and the second hemispherical projection model as the set of waiting-for-stitching surface region pairs.

### Claim 10 (depends on 9)

The image processing device according to claim 9, wherein, the processor is configured to
let a spherical distance between vertexes of each pair in the set of waiting-for-stitching surface region pairs be zero so as to create the customized projection model; and
set marks for respective surface regions in the customized projection model.

### Claim 11 (depends on 10)

The image processing device according to claim 10, wherein, the projector is configured to
project the first texture map and the second texture map onto the customized projection model; and
processing, based on the marks, respective surface regions in the first texture map and the second texture map projected, so as to create the third image.