# US 2023/0056036 A1 · Method and device for generating a panoramic image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0056036 A1
- Document ID: US20230056036A1
- Status: Published application
- Kind: A1
- Application number: 17/981,056
- Filed: 2022-11-04
- Published: 2023-02-23
- Earliest claimed: 2020-03-16
- Assignees: Realsee (Beijing) Technology Co., Ltd.
- Applicants: Realsee (Beijing) Technology Co., Ltd.
- Inventors: Tong RAO; Yonglin YANG; Mingyuan WANG; Yutong CHEN
- Web page: https://patentgrep.com/patent/US20230056036A1

## CPC

- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- G06T 15/10: Geometric effects
- G06T 15/503: {Blending, e.g. for anti-aliasing}
- H04N 5/265: Mixing
- H04N 23/84: for processing colour signals
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method for generating a panoramic image is disclosed. The method includes receiving a three-dimensional model of a target space and an initial panoramic image of the target space, the initial panoramic image having a latitude span less than a preset latitude span; determining a first set of coordinate parameters of a camera associated with the initial panoramic image and with respect to a reference frame associated with the three-dimensional model; mapping, based on the first set of coordinate parameters of the camera, data points of the three-dimensional model to a camera coordinate system associated with the camera to obtain an intermediate panoramic image; and obtaining a final panoramic image by merging the initial panoramic image and the intermediate panoramic image, the final panoramic image having a latitude span greater than or equal to the preset latitude span.

## Claims

### Claim 1

A method for generating a panoramic image, the method comprising:
receiving an image sequence of a target space, the image sequence including a plurality of effective images;
determining a connection relationship between the plurality of effective images of the image sequence;
determining a set of coordinate parameters of an image capturing device associated with the image sequence; and
obtaining the panoramic image based on the connection relationship, the set of coordinate parameters of the image capturing device, and the image sequence.

### Claim 2 (depends on 1)

The method according to claim 1, wherein obtaining the panoramic image based on the connection relationship, the set of coordinate parameters of the image capturing device, and the image sequence comprises:
mapping, based on the connection relationship and the set of coordinate parameters of the image capturing device, the plurality of effective images of the image sequence to a mapping surface defined with respect to the set of coordinate parameters of the image capturing device.

### Claim 3 (depends on 2)

The method according to claim 2, wherein determining the connection relationship between the plurality of effective images of the image sequence comprises:
in response to the image sequence being taken in a laterally discrete manner, determining that each image of the image sequence is an effective image, and determining the connection relationship between images of the image sequence by performing feature extraction and feature matching.

### Claim 4 (depends on 3)

The method according to claim 3, wherein mapping, based on the connection relationship and the set of coordinate parameters of the image capturing device, the plurality of effective images of the image sequence to the mapping surface defined with respect to the set of coordinate parameters of the image capturing device comprises:
determining, based on the connection relationship and the set of coordinate parameters of the image capturing device, a Euler angle of the image capturing device corresponding to the images of the image sequence;
obtaining a sub-panoramic image corresponding to the image sequence by mapping, based on the Euler angle of the image capturing device corresponding to the images of the image sequence, each of the image of the image sequence to the mapping surface;
determining characteristics of each sub-panoramic image; and
obtaining the panoramic image by merging, based on the characteristics of each of the sub-panoramic image, each of the sub-panoramic image.

### Claim 5 (depends on 2)

The method according to claim 2, wherein determining the connection relationship between the plurality of effective images of the image sequence comprises:
in response to the image sequence being taken in a longitudinally discrete manner, determining a mapping relationship between each target image and corresponding other images of the image sequence;
obtaining a fused image corresponding to the image sequence by fusing, based on the mapping relationship, the corresponding other images into each of the target image of the image sequence; and
determining the connection relationship between each fused image corresponding to the image sequence by performing feature extraction and feature matching.

### Claim 6 (depends on 5)

The method according to claim 5, wherein mapping, based on the connection relationship and the set of coordinate parameters of the image capturing device, the plurality of effective images of the image sequence to the mapping surface defined with respect to the set of coordinate parameters of the image capturing device comprises:
obtaining a sub-panoramic image corresponding to each fused image by mapping each of the fused image to the mapping surface;
obtaining characteristics of each sub-panoramic image; and
obtaining the panoramic image by merging, based on the characteristics of each of the sub-panoramic image, each of the sub-panoramic image.

### Claim 7 (depends on 2)

The method according to claim 2, wherein determining the connection relationship between the plurality of effective images of the image sequence comprises:
in response to the image sequence being taken in a laterally discrete manner and other image sequences being taken in a laterally continuous manner, determining that each image of the image sequence is an effective image, and determining a connection relationship between each of the image of the image sequence by performing feature extraction and feature matching; and
determining that a key frame image of each of the other image sequences is an effective image, and determining a connection relationship between each key frame image of each of the other image sequences and each of the other image sequences by performing feature extraction and feature matching.

### Claim 8 (depends on 7)

The method according to claim 7, wherein mapping, based on the connection relationship and the set of coordinate parameters of the image capturing device, the plurality of effective images of the image sequence to the mapping surface defined with respect to the set of coordinate parameters of the image capturing device comprises:
obtaining a sub-panoramic image corresponding to the image sequence by mapping each image of the image sequence to the mapping surface;
obtaining a mapping image by mapping each of the key frame image of each of the other image sequences to the mapping surface;
determining characteristics of each mapping image and each sub-panoramic image; and
obtaining the panoramic image by mapping, based on the characteristics of each of the mapping image and each of the sub-panoramic image, each of the mapping image and each of the sub-panoramic image.

### Claim 9

A device for generating a panoramic image, comprising:
one or more processors; and
a computer-readable medium including computer-executable instructions stored thereon, which, when executed by the one or more processors, cause the device to carry out:
receiving an image sequence of a target space, the image sequence including a plurality of effective images;
determining a connection relationship between the plurality of effective images of the image sequence;
determining a set of coordinate parameters of an image capturing device associated with the image sequence; and
obtaining the panoramic image based on the connection relationship, the set of coordinate parameters of the image capturing device, and the image sequence.

### Claim 10 (depends on 9)

The device according to claim 9, wherein obtaining the panoramic image based on the connection relationship, the set of coordinate parameters of the image capturing device, and the image sequence comprises the computer-executable instructions cause the device to carry out:
mapping, based on the connection relationship and the set of coordinate parameters of the image capturing device, the plurality of effective images of the image sequence to a mapping surface defined with respect to the set of coordinate parameters of the image capturing device.

### Claim 11 (depends on 10)

The device according to claim 10, wherein determining the connection relationship between the plurality of effective images of the image sequence comprises the computer-executable instructions cause the device to carry out:
in response to the image sequence being taken in a laterally discrete manner, determining that each image of the image sequence is an effective image, and determining the connection relationship between images of the image sequence by performing feature extraction and feature matching.

### Claim 12 (depends on 11)

The device according to claim 11, wherein mapping, based on the connection relationship and the set of coordinate parameters of the image capturing device, the plurality of effective images of the image sequence to the mapping surface defined with respect to the set of coordinate parameters of the image capturing device comprises the computer-executable instructions cause the device to carry out:
determining, based on the connection relationship and the set of coordinate parameters of the image capturing device, a Euler angle of the image capturing device corresponding to the images of the image sequence;
obtaining a sub-panoramic image corresponding to the image sequence by mapping, based on the Euler angle of the image capturing device corresponding to the images of the image sequence, each of the image of the image sequence to the mapping surface;
determining characteristics of each sub-panoramic image; and
obtaining the panoramic image by merging, based on the characteristics of each of the sub-panoramic image, each of the sub-panoramic image.

### Claim 13 (depends on 10)

The device according to claim 10, wherein determining the connection relationship between the plurality of effective images of the image sequence comprises the computer-executable instructions cause the device to carry out:
in response to the image sequence being taken in a longitudinally discrete manner, determining a mapping relationship between each target image and corresponding other images of the image sequence;
obtaining a fused image corresponding to the image sequence by fusing, based on the mapping relationship, the corresponding other images into each of the target image of the image sequence; and
determining the connection relationship between each fused image corresponding to the image sequence by performing feature extraction and feature matching.

### Claim 14 (depends on 13)

The device according to claim 13, wherein mapping, based on the connection relationship and the set of coordinate parameters of the image capturing device, the plurality of effective images of the image sequence to the mapping surface defined with respect to the set of coordinate parameters of the image capturing device comprises the computer-executable instructions cause the device to carry out:
obtaining a sub-panoramic image corresponding to each fused image by mapping each of the fused image to the mapping surface;
obtaining characteristics of each sub-panoramic image; and
obtaining the panoramic image by merging, based on the characteristics of each of the sub-panoramic image, each of the sub-panoramic image.

### Claim 15

A non-transitory computer-readable medium having computer-executable instructions stored thereon which, when executed by one or more processors, cause a device to carry out:
receiving an image sequence of a target space, the image sequence including a plurality of effective images;
determining a connection relationship between the plurality of effective images of the image sequence;
determining a set of coordinate parameters of an image capturing device associated with the image sequence; and
obtaining a panoramic image based on the connection relationship, the set of coordinate parameters of the image capturing device, and the image sequence.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium according to claim 15, wherein obtaining the panoramic image based on the connection relationship, the set of coordinate parameters of the image capturing device, and the image sequence comprises the computer-executable instructions cause the device to carry out:
mapping, based on the connection relationship and the set of coordinate parameters of the image capturing device, the plurality of effective images of the image sequence to a mapping surface defined with respect to the set of coordinate parameters of the image capturing device.

### Claim 17 (depends on 16)

The non-transitory computer-readable medium according to claim 16, wherein determining the connection relationship between the plurality of effective images of the image sequence comprises the computer-executable instructions cause the device to carry out:
in response to the image sequence being taken in a laterally discrete manner, determining that each image of the image sequence is an effective image, and determining the connection relationship between images of the image sequence by performing feature extraction and feature matching.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium according to claim 17, wherein mapping, based on the connection relationship and the set of coordinate parameters of the image capturing device, the plurality of effective images of the image sequence to the mapping surface defined with respect to the set of coordinate parameters of the image capturing device comprises the computer-executable instructions cause the device to carry out:
determining, based on the connection relationship and the set of coordinate parameters of the image capturing device, a Euler angle of the image capturing device corresponding to the images of the image sequence;
obtaining a sub-panoramic image corresponding to the image sequence by mapping, based on the Euler angle of the image capturing device corresponding to the images of the image sequence, each of the image of the image sequence to the mapping surface;
determining characteristics of each sub-panoramic image; and
obtaining the panoramic image by merging, based on the characteristics of each of the sub-panoramic image, each of the sub-panoramic image.

### Claim 19 (depends on 16)

The non-transitory computer-readable medium according to claim 16, wherein determining the connection relationship between the plurality of effective images of the image sequence comprises the computer-executable instructions cause the device to carry out:
in response to the image sequence being taken in a longitudinally discrete manner, determining a mapping relationship between each target image and corresponding other images of the image sequence;
obtaining a fused image corresponding to the image sequence by fusing, based on the mapping relationship, the corresponding other images into each of the target image of the image sequence; and
determining the connection relationship between each fused image corresponding to the image sequence by performing feature extraction and feature matching.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium according to claim 19, wherein mapping, based on the connection relationship and the set of coordinate parameters of the image capturing device, the plurality of effective images of the image sequence to the mapping surface defined with respect to the set of coordinate parameters of the image capturing device comprises the computer-executable instructions cause the device to carry out:
obtaining a sub-panoramic image corresponding to each fused image by mapping each of the fused image to the mapping surface;
obtaining characteristics of each sub-panoramic image; and
obtaining the panoramic image by merging, based on the characteristics of each of the sub-panoramic image, each of the sub-panoramic image.
