# US 2021/0174124 A1 · Method, device and storage medium for determining camera posture information

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174124 A1
- Document ID: US20210174124A1
- Status: Published application
- Kind: A1
- Application number: 17/183,200
- Filed: 2021-02-23
- Published: 2021-06-10
- Earliest claimed: 2017-07-07
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Xiangkai LIN; Linchao BAO; Wei LIU
- Web page: https://patentgrep.com/patent/US20210174124A1

## CPC

- G06T 7/73: using feature-based methods
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 3/02: Affine transformations (for image registration G06T3/147; for image mosaicing G06T3/4038)
- G06T 3/04: Context-preserving transformations, e.g. by using an importance map (panospheric to cylindrical image transformations G06T3/12)
- G06T 5/20: using local operators
- G06T 7/00: Image analysis
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/248: {involving reference images or patches}
- G06T 7/269: using gradient-based methods
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/74: {involving reference images or patches}
- G06V 10/10: Image acquisition (document image scanning and transmission H04N1/00; control of digital cameras H04N23/60)
- G06V 10/17: {using hand-held instruments}
- G06V 10/225: {based on a marking or identifier characterising the area}
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/40: Extraction of image or video features
- G06V 10/443: {by matching or filtering}
- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06V 10/75: Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V 10/757: {Matching configurations of points or features}
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/30204: Marker
- G06T 2207/30208: Marker matrix
- G06T 2207/30244: Camera pose
- G06V 20/20: in augmented reality scenes
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Embodiments of this application disclose a method for determining camera pose information of a camera of a mobile terminal. The method includes: obtaining a first image, a second image, and a template image, the first image being a previous frame of image of the second image, the first image and the second image being images including a respective instance of the template image captured by the mobile terminal using the camera at a corresponding spatial position; determining a first homography between the template image and the second image; determining a second homography between the first image and the second image; and performing complementary filtering processing on the first homography and the second homography, to obtain camera pose information of the camera, wherein the camera pose information of the camera represents a spatial position of the mobile terminal when the mobile terminal captures the second image using the camera.

## Claims

### Claim 1

A method for determining camera pose information of a camera of a mobile terminal having one or more processors and memory storing a plurality of instructions to be executed by the one or more processors, the method comprising:
obtaining a first image, a second image, and a template image, the first image being a previous frame of image of the second image, the first image and the second image being images including a respective instance of the template image captured by the mobile terminal using the camera at a corresponding spatial position, and the template image being a reference image used for matching;
determining a first homography between the template image and the second image;
determining a second homography between the first image and the second image; and
performing complementary filtering processing on the first homography and the second homography, to obtain camera pose information of the camera, wherein the camera pose information of the camera represents a spatial position of the mobile terminal when the mobile terminal captures the second image using the camera.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the operation of performing complementary filtering processing on the first homography and the second homography, to obtain camera pose information of the camera comprises:
determining a first rotation and translation matrix according to the first homography, and determining a second rotation and translation matrix according to the second homography, the first homography and the second homography being two-dimensional information, and the first rotation and translation matrix and the second rotation and translation matrix being three-dimensional information; and
performing complementary filtering processing on the first rotation and translation matrix and the second rotation and translation matrix, to obtain the camera pose information.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the operation of performing complementary filtering processing on the first rotation and translation matrix and the second rotation and translation matrix, to obtain the camera pose information comprises:
inputting the first rotation and translation matrix to a low-pass filter, to obtain a first filtering result;
inputting the second rotation and translation matrix to a high-pass filter, to obtain a second filtering result; and
determining the camera pose information according to the first filtering result and the second filtering result.

### Claim 4 (depends on 2)

The method according to claim 2, wherein the operation of determining a first rotation and translation matrix according to the first homography, and determining a second rotation and translation matrix according to the second homography comprises:
calculating the first rotation and translation matrix according to the first homography, a perspective projection matrix of the second image, and a perspective projection matrix of the template image; and
calculating the second rotation and translation matrix according to the second homography, the perspective projection matrix of the second image, and the perspective projection matrix of the template image.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the template image corresponds to a plurality of grids arranged in arrays; and
the operation of determining a first homography between the template image and the second image comprises:
matching a plurality of first feature points in the template image with a plurality of second feature points in the second image, and determining a set of feature point pairs in each target grid of the plurality of grids, the feature point pair comprising: a first feature point located in the target grid, and a corresponding second feature point that has the largest matching degree with the first feature point; and
calculating the first homography between the template image and the second image according to the set of feature point pairs in the target grid.

### Claim 6 (depends on 5)

The method according to claim 5, further comprising:
extracting the first feature point from each image layer of the template image, and determining the first feature point in an original image layer,
the original image layer being an image layer of the template image, and the original image layer comprising the plurality of grids.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the operation of extracting the first feature point from each image layer of the template image, and determining the first feature point in an original image layer comprises:
extracting the first feature point from a first image layer of the template image;
extracting the first feature point from a second image layer of the template image, the first image layer and the second image layer having different sizes; and
performing scaling processing on the first feature point in the first image layer and the first feature point in the second image layer, and performing projection to the original image layer.

### Claim 8 (depends on 5)

The method according to claim 5, wherein the operation of matching a plurality of first feature points in the template image with a plurality of second feature points in the second image, and determining a set of feature point pairs in each target grid of the plurality of grids further comprises:
matching the first feature point in each of the target grid of the original image layer with the plurality of second feature points, and obtaining at least one pair of mutually matched to-be-selected feature point pairs, each set of the to-be-selected feature point pairs corresponding to a matching score; and
selecting a feature point pair having the highest matching score from the at least one pair of to-be-selected feature point pairs, as a feature point pair determined in the target grid.

### Claim 9 (depends on 1)

The method according to claim 1, wherein the operation of determining a second homography between the first image and the second image comprises:
obtaining a first optical flow feature point in a preset region of the first image, the preset region being a region including the corresponding instance of the template image;
obtaining a second optical flow feature point according to the first optical flow feature point; and
calculating a first target homography from the first image to the second image according to the first optical flow feature point and the second optical flow feature point.

### Claim 10 (depends on 9)

The method according to claim 9, further comprising:
determining the second homography between the first image and the second image according to the first target homography from the first image to the second image and a second target homography from the template image to the first image.

### Claim 11

A mobile terminal having a camera, one or more processors, memory and a plurality of instructions stored in the memory that, when executed by the one or more processors, cause the mobile terminal to perform operations comprising:
obtaining a first image, a second image, and a template image, the first image being a previous frame of image of the second image, the first image and the second image being images including a respective instance of the template image captured by the mobile terminal using the camera at a corresponding spatial position, and the template image being a reference image used for matching;
determining a first homography between the template image and the second image;
determining a second homography between the first image and the second image; and
performing complementary filtering processing on the first homography and the second homography, to obtain camera pose information of the camera, wherein the camera pose information of the camera represents a spatial position of the mobile terminal when the mobile terminal captures the second image using the camera.

### Claim 12 (depends on 11)

The mobile terminal according to claim 11, wherein the operation of performing complementary filtering processing on the first homography and the second homography, to obtain camera pose information of the camera comprises:
determining a first rotation and translation matrix according to the first homography, and determining a second rotation and translation matrix according to the second homography, the first homography and the second homography being two-dimensional information, and the first rotation and translation matrix and the second rotation and translation matrix being three-dimensional information; and
performing complementary filtering processing on the first rotation and translation matrix and the second rotation and translation matrix, to obtain the camera pose information.

### Claim 13 (depends on 12)

The mobile terminal according to claim 12, wherein the operation of performing complementary filtering processing on the first rotation and translation matrix and the second rotation and translation matrix, to obtain the camera pose information comprises:
inputting the first rotation and translation matrix to a low-pass filter, to obtain a first filtering result;
inputting the second rotation and translation matrix to a high-pass filter, to obtain a second filtering result; and
determining the camera pose information according to the first filtering result and the second filtering result.

### Claim 14 (depends on 12)

The mobile terminal according to claim 12, wherein the operation of determining a first rotation and translation matrix according to the first homography, and determining a second rotation and translation matrix according to the second homography comprises:
calculating the first rotation and translation matrix according to the first homography, a perspective projection matrix of the second image, and a perspective projection matrix of the template image; and
calculating the second rotation and translation matrix according to the second homography, the perspective projection matrix of the second image, and the perspective projection matrix of the template image.

### Claim 15 (depends on 11)

The mobile terminal according to claim 11, wherein the template image corresponds to a plurality of grids arranged in arrays; and
the operation of determining a first homography between the template image and the second image comprises:
matching a plurality of first feature points in the template image with a plurality of second feature points in the second image, and determining a set of feature point pairs in each target grid of the plurality of grids, the feature point pair comprising: a first feature point located in the target grid, and a corresponding second feature point that has the largest matching degree with the first feature point; and
calculating the first homography between the template image and the second image according to the set of feature point pairs in the target grid.

### Claim 16 (depends on 11)

The mobile terminal according to claim 11, wherein the operation of determining a second homography between the first image and the second image comprises:
obtaining a first optical flow feature point in a preset region of the first image, the preset region being a region including the corresponding instance of the template image;
obtaining a second optical flow feature point according to the first optical flow feature point; and
calculating a first target homography from the first image to the second image according to the first optical flow feature point and the second optical flow feature point.

### Claim 17

A non-transitory computer readable storage medium storing a plurality of instructions in connection with a mobile terminal having a camera, one or more processors, memory and a plurality of instructions stored in the memory that, when executed by the one or more processors, cause the mobile terminal to perform a plurality of operations including:
obtaining a first image, a second image, and a template image, the first image being a previous frame of image of the second image, the first image and the second image being images including a respective instance of the template image captured by the mobile terminal using the camera at a corresponding spatial position, and the template image being a reference image used for matching;
determining a first homography between the template image and the second image;
determining a second homography between the first image and the second image; and
performing complementary filtering processing on the first homography and the second homography, to obtain camera pose information of the camera, wherein the camera pose information of the camera represents a spatial position of the mobile terminal when the mobile terminal captures the second image using the camera

### Claim 18 (depends on 17)

The non-transitory computer readable storage medium according to claim 17, wherein the operation of performing complementary filtering processing on the first homography and the second homography, to obtain camera pose information of the camera comprises:
determining a first rotation and translation matrix according to the first homography, and determining a second rotation and translation matrix according to the second homography, the first homography and the second homography being two-dimensional information, and the first rotation and translation matrix and the second rotation and translation matrix being three-dimensional information; and
performing complementary filtering processing on the first rotation and translation matrix and the second rotation and translation matrix, to obtain the camera pose information.

### Claim 19 (depends on 18)

The non-transitory computer readable storage medium according to claim 18, wherein the operation of performing complementary filtering processing on the first rotation and translation matrix and the second rotation and translation matrix, to obtain the camera pose information comprises:
inputting the first rotation and translation matrix to a low-pass filter, to obtain a first filtering result;
inputting the second rotation and translation matrix to a high-pass filter, to obtain a second filtering result; and
determining the camera pose information according to the first filtering result and the second filtering result.

### Claim 20 (depends on 18)

The non-transitory computer readable storage medium according to claim 18, wherein the operation of determining a first rotation and translation matrix according to the first homography, and determining a second rotation and translation matrix according to the second homography comprises:
calculating the first rotation and translation matrix according to the first homography, a perspective projection matrix of the second image, and a perspective projection matrix of the template image; and
calculating the second rotation and translation matrix according to the second homography, the perspective projection matrix of the second image, and the perspective projection matrix of the template image.
