US 11,538,229 B2Granted

Image processing method and apparatus, electronic device, and computer-readable storage medium

AssigneeTencent Technology (Shenzhen) Company Limited

InventorsYuanli Zheng, Zhaopeng Gu, Nianhua Xie

Life of this patent

Estimating the term…

Loading patent…

Abstract

The present disclosure provides an image processing method and apparatus, an electronic device, and a computer-readable storage medium. The method includes: obtaining a to-be-processed video comprising m frames of images, m being a positive integer greater than or equal to 2; placing a three-dimensional model on a target plane of a first frame of image of the to-be-processed video, a plurality of feature points of a model surface of the three-dimensional model falling on the target plane; determining three-dimensional coordinates of the plurality of feature points of the model surface in a world coordinate system and pixel coordinates of the plurality of feature points of the model surface on the first frame of image; determining, according to the three-dimensional coordinates of the plurality of feature points of the model surface in the world coordinate system and the pixel coordinates of the plurality of feature points of the model surface on the first frame of image, a pose of a camera coordinate system of the first frame of image relative to the world coordinate system; determining, according to the target plane, the three-dimensional coordinates of the plurality of feature points of the model surface in the world coordinate system, and the pixel coordinates of the plurality of feature points of the model surface on the first frame of image, poses of camera coordinate systems of a second frame of image to an mth frame of image of the to-be-processed video relative to the world coordinate system; and replacing the three-dimensional model with a target model and placing the target model on the world coordinate system to generate, according to the pose of the camera coordinate system of each frame of image of the to-be-processed video relative to the world coordinate system, a target video comprising the target model.

Claims17

Dependent claims

A method for image processing, comprising:

obtaining a video comprising m frames of images, m being a positive integer greater than or equal to 2;
placing a three-dimensional (3D) model on a target plane of a first frame of the video, a plurality of feature points of a model surface of the 3D model falling on the target plane;
determining 3D coordinates of the plurality of feature points of the model surface in a world coordinate system as reference 3D coordinates, and determining pixel coordinates of the plurality of feature points of the model surface on the first frame of the video as initial pixel coordinates;
determining, according to the reference 3D coordinates and the initial pixel coordinates, a pose of a camera coordinate system of the first frame of the video relative to the world coordinate system;
for a jth frame selected from a second frame of the video to an mth frame of the video, j being an integer greater than 1 and less than or equal to m:
determining a current homography matrix as a homography matrix of a target plane on the jth frame of the video relative to the target plane on the first frame of the video by:
in response to j being equal to 2, determining the current homography matrix as a unit matrix;
and
in response to j being greater than 2:
optimizing a homography matrix of a target plane on a (j−1)th frame of the vide according to:
a residual between a pixel value of each pixel of the (j−1)th frame and a corresponding pixel value of each pixel of a (j−2)th frame, to obtain the current homography matrix;
and
determining, according to the current homography matrix and the initial pixel coordinates, pixel coordinates of the plurality of feature points of the model surface on the jth frame of the video;
determining, according to a camera intrinsic parameter of the video, the reference 3D coordinates, and the pixel coordinates of the plurality of feature points of the model surface on the jth frame of the video, a pose of a camera coordinate system of the jth frame of the video relative to the world coordinate system;
and
replacing the 3D model with a target model and placing the target model on the world coordinate system to generate, according to the pose of the camera coordinate system of each frame of the video relative to the world coordinate system, a target video comprising the target model.

13 more elements

A device for image processing, comprising a memory for storing instructions and a processor in communication with the memory, wherein the processor is configured to execute the instructions to cause the device to:

obtain a video comprising m frames of images, m being a positive integer greater than or equal to 2;
place a 3D model on a target plane of a first frame of the video, a plurality of feature points of a model surface of the 3D model falling on the target plane;
determine 3D coordinates of the plurality of feature points of the model surface in a world coordinate system as reference 3D coordinates, and determine pixel coordinates of the plurality of feature points of the model surface on the first frame of the video as initial pixel coordinates;
determine, according to the reference 3D coordinates and the initial pixel coordinates, a pose of a camera coordinate system of the first frame of the video relative to the world coordinate system;
for a jth frame selected from a second frame of the video to an mth frame of the video, j being an integer greater than 1 and less than or equal to m:
determine a current homography matrix as a homography matrix of a target plane on the jth frame of the video relative to the target plane on the first frame of the video by:
in response to j being equal to 2, determine the current homography matrix as a unit matrix;
and
in response to j being greater than 2:
optimize a homography matrix of a target plane on a (j−1)th frame of the vide according to:
a residual between a pixel value of each pixel of the (j−1)th frame and a corresponding pixel value of each pixel of a (j−2)th frame, to obtain the current homography matrix;
and
determine, according to the current homography matrix and the initial pixel coordinates, pixel coordinates of the plurality of feature points of the model surface on the jth frame of the video;
determine, according to a camera intrinsic parameter of the video, the reference 3D coordinates, and the pixel coordinates of the plurality of feature points of the model surface on the jth frame of the video, a pose of a camera coordinate system of the jth frame of the video relative to the world coordinate system;
and
replace the 3D model with a target model and placing the target model on the world coordinate system to generate, according to the pose of the camera coordinate system of each frame of the video relative to the world coordinate system, a target video comprising the target model.

13 more elements

A non-transitory storage medium for storing computer readable instructions, the computer readable instructions, when executed by a processor, causing the processor to:

obtain a video comprising m frames of images, m being a positive integer greater than or equal to 2;
place a 3D model on a target plane of a first frame of the video, a plurality of feature points of a model surface of the 3D model falling on the target plane;
determine 3D coordinates of the plurality of feature points of the model surface in a world coordinate system as reference 3D coordinates, and determine pixel coordinates of the plurality of feature points of the model surface on the first frame of the video as initial pixel coordinates;
determine, according to the reference 3D coordinates and the initial pixel coordinates, a pose of a camera coordinate system of the first frame of the video relative to the world coordinate system;
for a jth frame selected from a second frame of the video to an mth frame of the video, j being an integer greater than 1 and less than or equal to m:
determine a current homography matrix as a homography matrix of a target plane on the jth frame of the video relative to the target plane on the first frame of the video by:
in response to j being equal to 2, determine the current homography matrix as a unit matrix;
and
in response to j being greater than 2:
optimize a homography matrix of a target plane on a (j−1)th frame of the vide according to:
a residual between a pixel value of each pixel of the (j−1)th frame and a corresponding pixel value of each pixel of a (j−2)th frame, to obtain the current homography matrix;
and
determine, according to the current homography matrix and the initial pixel coordinates, pixel coordinates of the plurality of feature points of the model surface on the jth frame of the video;
determine, according to a camera intrinsic parameter of the video, the reference 3D coordinates, and the pixel coordinates of the plurality of feature points of the model surface on the jth frame of the video, a pose of a camera coordinate system of the jth frame of the video relative to the world coordinate system;
and
replace the 3D model with a target model and placing the target model on the world coordinate system to generate, according to the pose of the camera coordinate system of each frame of the video relative to the world coordinate system, a target video comprising the target model.

13 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/185,393Filed 25 Feb 2021 · continuation
Priority: CN 201910854877.X10 Sep 2019
Published: US 2021/0183165 A117 Jun 2021
Granted: 27 Dec 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification17 codes

G06T 19/006
Manipulating three-dimensional [3D] models or images for computer graphics: mixed reality
G06T 19/20
Manipulating three-dimensional [3D] models or images for computer graphics: editing of three-dimensional [3D] images, e.g. changing shapes or…
G06N 3/0464
Architecture, e.g. interconnection topology: convolutional networks [CNN, ConvNet]
G06N 3/09
Learning methods: supervised learning