# US 10,373,369 B2 · Three-dimensional pose estimation of symmetrical objects > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,373,369 B2 - Document ID: US10373369B2 - Status: Granted - Kind: B2 - Application number: 15/692,401 - Filed: 2017-08-31 - Published: 2018-09-20 - Granted: 2019-08-06 - Earliest claimed: 2017-03-16 - Assignees: QUALCOMM Technologies, Inc. - Applicants: QUALCOMM Incorporated - Inventors: Mahdi Rad; Markus Oberweger; Vincent Lepetit - Web page: https://patentgrep.com/patent/US10373369B2 ## CPC - G06T 15/205: {Image-based rendering} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/2413: based on distances to training or reference patterns - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/75: {involving models} - G06V 20/647: {by matching two-dimensional images to three-dimensional objects} - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20068: Projection on vertical or horizontal image axis - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The present disclosure describes methods, apparatuses, and non-transitory computer-readable mediums for estimating a three-dimensional (“3D”) pose of an object from a two-dimensional (“2D”) input image which contains the object. Particularly, certain aspects of the disclosure are concerned with 3D pose estimation of a symmetric or nearly-symmetric object. An image or a patch of an image includes the object. A classifier is used to determine whether a rotation angle of the object in the image or the patch of the image is within a first predetermined range. In response to a determination that the rotation angle is within the first predetermined range, a mirror image of the object is determined. Two-dimensional (2D) projections of a three-dimensional (3D) bounding box of the object are determined by applying a trained regressor to the mirror image of the object in the image or the patch of the image. The 3D pose of the object is estimated based on the 2D projections. ## Claims ### Claim 1 A method for estimating a three-dimensional (“3D”) pose of an object in an image having an angle of rotational symmetry, the method comprising: obtaining the image or a patch of the image which includes the object; determining, via a classifier, whether a rotation angle of the object in the image or the patch of the image is within a first predetermined range; and in response to a determination that the rotation angle is within the first predetermined range: determining a mirror image of the object; determining two-dimensional (2D) projections of a three-dimensional (3D) bounding box of the object by applying a trained regressor to the mirror image of the object in the image or the patch of the image; and estimating the 3D pose of the object based on the 2D projections. ### Claim 2 (depends on 1) The method of claim 1 wherein the trained regressor is trained using a plurality of training images, wherein each of the training images is taken from a different point of view and includes the object with a different pose from a plurality of poses, wherein each of the plurality of poses has a respectively different rotation angle, wherein each respective rotation angle is determined with respect to a predetermined orientation for the object, and wherein each respective rotation angle of the plurality of poses is in a second predetermined range. ### Claim 3 (depends on 2) The method of claim 2 wherein the first predetermined range is β/2<α≤β, where α is the rotation angle, and β is the angle of rotational symmetry. ### Claim 4 (depends on 2) The method of claim 2 wherein a second predetermined range is 0°≤αp≤β/2, where αp is the rotation angle, and β is the angle of rotational symmetry. ### Claim 5 (depends on 2) The method of claim 2 wherein a second predetermined range is 0°−δ≤αp≤(β/2)+δ, wherein αp is the rotation angle, β is the angle of rotational symmetry, and δ is a non-zero angle. ### Claim 6 (depends on 5) The method of claim 5 wherein δ is smaller than β. ### Claim 7 (depends on 1) The method of claim 1, further comprising refining the 2D projections by inputting the image or the patch of the image, and the estimated 3D pose of the object to a neural network. ### Claim 8 (depends on 7) The method of claim 7, wherein the refining comprises: generating a respective mask or rendering of the object for each respective one of a plurality of poses around and including the estimated 3D pose of the object; and determining an update to the 2D projections based on which of the renderings or masks correlates most with the image or the patch of the image. ### Claim 9 (depends on 1) The method of claim 1, wherein determining 2D projections of a 3D bounding box of the object includes determining 2D projections of a plurality of corners of the 3D bounding box. ### Claim 10 (depends on 1) The method of claim 1, wherein determining the mirror image of the object includes: transforming a plurality of location coordinates of the object from an initial coordinate system into a second coordinate system having two orthogonal axes in a plane of symmetry of the object and a third orthogonal axis normal to the plane of symmetry; inverting the coordinates about the plane of symmetry; and transforming the mirror image into the initial coordinate system. ### Claim 11 (depends on 1) The method of claim 1 wherein estimating the 3D pose of the object includes: determining a first 3D bounding box of the mirror image of the object in the image or the patch of the image; determining a second 3D bounding box wherein the second 3D bounding box is a mirror image of the first 3D bounding box; determining a plurality of 2D projections of the second 3D bounding box; and estimating the 3D pose of the object using the plurality of 2D projections. ### Claim 12 (depends on 1) The method of claim 1 wherein the regressor includes a convolutional neural network. ### Claim 13 (depends on 1) The method of claim 1 wherein the regressor includes a deep neural network. ### Claim 14 (depends on 1) The method of claim 1 wherein the classifier includes a convolutional neural network. ### Claim 15 (depends on 1) The method of claim 1, further comprising, in response to a determination that the rotation angle is in a second predetermined range, estimating the 3D pose of the object by applying the trained regressor to the image or the patch of the image. ### Claim 16 (depends on 15) The method of claim 15 wherein each of a plurality of poses has a respective rotation angle, each respective rotation angle is determined with respect to a predetermined orientation for the object, each respective rotation angle of the plurality of poses is in a third predetermined range, and the third predetermined range from 0° to β/2, where β is the angle of rotational symmetry. ### Claim 17 (depends on 15) The method of claim 15 wherein the classifier is used to determine if α is in the second predetermined range. ### Claim 18 An apparatus for estimating a three-dimensional (“3D”) pose of an object in an image having an angle of rotational symmetry, comprising: a memory configured for storing the image or a patch of the image; and a processor configured to: determine, via a classifier, whether a rotation angle of the object in the image or the patch of the image is within a first predetermined range; and in response to a determination that the rotation angle is within the first predetermined range: determine a mirror image of the object; determine two-dimensional (2D) projections of a three-dimensional (3D) bounding box of the object by applying a trained regressor to the mirror image of the object in the image or the patch of the image; and estimate the 3D pose of the object based on the 2D projections. ### Claim 19 (depends on 18) The apparatus of claim 18 wherein the 3D pose of the object is estimated by: determining a first 3D bounding box of the mirror image of the object in the stored image or patch of the image; determining a second 3D bounding box wherein the second 3D bounding box is a mirror image of the first 3D bounding box; determining a plurality of 2D projections of the second 3D bounding box; and estimating the 3D pose of the object using the plurality of 2D projections. ### Claim 20 (depends on 18) The apparatus of claim 18 wherein the regressor is a convolutional neural network. ### Claim 21 (depends on 18) The apparatus of claim 18 wherein the classifier is a convolutional neural network. ### Claim 22 (depends on 18) The apparatus of claim 18, wherein the processor is configured to, estimate the 3D pose of the object by applying the trained regressor to the image or the patch of the image of the object, in response to a determination that a is in a second predetermined range. ### Claim 23 (depends on 18) The apparatus of claim 18, wherein the processor is further configured to refine the 2D projections by inputting the image or the patch of the image and the estimated 3D pose of the object to a neural network. ### Claim 24 (depends on 23) The apparatus of claim 23, wherein the neural network is configured to: generate a respective mask or rendering of the object for each respective one of a plurality of poses around and including the estimated 3D pose of the object; and determine an update to the 2D projections based on which of the renderings or masks correlates most with the image or the patch of the image. ### Claim 25 An apparatus for estimating a three-dimensional (“3D”) pose of an object in an image having an angle of rotational symmetry, comprising: means for storing the image or a patch of the image; means for using a classifier to determine whether a rotation angle of the object in the image or the patch of the image is within a first predetermined range; means for determining a mirror image of the object in response to a determination that the rotation angle of the object in the image or the patch of the image is within the first predetermined range; means for using a regressor to determine two-dimensional (2D) projections of a three-dimensional (3D) bounding box of the object to the mirror image of the object in the image or the patch of the image; and means for estimating the 3D pose of the object in response to the determination of 2D projections. ### Claim 26 A non-transitory computer-readable storage medium having stored thereon or therein instructions that, when executed by a processor, perform a method for estimating a three-dimensional (“3D”) pose of an object in an image having an angle of rotational symmetry, the method comprising: obtaining the image or a patch of the image which includes the object; determining, via a classifier, whether a rotation angle of the object in the image or the patch of the image is within a first predetermined range; and in response to a determination that the rotation angle is within the first predetermined range: determining a mirror image of the object, estimating two-dimensional (2D) projections of a three-dimensional (3D) bounding box of the object by applying a trained regressor to the mirror image of the object in the image or the patch of the image, and estimating the 3D pose of the object given the 2D projections. ### Claim 27 (depends on 26) The non-transitory computer-readable medium of claim 26, wherein α is in a second predetermined range, estimating the 3D pose of the object by applying the trained regressor to the image or the patch of the image of the object.