# US 9,519,971 B2 · Position and orientation measurement device and position and orientation measurement method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,519,971 B2 - Document ID: US9519971B2 - Status: Granted - Kind: B2 - Application number: 13/323,504 - Filed: 2011-12-12 - Published: 2012-06-14 - Granted: 2016-12-13 - Earliest claimed: 2010-12-14 - Assignees: Canon Kabushiki Kaisha - Applicants: Daisuke Kotake; Keisuke Tateno; Shinji Uchiyama - Inventors: Daisuke Kotake; Keisuke Tateno; Shinji Uchiyama - Web page: https://patentgrep.com/patent/US9519971B2 ## CPC - G06T 7/75: {involving models} - G06T 2207/10004: Still image; Photographic image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A position and orientation measurement device includes a grayscale image input unit that inputs a grayscale image of an object, a distance image input unit that inputs a distance image of the object, an approximate position and orientation input unit that inputs an approximate position and orientation of the object with respect to the position and orientation measurement device, and a position and orientation calculator that updates the approximate position and orientation. The position and orientation calculator calculates a first position and orientation so that an object image on an image plane and a projection image of the three-dimensional shape model overlap each other, associates the three-dimensional shape model with the image features of the grayscale image and the distance image, and calculates a second position and orientation on the basis of a result of the association. ## Claims ### Claim 1 A position and orientation measurement device that measures a position and orientation of an object to be measured, the device comprising: a processor; and a memory containing instructions that, when executed by the processor, perfom operations as: a storage unit configured to store a three-dimensional shape model of the object; a two-dimensional image input unit configured to input a two-dimensional image representing the object; a distance image input unit configured to input a distance image sensed by a distance sensor, which indicates a distance from the distance sensor to the object; an approximate position and orientation input unit configured to input an approximate position and orientation of the object; and a position and orientation calculation unit configured to calculate, a first position and orientation of the object by associating the three-dimensional shape model with the two-dimensional image based on the approximate position and orientation of the object and thereafter calculate a second position and orientation of the object by associating the three-dimensional shape model with both the two-dimensional image and the distance image based on the first position and orientation. ### Claim 2 (depends on 1) The position and orientation measurement device according to claim 1, wherein the position and orientation calculation unit calculates the first position and orientation of the object by updating a component in six freedoms representing an approximate position and orientation based on the two-dimensional image. ### Claim 3 (depends on 2) The position and orientation measurement device according to claim 2, wherein the component includes a position perpendicular to an optical axis of a camera of the position and orientation measurement device and an orientation around the optical axis. ### Claim 4 (depends on 1) The position and orientation measurement device according to claim 1, wherein the approximate position and orientation input unit inputs a position and orientation of the object obtained from information of at least one of the two-dimensional image and the distance image by a calculation method that does not require an initial value of the position and orientation as an approximate position and orientation. ### Claim 5 (depends on 1) The position and orientation measurement device according to claim 1, wherein the position and orientation calculation unit calculates a position corresponding to a distance from the position and orientation measurement device in the position and orientation by using information of the distance image before deriving the first position and orientation. ### Claim 6 (depends on 1) The position and orientation measurement device according to claim 1, wherein image features in the two-dimensional image used in measuring the position and orientation of the object are edges. ### Claim 7 (depends on 1) The position and orientation measurement device according to claim 1, wherein image features in the two-dimensional image used in measuring the position and orientation of the object are feature points. ### Claim 8 A position and orientation measurement method by a position and orientation measurement device that measures position and orientation of an object to be measured, the method comprising: a storage step for storing a three-dimensional shape model of the object; a two-dimensional image input step for inputting a two-dimensional image representing the object; a distance image input step for inputting a distance image sensed by a distance sensor, which indicates a distance from the distance sensor to the object; an approximate position and orientation input step for inputting an approximate position and orientation of the object; a position and orientation calculation step for calculating, without using the distance image, a first position and orientation of the object by associating the three-dimensional shape model with the two-dimensional image based on the approximate position and orientation of the object and for thereafter calculating a second position and orientation of the object by associating the three-dimensional shape model with both the two-dimensional image of the object and the distance image based on the first position and orientation. ### Claim 9 (depends on 1) The device according to claim 1, further comprising: a two-dimensional image capturing unit configured to sense a two-dimensional image of the object. ### Claim 10 (depends on 1) The device according to claim 1, further comprising: the distance sensor which senses the distance image. ### Claim 11 A non-transitory storage medium having instructions that, when executed by a processor, cause the processor to perform operations comprising: storing a three-dimensional shape model of the object; inputting a two-dimensional image representing of the object; inputting a distance image sensed by a distance sensor, which indicates a distance from the distance sensor to the object; inputting an approximate position and orientation of the object; calculating, without using the distance image, a first position and orientation of the object by associating the three-dimensional shape model with the two-dimensional image based on the approximate position and orientation of the object, and calculating a second position and orientation of the object by associating the three-dimensional shape model with both the two-dimensional image and the distance image based on the first position and orientation. ### Claim 12 (depends on 1) The position and orientation measurement device according to claim 1, wherein the position and orientation calculation unit calculates the first position and orientation of the object by associating the three-dimensional shape model with a feature of the two-dimensional image of the object using the approximate position and orientation of the object and derives a second position and orientation of the object by associating the three-dimensional shape model with a feature of the two-dimensional image of the object and a feature of the distance image of the object using the first position and orientation.