# US 2024/0198535 A1 · Method and device for annotating images of an object captured using a camera > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2024/0198535 A1 - Document ID: US20240198535A1 - Status: Published application - Kind: A1 - Application number: 18/555,693 - Filed: 2022-03-24 - Published: 2024-06-20 - Earliest claimed: 2021-04-16 - Assignees: Volkswagen Aktiengesellschaft - Applicants: Volkswagen Aktiengesellschaft - Inventors: Kai Londenberg; Zsolt Farkas; Daniel Schulze; Maxim Derksen; Lisa Rangosch; Peter Motzko; Robert Schirmer - Web page: https://patentgrep.com/patent/US20240198535A1 ## CPC - B25J 9/1692: {Calibration of manipulator} - G06T 7/75: {involving models} - B25J 9/161: {Hardware, e.g. neural networks, fuzzy logic, interfaces, processor} - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]} - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - B25J 9/1687: {Assembly, peg and hole, palletising, straight line, weaving pattern movement} - G06T 2207/10016: Video; Image sequence - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30164: Workpiece; Machine component - G06T 2207/30204: Marker - G06T 2207/30244: Camera pose - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Technologies and techniques for annotating images of an object captured using a camera, wherein the camera is positioned on a robot arm, and wherein the robot arm includes an end effector. A position and an orientation of the camera is determined, relative to the end effector, and a first keypoint is selected on the object. Information regarding additional keypoints relative to the first keypoint is provided in object coordinates, and the end effector is guided to the position of the first keypoint. The position and the orientation of the end effector is determined at the first keypoint, and based on other determined world coordinates, the position and/or the orientation of the end effector is changed based on converted camera image coordinates. ## Claims ### Claim at position 1 1-12. (canceled) ### Claim 13 A method for annotating images of an object captured using a camera, the camera being configured on a robot arm comprising an end effector, comprising: determining a position and an orientation of the camera relative to the end effector; selecting a first keypoint on the object; providing information regarding additional keypoints relative to the first keypoint in object coordinates; guiding the end effector to the position of the first keypoint; determining the position and the orientation of the end effector at the first keypoint in world coordinates; determining the position of the first keypoint in world coordinates; determining the positions of the additional keypoints in world coordinates; altering the position and/or the orientation of the end effector and capturing an image via the camera; determining camera coordinates of the keypoints by converting the world coordinates of the keypoints into a camera coordinate system of the camera; and determining camera image coordinates of all keypoints using projection from the camera coordinates thereof. ### Claim 14 (depends on 13) The method according to claim 13, further comprising not manually identifying keypoints on images captured by means of the camera. ### Claim 15 (depends on 13) The method according to claim 13, further comprising not applying markings on the object so as to recognize keypoints on the captured image. ### Claim 16 (depends on 13) The method according to claim 13, wherein the determining of a position and of an orientation of the camera relative to the end effector comprises a hand-eye calibration. ### Claim 17 (depends on 13) The method according to claim 13, wherein the providing of information regarding additional keypoints comprises measuring the respective positions of the additional keypoints with respect to the first keypoint at the object. ### Claim 18 (depends on 13) The method according to claim 13, wherein the providing of information regarding additional keypoints comprises utilizing existing information regarding dimensions of the object. ### Claim 19 (depends on 13) The method according to claim 13, wherein the guiding the end effector to the position of the first keypoint comprises manually moving the end effector to the first keypoint. ### Claim 20 (depends on 13) The method according to claim 13, further comprising generating training data for an artificial network comprising the annotated images. ### Claim 21 A device for annotating images of an object, comprising: a robot arm comprising an end effector; a camera configured to capture images, the camera being configured on the robot arm; a processing apparatus, operatively coupled to the camera, the processing device being configured to determine a position and an orientation of the camera relative to the end effector; select a first keypoint on the object; provide information regarding additional keypoints relative to the first keypoint in object coordinates; guide the end effector to the position of the first keypoint; determine the position and the orientation of the end effector at the first keypoint in world coordinates; determine the position of the first keypoint in world coordinates; determine the positions of the additional keypoints in world coordinates; alter the position and/or the orientation of the end effector and capturing an image via the camera; determine camera coordinates of the keypoints by converting the world coordinates of the keypoints into a camera coordinate system of the camera; and determine camera image coordinates of all keypoints using projection from the camera coordinates thereof. ### Claim 22 (depends on 21) The device according to claim 21, wherein the processing apparatus is configured to not manually identify keypoints on images captured via the camera. ### Claim 23 (depends on 21) The device according to claim 21, wherein the processing apparatus is configured to not apply markings on the object so as to recognize keypoints on the captured image. ### Claim 24 (depends on 21) The device according to claim 21, wherein the processing apparatus is configured to determine a position and of an orientation of the camera relative to the end effector using a hand-eye calibration. ### Claim 25 (depends on 21) The device according to claim 21, wherein the processing apparatus is configured to provide information regarding additional keypoints by measuring the respective positions of the additional keypoints with respect to the first keypoint at the object. ### Claim 26 (depends on 21) The device according to claim 21, wherein the processing apparatus is configured to provide of information regarding additional keypoints comprises utilizing existing information regarding dimensions of the object. ### Claim 27 (depends on 21) The device according to claim 21, wherein the processing apparatus is configured to guide the end effector to the position of the first keypoint by moving the end effector to the first keypoint. ### Claim 28 (depends on 21) The device according to claim 21, wherein the processing apparatus is configured to generate training data for an artificial network comprising the annotated images. ### Claim 29 A method for annotating images of an object captured using a camera for training data of an artificial neural network, the camera being configured on a robot arm comprising an end effector, comprising: determining a position and an orientation of the camera relative to the end effector; selecting a first keypoint on the object; providing information regarding additional keypoints relative to the first keypoint in object coordinates; guiding the end effector to the position of the first keypoint; determining the position and the orientation of the end effector at the first keypoint in world coordinates; determining the position of the first keypoint in world coordinates; determining the positions of the additional keypoints in world coordinates; altering the position and/or the orientation of the end effector and capturing an image via the camera; determining camera coordinates of the keypoints by converting the world coordinates of the keypoints into a camera coordinate system of the camera; and determining camera image coordinates of all keypoints for the training data using projection from the camera coordinates. ### Claim 30 (depends on 29) The method according to claim 29, wherein the providing of information regarding additional keypoints comprises measuring the respective positions of the additional keypoints with respect to the first keypoint at the object. ### Claim 31 (depends on 29) The method according to claim 29, wherein the providing of information regarding additional keypoints comprises utilizing existing information regarding dimensions of the object. ### Claim 32 (depends on 26) The method according to claim 26, wherein the guiding the end effector to the position of the first keypoint comprises manually moving the end effector to the first keypoint.