# US 2023/0410360 A1 · System and Method for Load-Carrier Pose Estimation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0410360 A1 - Document ID: US20230410360A1 - Status: Published application - Kind: A1 - Application number: 17/886,839 - Filed: 2022-08-12 - Published: 2023-12-21 - Earliest claimed: 2022-06-15 - Assignees: Avular Holding B.V. - Applicants: Ai In Motion B.V. - Inventors: Gijs Dubbelman; Jonathan Broere; Guus Engels - Web page: https://patentgrep.com/patent/US20230410360A1 ## CPC - G06T 7/74: {involving reference images or patches} - G06T 7/75: {involving models} - B66F 9/063: {Automatically guided} - B66F 9/0755: {Position control; Position detectors} - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30164: Workpiece; Machine component - G06T 2207/30268: Vehicle interior - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - B66F: HOISTING, LIFTING, HAULING OR PUSHING, NOT OTHERWISE PROVIDED FOR, e.g. DEVICES WHICH APPLY A LIFTING OR PUSHING FORCE DIRECTLY TO THE SURFACE OF A LOAD ({invalid lifting devices A61G7/10; } mounting artificial islands on piles or like supports E02B17/00; scaffolding combined with lifting devices E04G1/22; lifting devices for sliding forms E04G11/24; lifting of buildings E04G23/06; shores or struts E04G25/00; lifts or other hoisting devices on ladders E06C7/12; props for mining E21D15/00) ## Abstract A system for Load-carrier Pose estimation. The system includes a Load-carrier for supporting a load with Load-carrier handling indicators or Key-points, and a driving vehicle. The Pose represents a relative position and orientation of the Load-carrier with reference to the vehicle. The system includes a 2D sensor and a 3D sensor. The system is arranged to detect 2D Keypoints in images of the Load-carrier obtained with the 2D sensor, and to match these 2D Keypoints with 3D data captured with the 3D sensor to establish a 3D position and orientation of the detected Keypoints. ## Claims ### Claim 1 A system for Load-carrier Pose estimation, the system comprising: a Load-carrier for supporting a load, wherein the Load-carrier is provided with Load-carrier handling indicators or Keypoints and a driving vehicle; a Pose representing a relative position and orientation of the Load-carrier with reference to the vehicle; and a 2D sensor and a 3D sensor, wherein the system is arranged to carry out a first estimation phase in which a Load-carrier detection is carried out, followed by a second estimation phase wherein a Pose estimation is applied, and wherein the second estimation phase is embodied with 2D key point detection and 3D matching of such 2D Keypoints. ### Claim 2 (depends on 1) The system according to claim 1, wherein the system is arranged to detect 2D Keypoints in images of the Load-carrier obtained with the 2D sensor, and to match these 2D Keypoints with 3D data captured with the 3D sensor to establish a 3D position and orientation of the detected Keypoints. ### Claim 3 (depends on 1) The system according to claim 1, wherein in a third estimation phase the system employs Pose estimation using 3D template matching initialized by the Pose estimation resulting from the second phase. ### Claim 4 (depends on 3) The system according to claim 3, wherein the 3D position and orientation of the detected Keypoints of the Load-carrier in the place for a final alignment is supplied to a processor of the system that carries out the 3D template matching optimization. ### Claim 5 (depends on 1) The system according to claim 1, wherein the system comprises a 3D point cloud template of the Load-carrier, wherein the system is arranged to superimpose the 3D point cloud template onto 3D data derived with the 3D sensor, which 3D data represents the Load-carrier in 3D space, and wherein said superposition enables computing the Load-carrier's position. ### Claim 6 (depends on 1) The system according to claim 1, wherein the 2D sensor and/or the 3D sensor are mounted on the vehicle, thus providing said sensors with a fixed 6 degrees of freedom offset for use in a 2D to 3D coordinate matrix transformation. ### Claim 7 (depends on 1) The system according to claim 1, further comprising an IMU sensor. ### Claim 8 (depends on 1) The system according to claim 1, wherein the system operates in real-time enabling to estimate the Pose at least once while the vehicle is moving. ### Claim 9 (depends on 1) The system according to claim 1, further comprising an image processor connected to the 2D sensor for detecting the Load-carrier and/or 2D Keypoints which generates image data at a frequency of more than 30 Hz. ### Claim 10 (depends on 9) The system according to claim 9, wherein the image processor combines the detection of the 2D Key-points with Load-carrier detection. ### Claim 11 (depends on 9) The system according to claim 9, wherein the image processor detects the Load-carrier and the 2D Keypoints using machine learning/deep learning by applying the same deep neural network for detecting the Load-carrier and the 2D Keypoints simultaneously. ### Claim 12 (depends on 1) The system according to claim 1, wherein the Load-carrier is a pallet. ### Claim 13 (depends on 1) The system according to claim 1, wherein the vehicle is a pallet lifter or forklift. ### Claim 14 A method for Load-carrier Pose estimation of a Load-carrier, wherein the Load-carrier comprises Load-carrier handling indicators or Keypoints, wherein the Pose represents a relative position and orientation of the Load-carrier with reference to a vehicle, the method comprising the steps of: employing a 2D sensor and a 3D sensor; executing a first estimation phase in which a Load-carrier detection is carried out; and executing a second estimation phase subsequent to the first estimation phase applying a Pose estimation and employing 2D key point detection and 3D matching of such 2D Keypoints. ### Claim 15 (depends on 14) The method according to claim 14 further comprising the steps of: detecting 2D Key-points in images of the Load-carrier obtained with the 2D sensor; and matching these 2D Keypoints with 3D data captured with the 3D sensor to establish a 3D position and orientation of the detected Keypoints. ### Claim 16 (depends on 1) The method of claim 1 further comprising the step of executing a third estimation phase employing Pose estimation using 3D template matching initialized by the Pose estimation resulting from the second phase. ### Claim 17 (depends on 16) The method of claim 16 further comprising the step of: providing the 3D position and orientation of the detected Keypoints of the Load-carrier in the place for a final alignment to initialize the 3D template matching optimization. ### Claim 18 (depends on 16) The method of claim 16 further comprising the step of superimposing a template of the Load-carrier onto 3D data derived with the 3D sensor, which 3D data represents the Load-carrier in 3D space, wherein said superposition enables computing the Load-carrier's position. ### Claim 19 (depends on 14) The method according to claim 14 further comprising the step of estimating the Pose while the vehicle is moving. ### Claim 20 (depends on 14) The method according to claim 14, further comprising the step of providing an image processor connected to the 2D sensor for detecting the Load-carrier and/or the 2D Keypoints which generates image data at a frequency of more than 30 Hz. ### Claim 21 (depends on 20) The method according to claim 20, further comprising the step of using the image processor to detect the Load-carrier and/or 2D Keypoints using machine learning/deep learning. ### Claim 22 (depends on 20) The method according to claim 20 further comprising the step of arranging that the image processor combines the detection of the 2D Keypoints with Load-carrier detection. ### Claim 23 (depends on 22) The method according to claim 22 further comprising the step of detecting the 2D Keypoints and the Load-carrier by using the same deep neural network simultaneously. ### Claim 24 A system for Load-carrier Pose estimation, comprising: a Load-carrier for supporting a load, wherein the Load-carrier comprises Load-carrier handling indicators or Keypoints, and a driving vehicle; a Pose representing a relative position and orientation of the Load-carrier with reference to the vehicle; a 2D sensor and a 3D sensor, wherein the system is arranged to detect 2D Keypoints in images of the Load-carrier obtained with the 2D sensor, and to match these 2D Keypoints with 3D data captured with the 3D sensor to establish a 3D position and orientation of the detected Keypoints.