# US 12,602,822 B2 · Method device and storage medium for back-end optimization of simultaneous localization and mapping > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,822 B2 - Document ID: US12602822B2 - Status: Granted - Kind: B2 - Application number: 18/569,692 - Filed: 2022-05-23 - Published: 2024-07-04 - Granted: 2026-04-14 - Earliest claimed: 2021-07-07 - Assignees: Beijing Zitiao Network Technology Co., Ltd. - Applicants: Beijing Zitiao Network Technology Co., Ltd. - Inventors: Jiawei Wen - Web page: https://patentgrep.com/patent/US12602822B2 ## CPC - G06T 7/579: from motion - G06T 7/73: using feature-based methods - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06T 2207/10016: Video; Image sequence - G06T 2207/30244: Camera pose - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The disclosure provides a method, device and storage medium for back-end optimization of simultaneous localization and mapping. In the method, a target three-dimensional space point is determined in three-dimensional space points of a plurality of key frames located by a simultaneous localization and mapping system. In addition, a reprojection error is determined based on the target three-dimensional space point, and a weight is provided for the reprojection error by preprocessing the maximum parallax corresponding to the target three-dimensional space point, to obtain the relative reprojection error. ## Claims ### Claim 1 A method of back-end optimization of simultaneous localization and mapping, comprising: determining a target three-dimensional space point in three-dimensional space points of a plurality of key frames located by a simultaneous localization and mapping system; determining a reprojection error based on the target three-dimensional space point; determining a relative reprojection error based on the reprojection error and a maximum parallax corresponding to the target three-dimensional space point after preprocessing, the preprocessing comprising an operation of removing influence of rotation; and performing back-end optimization to the simultaneous localization and mapping system based on the relative reprojection error. ### Claim 2 (depends on 1) The method according to claim 1, wherein before the determining a relative reprojection error based on the reprojection error and a maximum parallax corresponding to the target three-dimensional space point after preprocessing, the method further comprises: determining a plurality of target key frames corresponding to the target three-dimensional space point; determining a plurality of parallaxes corresponding to the target three-dimensional space point after the preprocessing based on key point coordinates of a key frame ui, key point coordinates of a key frame uj, an intrinsic matrix of an image obtaining apparatus in the simultaneous localization and mapping system, and a rotation matrix of the key frame uj to the key frame ui, wherein the key frames ui and uj are any two of the plurality of target key frames, i=1, 2, . . . m, j=1, 2, . . . m, and m is equal to the number of the plurality of target key frames; and determining the maximum parallax from the plurality of parallaxes corresponding to the target three-dimensional space point after the preprocessing. ### Claim 3 (depends on 2) The method according to claim 2, wherein determining the plurality of parallaxes corresponding to the target three-dimensional space point after the preprocessing based on key point coordinates of a key frame ui, key point coordinates of a key frame uj, an intrinsic matrix of an image obtaining apparatus in the simultaneous localization and mapping system, and a rotation matrix of the key frame uj to the key frame ui comprises: calculating a product of the intrinsic matrix, the rotation matrix, an inverse matrix of the intrinsic matrix, and the key point coordinates of the key frame uj; and determining the plurality of parallaxes corresponding to the target three-dimensional space point after the preprocessing based on a difference between the key point coordinates of the key frame ui and the product. ### Claim 4 (depends on 1) The method according to claim 1, wherein determining the relative reprojection error based on the reprojection error and a maximum parallax corresponding to the target three-dimensional space point after preprocessing comprises: calculating a ratio of the reprojection error to the maximum parallax; and determining the ratio as the relative reprojection error. ### Claim 5 (depends on 1) The method according to claim 1, wherein determining the reprojection error based on the target three-dimensional space point comprises: determining a position obtained by projecting the target three-dimensional space point; and determining the reprojection error based on the position obtained by the projection. ### Claim 6 (depends on 1) The method according to claim 1, wherein performing back-end optimization to the simultaneous localization and mapping system based on the relative reprojection error comprises: deciding whether the relative reprojection error reaches a predetermined error threshold or not; and if the relative reprojection error does not reach the predetermined error threshold, reperforming the act of determining the target three-dimensional space point in the three-dimensional space points of a plurality of key frames located by the simultaneous localization and mapping system, so as to cause the relative reprojection error to reach the predetermined error threshold, and performing the back-end optimization to the simultaneous localization and mapping system based on the relative reprojection error. ### Claim 7 (depends on 1) The method according to claim 1, wherein before the determining a target three-dimensional space point in three-dimensional space points of a plurality of key frames located by a simultaneous localization and mapping system, the method further comprises: initializing the simultaneous localization and mapping system. ### Claim 8 (depends on 7) The method according to claim 7, wherein initializing the simultaneous localization and mapping system comprises: obtaining a predetermined number of continuous frame images, and performing the preprocessing to the predetermined number of continuous frame images; screening, with a pre-built adaptive-size sliding window, initial key frames in the predetermined number of continuous frame images with the influence of rotation removed, the initial key frame comprising the plurality of key frames; and performing simultaneous localization and mapping initialization based on the plurality of key frames. ### Claim 9 (depends on 8) The method according to claim 8, wherein performing simultaneous localization and mapping initialization based on the plurality of key frame comprises: determining relative poses of a first key frame and a last key frame in the plurality of key frames; obtaining three-dimensional space points of respective key frames in the plurality of key frames based on the relative poses of the first key frame and the last key frame; determining relative poses of respective key frames in the plurality of key frames based on the relative poses of the first key frame and the last key frame, and the three-dimensional space points of respective key frames in the plurality of key frames; and establishing an initial map based on the three-dimensional space points of respective key frames in the plurality of key frames and the relative poses of respective key frames in the plurality of key frames. ### Claim 10 A device for back-end optimization of simultaneous localization and mapping, comprising: a processor; a memory; and a computer program; wherein the computer program is stored in the memory and is configured to be executed by the processor, the computer program comprising instructions for performing operations comprising: determining a target three-dimensional space point in three-dimensional space points of a plurality of key frames located by a simultaneous localization and mapping system, determining a reprojection error based on the target three-dimensional space point, determining a relative reprojection error based on the reprojection error and a maximum parallax corresponding to the target three-dimensional space point after preprocessing, the preprocessing comprising an operation of removing influence of rotation; and performing back-end optimization to the simultaneous localization and mapping system based on the relative reprojection error. ### Claim 11 (depends on 10) The device according to claim 10, wherein the operations further comprise: before the determining a relative reprojection error based on the reprojection error and a maximum parallax corresponding to the target three-dimensional space point after preprocessing, further comprising: determining a plurality of target key frames corresponding to the target three-dimensional space point; determining a plurality of parallaxes corresponding to the target three-dimensional space point after the preprocessing based on key point coordinates of a key frame ui, key point coordinates of a key frame uj, an intrinsic matrix of an image obtaining apparatus in the simultaneous localization and mapping system, and a rotation matrix of the key frame uj to the key frame ui, wherein the key frames ui and uj are any two of the plurality of target key frames, i=1, 2, . . . m, j=1, 2, . . . m, and m is equal to the number of the plurality of target key frames; and determining the maximum parallax from the plurality of parallaxes corresponding to the target three-dimensional space point after the preprocessing. ### Claim 12 (depends on 11) The device according to claim 11, wherein determining the plurality of parallaxes comprises: calculating a product of the intrinsic matrix, the rotation matrix, an inverse matrix of the intrinsic matrix, and the key point coordinates of the key frame uj; and determining the plurality of parallaxes corresponding to the target three-dimensional space point after the preprocessing based on a difference between the key point coordinates of the key frame ui and the product. ### Claim 13 (depends on 10) The device according to claim 10, wherein determining the relative reprojection error comprises: calculating a ratio of the reprojection error to the maximum parallax; and determining the ratio as the relative reprojection error. ### Claim 14 (depends on 10) The device according to claim 10, wherein determining the reprojection error based on the target three-dimensional space point comprises: determining a position obtained by projecting the target three-dimensional space point; and determining the reprojection error based on the position obtained by the projection. ### Claim 15 (depends on 10) The device according to claim 10, wherein performing back-end optimization to the simultaneous localization and mapping system comprises: deciding whether the relative reprojection error reaches a predetermined error threshold or not; and if the relative reprojection error does not reach the predetermined error threshold, reperforming the act of determining the target three-dimensional space point in the three-dimensional space points of a plurality of key frames located by the simultaneous localization and mapping system, so as to cause the relative reprojection error to reach the predetermined error threshold, and performing the back-end optimization to the simultaneous localization and mapping system based on the relative reprojection error. ### Claim 16 (depends on 10) The device according to claim 10, wherein the operations further comprise: before the determining the target three-dimensional space point in three-dimensional space points of a plurality of key frames located by a simultaneous localization and mapping system, initializing the simultaneous localization and mapping system. ### Claim 17 (depends on 16) The device according to claim 16, wherein initializing the simultaneous localization and mapping system comprises: obtaining a predetermined number of continuous frame images, and performing the preprocessing to the predetermined number of continuous frame images; screening, with a pre-built adaptive-size sliding window, initial key frames in the predetermined number of continuous frame images with the influence of rotation removed, the initial key frame comprising the plurality of key frames; and performing simultaneous localization and mapping initialization based on the plurality of key frames. ### Claim 18 (depends on 17) The device according to claim 17, wherein performing simultaneous localization and mapping initialization comprises: determining relative poses of a first key frame and a last key frame in the plurality of key frames; obtaining three-dimensional space points of respective key frames in the plurality of key frames based on the relative poses of the first key frame and the last key frame; determining relative poses of respective key frames in the plurality of key frames based on the relative poses of the first key frame and the last key frame, and the three-dimensional space points of respective key frames in the plurality of key frames; and establishing an initial map based on the three-dimensional space points of respective key frames in the plurality of key frames and the relative poses of respective key frames in the plurality of key frames. ### Claim 19 A non-transitory computer-readable storage medium storing a computer program, wherein when executed by a processor, causes the processor to: determine a target three-dimensional space point in three-dimensional space points of a plurality of key frames located by a simultaneous localization and mapping system; determine a reprojection error based on the target three-dimensional space point; determine a relative reprojection error based on the reprojection error and a maximum parallax corresponding to the target three-dimensional space point after preprocessing, the preprocessing comprising an operation of removing influence of rotation; and perform back-end optimization to the simultaneous localization and mapping system based on the relative reprojection error. ### Claim 20 (depends on 19) The non-transitory computer-readable storage medium of claim 19, wherein the processor is further caused to: before the determining a relative reprojection error based on the reprojection error and a maximum parallax corresponding to the target three-dimensional space point after preprocessing, determine a plurality of target key frames corresponding to the target three-dimensional space point; determine a plurality of parallaxes corresponding to the target three-dimensional space point after the preprocessing based on key point coordinates of a key frame ui, key point coordinates of a key frame uj, an intrinsic matrix of an image obtaining apparatus in the simultaneous localization and mapping system, and a rotation matrix of the key frame uj to the key frame ui, wherein the key frames ui and uj are any two of the plurality of target key frames, i=1, 2, . . . m, j=1, 2, . . . m, and m is equal to the number of the plurality of target key frames; and determine the maximum parallax from the plurality of parallaxes corresponding to the target three-dimensional space point after the preprocessing.