# US 10,307,910 B2 · Apparatus of recognizing position of mobile robot using search based correlative matching and method thereof > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,307,910 B2 - Document ID: US10307910B2 - Status: Granted - Kind: B2 - Application number: 15/382,331 - Filed: 2016-12-16 - Published: 2017-06-01 - Granted: 2019-06-04 - Earliest claimed: 2014-06-17 - Applicants: Yujin Robot Co., Ltd. - Inventors: Kyung Chul Shin; Seong Ju Park; Jae Young Lee - Web page: https://patentgrep.com/patent/US10307910B2 ## CPC - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - B25J 9/16: Program controls (total factory control, i.e. centrally controlling a plurality of machines, G05B19/418) - B25J 9/1697: {Vision controlled systems} - B25J 13/08: by means of sensing devices, e.g. viewing or touching devices - B25J 19/023: {including video camera means} - G01C 21/1656: {with passive imaging devices, e.g. cameras} - G05D 1/0246: {using a video camera in combination with image processing means} - G05D 1/0251: {extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision (stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593)} - G05D 1/0253: {extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow (determining position or orientation from images G06T7/70)} - G05D 1/027: {comprising intertial navigation means, e.g. azimuth detector (inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165)} - G05D 1/0272: {comprising means for registering the travel distance, e.g. revolutions of wheels (measuring distance traversed on the ground by vehicles, e.g. using odometers G01C22/00)} - G05D 1/0274: {using mapping information stored in a memory device (navigation using map-matching G01C21/30)} - G06T 7/11: Region-based segmentation - G06T 7/13: Edge detection - G06T 7/174: involving the use of two or more images - G06T 7/74: {involving reference images or patches} - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G06V 10/507: {Summing image-intensity values; Histogram projection analysis} - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06T 2207/10012: Stereo images - G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence - G06T 2207/20021: Dividing image into blocks, subimages or windows - G06T 2207/30244: Camera pose - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06V 20/64: Three-dimensional [3D] objects - Y10S 901/01: Mobile robot - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - Y10S: TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS ## Abstract Disclosed are an apparatus of recognizing a position of a mobile robot using search based correlative matching and a method thereof. An apparatus of recognizing a position of a mobile robot includes two wide angle cameras which obtain one pair of stereo images on a region of interest including a vertical direction and a horizontal direction in accordance with movement of a mobile robot; and a position recognizing unit which extracts an edge from the received stereo image, forms a virtual space using at least one key frame selected based on the extracted edge, and estimates the current position using the edge size which is calculated for the formed space. ## Claims ### Claim 1 A mobile robot comprising: a camera system; at least one processor; and a non-transitory computer-readable medium storing instructions that, when executed by the at least one processor, cause the mobile robot to capture, by the camera system, image data of an object, generate at least one pair of stereo images from the image data, extract an edge from the at least one pair of stereo images, form a virtual space using at least one key frame selected based on the extracted edge, calculate an edge size for the virtual space, and estimate a position of the mobile robot using the edge size. ### Claim 2 (depends on 1) The mobile robot of claim 1, wherein the non-transitory computer-readable medium further stores instructions that, when executed by the at least one processor, cause the mobile robot to divide the virtual space into a plurality of regions and calculate the edge size from the plurality of regions. ### Claim 3 (depends on 1) The mobile robot of claim 1, wherein the camera system includes first and second cameras. ### Claim 4 (depends on 2) The mobile robot of claim 2, wherein the non-transitory computer-readable medium further stores instructions that, when executed by the at least one processor, cause the mobile robot to: divide the virtual space into the plurality of regions; sequentially project each of a plurality of seeds in the key frame onto the at least one pair of stereo images in each of the plurality of divided regions; calculate the edge size in accordance with a coordinate of each of the plurality of seeds in the key frame; and estimate which of the plurality of regions has a largest edge sizes as the position of the mobile robot. ### Claim 5 (depends on 2) The mobile robot of claim 2, wherein the non-transitory computer-readable medium further stores instructions that, when executed by the at least one processor, cause the mobile robot to select which of the plurality of regions has a largest edge sizes, check whether to satisfy a predetermined repetition condition to re-estimate the position in the region of the selected position candidate, and estimate the selected position candidate as the position of the mobile robot when the repetition condition is not satisfied as the checking result. ### Claim 6 (depends on 5) The mobile robot of claim 5, wherein the non-transitory computer-readable medium further stores instructions that, when executed by the at least one processor, cause the mobile robot to: divide the region of the selected position candidate into the plurality of regions when the repetition condition is satisfied as the checking result; and select a region having the largest sum of the edge sizes calculated in each of the plurality of divided regions as a position candidate. ### Claim 7 (depends on 5) The mobile robot of claim 5, wherein the repetition condition includes whether the number of times of dividing the region reaches a predetermined threshold value or whether a variation on the sum of the calculated edge sizes is within a predetermined threshold value. ### Claim 8 (depends on 2) The mobile robot of claim 2, wherein the non-transitory computer-readable medium further stores instructions that, when executed by the at least one processor, cause the mobile robot to obtain a gradient in each pixel of an image including the edge in a first axis direction and a second axis direction, calculate a magnitude of the gradient using the calculated gradient in the first axis direction and the second axis direction, and set the calculated magnitude of the gradient as a size of the edge. ### Claim 9 (depends on 8) The mobile robot of claim 8, wherein the first axis and the second axis are orthogonal to each other in a coordinate system of an image plane including the edge. ### Claim 10 (depends on 1) The mobile robot of claim 1, wherein the non-transitory computer-readable medium further stores instructions that, when executed by the at least one processor, cause the mobile robot to predict a movement point using odometry information calculated based on the edge and estimate the position of the mobile robot using the predicted movement point and a key frame that is adjacent to the movement point. ### Claim 11 (depends on 1) The mobile robot of claim 1, wherein the non-transitory computer-readable medium further stores instructions that, when executed by the at least one processor, cause the mobile robot to predict a movement point using one of first odometry information calculated based on the edge and second odometry information calculated based on inertia information and estimate the position of the mobile robot using the predicted movement point and a key frame that is adjacent to the movement point. ### Claim 12 (depends on 11) The mobile robot of claim 11, wherein the non-transitory computer-readable medium further stores instructions that, when executed by the at least one processor, cause the mobile robot to select the first odometry information when the first odometry information satisfies the predetermined condition and select the second odometry information when the first odometry information does not satisfy the predetermined condition. ### Claim 13 (depends on 11) The mobile robot of claim 11, wherein the non-transitory computer-readable medium further stores instructions that, when executed by the at least one processor, cause the mobile robot to search for at least one adjacent key frame from among a previously stored key frame set based on the predicted state information and estimate the position of the mobile robot using the at least one adjacent key frame. ### Claim 14 (depends on 1) The mobile robot of claim 1, wherein the non-transitory computer-readable medium further stores instructions that, when executed by the at least one processor, cause the mobile robot to extract the edge by either: obtaining gradients in a first axis direction and a second axis direction from each pixel of the received stereo image to extract the edge based on the gradients in the first axis direction and the second axis direction, or performing smoothing processing on the received stereo image and obtaining a smoothing image as the smoothing result and comparing the obtained smoothing image and the stereo image. ### Claim 15 A non-transitory computer-readable medium storing instructions that, when executed by a computing apparatus cause the computing apparatus to: receive at least one pair of stereo images from a camera system of a mobile robot; extract an edge from the at least one pair of stereo images; form a virtual space using at least one key frame selected based on the extracted edge; calculate an edge size for the virtual space; and estimate a position of the mobile robot using the edge size. ### Claim 16 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the instructions, when executed by the computing apparatus, further cause the computing apparatus to divide the virtual space into a plurality of regions and calculate the edge size from the plurality of divided regions. ### Claim 17 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein the estimating includes: forming the virtual space having a predetermined region with respect to the key frame; sequentially projecting each of a plurality of seeds in the key frame onto the stereo images in each of the plurality of regions; calculating the edge size in accordance with a coordinate of each projected seed in the key frame; and setting a region having the largest edge sizes calculated in each of the plurality of regions as the position of the mobile robot. ### Claim 18 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein in the estimating, a region having the largest sum of the edge sizes calculated in each of the plurality of regions is selected as a position candidate, it is checked whether to satisfy a predetermined repetition condition to re-estimate the position in the region of the selected position candidate, and the selected position candidate is estimated as the position of the mobile robot when the repetition condition is not satisfied as the checking result. ### Claim 19 (depends on 18) The non-transitory computer-readable medium of claim 18, wherein the estimating includes repeatedly performing; dividing the region of the selected position candidate into the plurality of regions when the repetition condition is satisfied as the checking result; and selecting a region having the largest sum of the edge sizes calculated in each of the plurality of regions as the position. ### Claim 20 (depends on 18) The non-transitory computer-readable medium of claim 18, wherein the repetition condition includes whether the number of times of dividing the region reaches a predetermined threshold value or whether a variation on the sum of the calculated edge sizes is within a predetermined threshold value.