# US 9,519,289 B2 · Systems and methods for performing simultaneous localization and mapping using machine vision systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,519,289 B2 - Document ID: US9519289B2 - Status: Granted - Kind: B2 - Application number: 14/856,526 - Filed: 2015-09-16 - Published: 2016-05-26 - Granted: 2016-12-13 - Earliest claimed: 2014-11-26 - Assignees: iRobot Corporation - Applicants: iRobot Corporation - Inventors: Mario Munich; Nikolai Romanov; Dhiraj Goel; Philip Fong - Web page: https://patentgrep.com/patent/US9519289B2 ## CPC - G05D 1/0246: {using a video camera in combination with image processing means} - B25J 5/007: {mounted on wheels} - B25J 19/005: {using batteries, e.g. as a back-up power source} - G01C 21/206: {specially adapted for indoor navigation} - G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00) - G05D 1/0274: {using mapping information stored in a memory device (navigation using map-matching G01C21/30)} - Y10S 901/01: Mobile robot - Y10S 901/47: Optical - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - Y10S: TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS ## Abstract The present invention provides a mobile robot configured to navigate an operating environment, that includes a controller circuit that directs a drive of the mobile robot to navigate the mobile robot through an environment using camera-based navigation system and a camera including optics defining a camera field of view and a camera optical axis, where the camera is positioned within the recessed structure and is tilted so that the camera optical axis is aligned at an acute angle of above a horizontal plane in line with the top surface and is aimed in a forward drive direction of the robot body, and the camera is configured to capture images of the operating environment of the mobile robot. ## Claims ### Claim 1 A mobile robot configured to navigate an operating environment, comprising: a body having a top surface; a drive mounted to the body; a recessed structure beneath the plane of the top surface near a geometric center of the body; a controller circuit in communication with the drive, wherein the controller circuit directs the drive to navigate the mobile robot through an environment using camera-based navigation system; and a camera including optics defining a camera field of view and a camera optical axis, wherein: the camera is positioned within the recessed structure and is tilted so that the camera optical axis is aligned at an acute angle of 30-40 degrees above a horizontal plane in line with the top surface and is aimed in a forward drive direction of the robot body, the field of view of the camera spans a frustum of 45-65 degrees in the vertical direction, and the camera is configured to capture images of the operating environment of the mobile robot. ### Claim 2 (depends on 1) The mobile robot of claim 1, wherein the camera is protected by a lens cover aligned at an acute angle with respect to the optical axis of the camera. ### Claim 3 (depends on 2) The mobile robot of claim 2, wherein the lens cover is set back relative to an opening of the recessed structure and an acute angle with respect to the optical axis of the camera that is closer to perpendicular than an angle formed between a plane defined by the top surface and the optical axis of the camera. ### Claim 4 (depends on 2) The mobile robot of claim 2, wherein the acute angle is between 15 and 70 degrees. ### Claim 5 (depends on 3) The mobile robot of claim 3, wherein the angle formed between a plane defined by the opening in the recessed structure and the optical axis of the camera ranges between 10 and 60 degrees. ### Claim 6 (depends on 1) The mobile robot of claim 1, wherein the camera field of view is aimed at static features located in a range of 3 feet to 8 feet from a floor surface at a distance of 3 feet to 10 feet from the static features. ### Claim 7 (depends on 6) The mobile robot of claim 6, wherein the camera images contain about 6-12 pixels per inch and features at the top of the image move upward between successive images more quickly than the speed at which the mobile robot moves and features at the bottom of the image move downward between successive images more slowly than the speed at which the mobile robot moves and wherein the controller is configured to determine the speed of the mobile robot and location of features in the image in identifying disparity between successive images. ### Claim 8 (depends on 7) The mobile robot of claim 7 wherein the mobile robot moves at a velocity of 220 mm per second to 450 mm per second and features lower than 45 degrees relative to the horizon will track slower than approximately 306 mm per second and features higher than 45 degrees will track faster than 306 mm per second. ### Claim 9 (depends on 1) The mobile robot of claim 1, wherein the body further contains: a memory in communication with the controller circuit; and an odometry sensor system in communication with the drive, wherein the memory further contains a visual measurement application, a simultaneous location and mapping (SLAM) application, a landmarks database, and a map of landmarks, wherein the controller circuit directs a processor to: actuate the drive and capture odometry data using the odometry sensor system; acquire a visual measurement by providing at least the captured odometry information and the captured image to the visual measurement application; determine an updated robot pose within an updated map of landmarks by providing at least the odometry information, and the visual measurement as inputs to the SLAM application; and determine robot behavior based upon inputs including the updated robot pose within the updated map of landmarks. ### Claim 10 (depends on 9) The mobile robot of claim 9, wherein the landmarks database comprises: descriptions of a plurality of landmarks; a landmark image of each of the plurality of landmarks and an associated landmark pose from which the landmark image was captured; and descriptions of a plurality of features associated with a given landmark from the plurality of landmarks including a 3D position for each of the plurality of features associated with the given landmark. ### Claim 11 (depends on 10) The mobile robot of claim 10, wherein the visual measurement application directs the processor to: identify features within an input image; identify a landmark from the landmark database in the input image based upon the similarity of the features identified in the input image to matching features associated with a landmark image of the identified landmark in the landmark database; and estimate a most likely relative pose by determining a rigid transformation of the 3D structure of the matching features associated with the identified landmark that results in the highest degree of similarity with the identified features in the input image, where the rigid transformation is determined based upon an estimate of relative pose and the acute angle at which the optical axis of the camera is aligned above the direction of motion of the mobile robot. ### Claim 12 (depends on 11) The mobile robot of claim 11, wherein identifying a landmark in the input image comprises comparing unrectified image patches from the input image to landmark images within the landmark database. ### Claim 13 (depends on 11) The mobile robot of claim 11, wherein the SLAM application directs the processor to: estimate the location of the mobile robot within the map of landmarks based upon a previous location estimate, odometry data and at least one visual measurement; and update the map of landmarks based upon the estimated location of the mobile robot, the odometry data, and the at least one visual measurement. ### Claim 14 (depends on 9) The mobile robot of claim 9, wherein the visual measurement application further directs the processor to generate new landmarks by: detecting features within images in a sequence of images; identifying a set of features forming a landmark in multiple images from the sequence of images; estimating 3D structure of the set of features forming a landmark and relative robot poses at the times each of the multiple images is captured using the identified set of features forming the landmark in each of the multiple images; recording a new landmark in the landmark database, where recording the new landmark comprises storing: an image of the new landmark, at least the set of features forming the new landmark, and the 3D structure of the set of features forming the new landmark; and notify the SLAM application of the creation of a new landmark. ### Claim 15 (depends on 14) The mobile robot of claim 14, wherein the SLAM application directs a processor to: determine a landmark pose as the pose of the mobile robot at the time the image of the new landmark stored in the landmark database is captured; and recording the landmark pose for the new landmark in the landmark database. ### Claim 16 (depends on 1) The mobile robot of claim 1, wherein the machine vision sensor system further comprises a plurality of cameras under the top surface of the robot and having focal axes angled upward relative to the horizontal plane of the top surface and wherein at least one camera faces a reverse drive direction opposite to the forward drive direction. ### Claim 17 (depends on 1) The mobile robot of claim 1, wherein the machine vision system comprises a stereo pair of cameras having overlapping fields of view. ### Claim 18 (depends on 1) The mobile robot of claim 1, wherein the camera is positioned within the recessed structure at a distance of at most 6 inches from a floor surface. ### Claim 19 A mobile robot configured to navigate an operating environment, comprising: a body that comprises a top surface located at a height not more than 6 inches from a bottom surface, the body comprising a recessed structure under the top surface, the body containing: at least one processor; memory containing a behavioral controller application, where the behavioral controller application directs a processor to navigate an environment based on the captured images; a machine vision sensor system comprising a camera configured to capture images of the operating environment of the mobile robot, the camera including optics defining a camera field of view and a camera optical axis, where the camera is positioned within the recessed structure and is tilted so that the optical axis is aligned at an acute angle between 10 and 60 degrees above a forward drive direction of motion of the mobile robot. ### Claim 20 (depends on 19) The mobile robot of claim 19, the body further containing: a lens cover protecting the camera, wherein the lens cover is set back relative to an opening of the recessed structure at an acute angle with respect to the optical axis of the camera that is closer to perpendicular than an angle formed between a plane defined by the opening in the recessed structure and the optical axis of the camera.