# US 11,443,508 B1 · Methods for an autonomous robotic device to identify locations captured in an image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,508 B1 - Document ID: US11443508B1 - Status: Granted - Kind: B1 - Application number: 17/021,175 - Filed: 2020-09-15 - Granted: 2022-09-13 - Earliest claimed: 2017-12-15 - Assignees: AI Incorporated - Applicants: Ali Ebrahimi Afrouzi; Sebastian Schweigert; Chen Zhang; Hao Yuan - Inventors: Ali Ebrahimi Afrouzi; Sebastian Schweigert; Chen Zhang; Hao Yuan - Web page: https://patentgrep.com/patent/US11443508B1 ## CPC - G06T 7/74: {involving reference images or patches} - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - A46B 3/00: Brushes characterised by the way in which the bristles are fixed or joined in or on the brush body or carrier (machines or appliances therefor A46D) - A47L 9/0488: {Combinations or arrangements of several tools, e.g. edge cleaning tools} - A47L 9/281: {the amount or condition of incoming dirt or dust} - A47L 9/2852: {Elements for displacement of the vacuum cleaner or the accessories therefor, e.g. wheels, casters or nozzles} - A47L 9/2894: {Details related to signal transmission in suction cleaners} - A47L 11/24: Floor-sweeping machines, motor-driven - A47L 11/28: Floor-scrubbing machines, motor-driven (A47L11/29 takes precedence) - A47L 11/4002: {Installations of electric equipment} - A47L 11/4011: {Regulation of the cleaning machine by electric means; Control systems and remote control systems therefor} - A47L 11/4036: {Parts or details of the surface treating tools} - A47L 11/4066: {Propulsion of the whole machine} - B25J 9/1697: {Vision controlled systems} - G01C 21/3837: {Data obtained from a single source} - G05D 1/0219: {ensuring the processing of the whole working surface} - G05D 1/0246: {using a video camera in combination with image processing means} - G05D 1/0274: {using mapping information stored in a memory device (navigation using map-matching G01C21/30)} - G05D 1/648: Performing a task within a working area or space, e.g. cleaning - G06T 7/12: Edge-based segmentation - G06T 7/136: involving thresholding - G06T 7/162: involving graph-based methods - G06T 7/187: involving region growing; involving region merging; involving connected component labelling - G06V 10/443: {by matching or filtering} - G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching} - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - G06V 40/162: {using pixel segmentation or colour matching} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - A46B 2200/3033: Household brush, i.e. brushes for cleaning in the house or dishes - A47L 2201/04: Automatic control of the travelling movement; Automatic obstacle detection - G06T 2207/10024: Color image - G06T 2207/20072: Graph-based image processing - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G10L 2015/223: {Execution procedure of a spoken command} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - A46B: BRUSHES (handles not integral with brushware B25G) - A47L: DOMESTIC WASHING OR CLEANING (brushes A46B; cleaning quantities of bottles or of other hollow articles of one kind B08B9/00; laundry D06F); SUCTION CLEANERS IN GENERAL (cleaning in general B08) - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Provided are operations including: receiving, with one or more processors of a robot, an image of an environment from an imaging device separate from the robot; obtaining, with the one or more processors, raw pixel intensity values of the image; extracting, with the one or more processors, objects and features in the image by grouping pixels with similar raw pixel intensity values, and by identifying areas in the image with greatest change in raw pixel intensity values; determining, with the one or more processors, an area within a map of the environment corresponding with the image by comparing the objects and features of the image with objects and features of the map; and, inferring, with the one or more processors, one or more locations captured in the image based on the location of the area of the map corresponding with the image. ## Claims ### Claim 1 One or more tangible, non-transitory, machine-readable medium storing instructions that when executed by one or more processors effectuate operations comprising: receiving, with the one or more processors of a robot, an image of an environment from an imaging device separate from the robot; obtaining, with the one or more processors of the robot, raw pixel intensity values of the image; extracting, with the one or more processors of the robot, objects and features in the image by grouping pixels with similar raw pixel intensity values, and by identifying areas in the image with greatest change in raw pixel intensity values; determining, with the one or more processors of the robot, an area within a map of the environment corresponding with the image by comparing the objects and features of the image with objects and features of the map; inferring, with the one or more processors of the robot, one or more locations captured in the image based on the location of the area of the map corresponding with the image; and, instructing, with the one or more processors of the robot, the robot to navigate to the one or more locations captured in the image; wherein the robot is paired with an application of a smart device configured to: receive an input designating a modification to the map of the environment; and transmit the modified map to at least one of: the one or more processors of the robot via the cloud, the cloud, the one or more processors of the robot and the cloud, and the one or more processors of the robot. ### Claim 2 (depends on 1) The one or more media of claim 1, wherein obtaining the raw pixel intensity values of the image comprises normalizing pixel intensity values of the image, and forming the raw pixel intensity values of the image based on the normalized pixel intensity values. ### Claim 3 (depends on 1) The one or more media of claim 1, wherein pixels with similar pixel intensity values are grouped together using thresholding. ### Claim 4 (depends on 1) The one or more media of claim 1, wherein the areas of the image with greatest change in raw pixel intensity values are determined by mapping a continuous function to at least a portion of the raw pixel intensity values and taking the derivative of the function. ### Claim 5 (depends on 1) The one or more media of claim 1, wherein the areas of the image with greatest change in raw pixel intensity values are edges, corners, or doors. ### Claim 6 (depends on 1) The one or more media of claim 1, wherein the area within the map of the environment corresponding with the image is determined by matching a pattern in raw pixel intensity values of the image to raw pixel intensity values of different areas within the map and choosing the area within the map that best matches the pattern in the raw pixel intensity values of the image as the area corresponding with the image. ### Claim 7 (depends on 6) The one or more media of claim 6, wherein the pattern is an edge, corner, or door. ### Claim 8 (depends on 1) The one or more media of claim 1, wherein the operations further comprise: instructing, with the one or more processors of the robot, the robot to execute a task in the one or more locations. ### Claim 9 (depends on 1) The one or more media of claim 1, wherein the robot performs work in the one or more locations captured in the image. ### Claim 10 (depends on 1) The one or more media of claim 1, wherein the operations further comprise: extracting, with the one or more processor of the robot, features in the image based on raw sensor data of the image and a first and second derivative of the raw sensor data. ### Claim 11 (depends on 1) The one or more media of claim 1, wherein: the application is further configured to receive an input designating at least one of: a type of work to be performed by the robot in a particular area chosen within the image, a particular setting of the robot in a particular area chosen within the image, a particular time to perform work by the robot in a particular area chosen within the image, and a virtual boundary in a particular location within the image; and the operations further comprise: updating, with the one or more processors of the robot, the map based on the virtual boundary. ### Claim 12 (depends on 1) The one or more media of claim 1, wherein the application is further configured to receive an input designating an instruction comprising at least one of: a path of the robot, an adjustment to the path of the robot, an adjustment to the map, an order of rooms within the environment the robot is to perform work within, areas within the environment the robot is to perform work within, areas within the environment the robot is to avoid, a type of work the robot is to perform within an area, a location within the environment the robot is to navigate to, a modification to a pre-existing label of an area within the environment, and a label for an area within the environment. ### Claim 13 An apparatus, comprising: a robot comprising: a chassis; drive wheels connected to the chassis; one or more processors; one or more sensors; and, one or more tangible, non-transitory, machine-readable medium storing instructions that when executed by the one or more processors effectuate operations comprising: receiving, with the one or more processors of the robot, an image of an environment captured by an imaging device separate from the robot; obtaining, with the one or more processors of the robot, raw pixel intensity values of the image; extracting, with the one or more processors of the robot, objects and features in the image by grouping pixels with similar raw pixel intensity values, and by identifying areas in the image with greatest change in raw pixel intensity values; determining, with the one or more processors of the robot, an area within a map of the environment corresponding with the image by comparing the objects and features of the image with objects and features of the map; inferring, with the one or more processors of the robot, one or more locations captured in the image based on the location of the area of the map corresponding with the image; and, instructing, by the one or more processors of the robot, the robot to navigate to the location captured in the image and execute a task; wherein the robot is paired with an application of a smart device configured to: receive an input designating a modification to the map of the environment; and transmit the modified map to at least one of: the one or more processors of the robot via the cloud, the cloud, the one or more processors of the robot and the cloud, and the one or more processors of the robot. ### Claim 14 A method, comprising: generating, with a processor of a robot, a map of an environment of the robot and a path of the robot within the environment; transmitting, with the processor of the robot, the map of the environment and the path of the robot to an application of a computing device paired with a wireless controller of the robot; displaying, with the application of the computing device, the map and the path of the robot; receiving, with the application of the computing device, an input designating an instruction comprising at least one of: the path of the robot, an adjustment to the path of the robot, an adjustment to the map, an order of rooms within the environment the robot is to perform work within, areas within the environment the robot is to perform work within, areas within the environment the robot is to avoid, a type of work the robot is to perform within an area, a location within the environment the robot is to navigate to, a modification to a pre-existing label of an area within the environment, and a label for an area within the environment, wherein: the robot comprises: a chassis; drive wheels connected to the chassis; a plurality of sensors comprising at least one of a laser distance sensor and a camera; and a tangible, non-transitory, machine-readable medium; and the computing device comprises: a processor; a tangible, non-transitory, machine-readable medium; a microphone; a speaker; a screen; at least one user interface; and a camera. ### Claim 15 (depends on 14) The method of claim 14, wherein a first operation is executed with the processor of the robot and a second operation is executed with the application of the computing device or the first operation is executed with the application of the computing device and the second operation is executed with the processor of the robot. ### Claim 16 (depends on 14) The method of claim 14, further comprising: identifying, with the processor of the robot, an area of the environment by matching sensor data indicative of features within the area of the environment with features in the map of the environment. ### Claim 17 (depends on 14) The method of claim 14, further comprising: receiving, with the processor of the robot, at least one voice command comprising at least one of: an instruction to navigate to a particular area of the environment, an instruction to perform work within a particular area of the environment, and an instruction to perform a particular type of work within a particular area of the environment; and executing, with the robot, the at least one voice command. ### Claim 18 (depends on 14) The method of claim 14, further comprising: displaying, with the application of the computing device, icons representing different areas within the map of the environment. ### Claim 19 (depends on 14) The method of claim 14, further comprising: receiving, with the application of the computing device, an input designating a particular color assignment for areas within the environment, wherein each color is indicative of at least one of: a type of work to be performed by the robot, a particular setting of the robot, and a particular time to perform work by the robot. ### Claim 20 (depends on 14) The method of claim 14, further comprising: receiving, with the processor of the robot, an image of the environment from an imaging device separate from the robot; obtaining, with the processor of the robot, raw pixel intensity values of the image; extracting, with the processor of the robot, objects and features in the image by grouping pixels with similar raw pixel intensity values, and by identifying areas in the image with greatest change in raw pixel intensity values; determining, with the processor of the robot, an area within the map of the environment corresponding with the image by comparing the objects and features of the image with objects and features of the map; inferring, with the processor of the robot, one or more locations captured in the image based on the location of the area of the map corresponding with the image; and, instructing, with the processor of the robot, the robot to navigate to the one or more locations captured in the image.