# US 9,744,671 B2 · Information technology asset type identification using a mobile vision-enabled robot > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,744,671 B2 - Document ID: US9744671B2 - Status: Granted - Kind: B2 - Application number: 15/074,168 - Filed: 2016-03-18 - Published: 2016-07-14 - Granted: 2017-08-29 - Earliest claimed: 2014-05-20 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Jonathan H. Connell, II; Jeffrey O. Kephart; Jonathan Lenchner - Web page: https://patentgrep.com/patent/US9744671B2 ## CPC - B25J 9/1697: {Vision controlled systems} - B25J 9/16: Program controls (total factory control, i.e. centrally controlling a plurality of machines, G05B19/418) - G06F 16/285: {Clustering or classification} - G06F 16/51: Indexing; Data structures therefor; Storage structures - G06F 16/86: {Mapping to a database} - G06F 18/24: Classification techniques - G06V 10/764: using classification, e.g. of video objects - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 20/20: in augmented reality scenes - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G05B 2219/39369: Host and robot controller and vision processing - G05B 2219/40005: Vision, analyse image at one station during manipulation at next station - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract Mechanisms are provided for classifying an obstacle as an asset type. The mechanisms receive a digital image of an obstacle from an image capture device of an automated robot. The mechanisms perform a classification operation on the digital image of the obstacle to identify a proposed asset type classification for the obstacle. The mechanisms determine a final asset type for the obstacle based on the proposed asset type classification for the obstacle. The mechanisms update a map data structure for a physical premises in which the obstacle is present based on the final asset type. ## Claims ### Claim 1 A method, in a data processing system comprising a processor and a memory, for classifying an obstacle as an asset type, comprising: receiving, by the data processing system, a digital image of an obstacle from an image capture device of an automated robot; performing, by the data processing system, a classification operation on the digital image of the obstacle to identify a proposed asset type classification for the obstacle; determining, by the data processing system, a final asset type for the obstacle based on the proposed asset type classification for the obstacle; and updating, by the data processing system, a map data structure for a physical premises in which the obstacle is present based on the final asset type, wherein performing the classification operation further comprises: gathering additional sensor information from one or more other sensors provided on either the robot or in the physical premises; and performing the classification operation based on a classification of characteristics of the obstacle obtained from analysis of the digital image and analysis of the additional sensor information, wherein the additional sensor information comprises information indicative of environmental conditions within a vicinity of the obstacle, that together with the digital image of the obstacle, are indicative of an asset type classification of the obstacle. ### Claim 2 (depends on 1) The method of claim 1, wherein determining a final asset type for the obstacle based on the proposed asset type classification for the obstacle comprises: calculating, by the data processing system, a confidence value indicating a confidence in the identification of the proposed asset type classification for the obstacle; and in response to the confidence value being less than a predetermined threshold value: presenting, by the data processing system, the proposed asset type classification and the digital image to a human user via a user interface output on a computing device associated with the human user, wherein the user interface identifies the proposed asset type classification, the digital image, and provides a mechanism for user feedback input indicating whether or not the proposed asset type classification is correct or incorrect; and determining, by the data processing system, the final asset type for the obstacle based on input from the human user via the user interface. ### Claim 3 (depends on 1) The method of claim 1, wherein the method is implemented automatically without human intervention. ### Claim 4 (depends on 1) The method of claim 1, wherein the additional sensor information comprises at least one of temperature information in proximity to the obstacle or humidity information in proximity to the obstacle, and at least one of infrared profile information of the obstacle, proximity information indicating a proximity of the obstacle to other known types of assets, or configuration information for other known types of nearby assets relative to the obstacle. ### Claim 5 (depends on 1) The method of claim 1, wherein performing the classification operation on the digital image of the obstacle comprises utilizing a plurality of classification algorithms that each classify the obstacle with regard to one or more different characteristics of the obstacle as identified in at least one of the digital image or the additional sensor information. ### Claim 6 (depends on 1) The method of claim 1, further comprising: outputting, by the data processing system, a user interface representing the map of the physical premises in which assets in the physical premises are distinguished by asset type. ### Claim 7 (depends on 6) The method of claim 6, wherein outputting the user interface representing the map of the physical premises comprises outputting the user interface with user interface elements for filtering assets in the physical premises to accentuate or remove assets from the map based on associated asset type. ### Claim 8 (depends on 1) The method of claim 1, wherein the physical premises is a data center and the obstacle is a physical asset of the data center. ### Claim 9 (depends on 1) The method of claim 1, wherein performing the classification operation further comprises: performing an initial asset classification operation based on analysis of the digital image to identify a subset of asset types in which the obstacle is classifiable; and performing a refined asset classification operation based on the additional sensor information, at least by comparing the additional sensor information to known patterns of additional sensor information for asset type classifications to identify a particular asset type classification within the subset of asset types. ### Claim 10 (depends on 1) The method of claim 1, further comprising: training the data processing system to perform the classification operation by performing iterative machine learning comprising: analyzing a digital image and additional sensor information associated with an obstacle present in a physical premises used for training to generate a proposed asset type classification of the obstacle; determining an actual asset type classification of the obstacle based on an asset identifier tag affixed to the obstacle; comparing the proposed asset type classification of the obstacle with the actual asset type classification of the obstacle; and modifying at least one parameter of the classification operation based on the results of comparing the proposed asset type classification of the obstacle with the actual asset type classification of the obstacle indicating a difference between the proposed asset type classification and the actual asset type classification. ### Claim 11 A computer program product comprising a non-transitory computer readable medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to: receive a digital image of an obstacle from an image capture device of an automated robot; perform a classification operation on the digital image of the obstacle to identify a proposed asset type classification for the obstacle; determine a final asset type for the obstacle based on the proposed asset type classification for the obstacle; and update a map data structure for a physical premises in which the obstacle is present based on the final asset type, wherein the computer readable program further causes the computing device to perform the classification operation at least by: gathering additional sensor information from one or more other sensors provided on either the robot or in the physical premises; and performing the classification operation based on a classification of characteristics of the obstacle obtained from analysis of the digital image and analysis of the additional sensor information, wherein the additional sensor information comprises information indicative of environmental conditions within a vicinity of the obstacle, that together with the digital image of the obstacle, are indicative of an asset type classification of the obstacle. ### Claim 12 (depends on 11) The computer program product of claim 11, wherein the computer readable program further causes the computing device to determine a final asset type for the obstacle based on the proposed asset type classification for the obstacle at least by: calculating a confidence value indicating a confidence in the identification of the proposed asset type classification for the obstacle; and in response to the confidence value being less than a predetermined threshold value: presenting the proposed asset type classification and the digital image to a human user via a user interface output on a computing device associated with the human user, wherein the user interface identifies the proposed asset type classification, the digital image, and provides a mechanism for user feedback input indicating whether or not the proposed asset type classification is correct or incorrect; and determining the final asset type for the obstacle based on input from the human user via the user interface. ### Claim 13 (depends on 11) The computer program product of claim 11, wherein the computer readable program further causes the computing device to execute the receive, perform, determine, and update operations automatically without human intervention. ### Claim 14 (depends on 11) The computer program product of claim 11, wherein the additional sensor information comprises at least one of temperature information in proximity to the obstacle or humidity information in proximity to the obstacle, and at least one of infrared profile information of the obstacle, proximity information indicating a proximity of the obstacle to other known types of assets, or configuration information for other known types of nearby assets relative to the obstacle. ### Claim 15 (depends on 11) The computer program product of claim 11, wherein the computer readable program further causes the computing device to perform the classification operation on the digital image of the obstacle at least by utilizing a plurality of classification algorithms that each classify the obstacle with regard to one or more different characteristics of the obstacle as identified in at least one of the digital image or the additional sensor information. ### Claim 16 (depends on 11) The computer program product of claim 11, wherein the computer readable program further causes the computing device to: output a user interface representing the map of the physical premises in which assets in the physical premises are distinguished by asset type. ### Claim 17 (depends on 16) The computer program product of claim 16, wherein the computer readable program further causes the computing device to output the user interface representing the map of the physical premises at least by outputting the user interface with user interface elements for filtering assets in the physical premises to accentuate or remove assets from the map based on associated asset type. ### Claim 18 (depends on 11) The computer program product of claim 11, wherein the computer readable program further causes the computing device to perform the classification operation at least by: performing an initial asset classification operation based on analysis of the digital image to identify a subset of asset types in which the obstacle is classifiable; and performing a refined asset classification operation based on the additional sensor information, at least by comparing the additional sensor information to known patterns of additional sensor information for asset type classifications to identify a particular asset type classification within the subset of asset types. ### Claim 19 (depends on 11) The computer program product of claim 11, wherein the computer readable program further causes the computing device to: train the data processing system to perform the classification operation by performing iterative machine learning comprising: analyzing a digital image and additional sensor information associated with an obstacle present in a physical premises used for training to generate a proposed asset type classification of the obstacle; determining an actual asset type classification of the obstacle based on an asset identifier tag affixed to the obstacle; comparing the proposed asset type classification of the obstacle with the actual asset type classification of the obstacle; and modifying at least one parameter of the classification operation based on the results of comparing the proposed asset type classification of the obstacle with the actual asset type classification of the obstacle indicating a difference between the proposed asset type classification and the actual asset type classification. ### Claim 20 An apparatus comprising: a processor; and a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to: receive a digital image of an obstacle from an image capture device of an automated robot; perform a classification operation on the digital image of the obstacle to identify a proposed asset type classification for the obstacle; determine a final asset type for the obstacle based on the proposed asset type classification for the obstacle; and update a map data structure for a physical premises in which the obstacle is present based on the final asset type, wherein the instructions further cause the processor to perform the classification operation at least by: gathering additional sensor information from one or more other sensors provided on either the robot or in the physical premises; and performing the classification operation based on a classification of characteristics of the obstacle obtained from analysis of the digital image and analysis of the additional sensor information, wherein the additional sensor information comprises information indicative of environmental conditions within a vicinity of the obstacle, that together with the digital image of the obstacle, are indicative of an asset type classification of the obstacle.