# US 2015/0146007 A1 · Maintenance assistant system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2015/0146007 A1 - Document ID: US20150146007A1 - Status: Published application - Kind: A1 - Application number: 14/090,095 - Filed: 2013-11-26 - Published: 2015-05-28 - Earliest claimed: 2013-11-26 - Assignees: Honeywell International Inc. - Applicants: Honeywell International Inc. - Inventors: Matej Dusik; Dinkar Mylaraswamy; Jiri Vasek; Jindrich Finda; Michal Kosik - Web page: https://patentgrep.com/patent/US20150146007A1 ## CPC - G06V 20/20: in augmented reality scenes - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G02B 2027/0141: {characterised by the informative content of the display} - G06V 2201/06: Recognition of objects for industrial automation - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS ## Abstract A maintenance assistance system and method of operating are provided. The maintenance assistance system may include, but is not limited to, a camera, a heads-up display, a memory configured to maintenance task data, and a processor communicatively coupled to the camera, the heads-up display and the memory, the processor configured to determine a component to be serviced, determine a location of the component based upon data from the camera and the maintenance task data stored in the memory, generate graphical data based upon a maintenance step associated with the component, and output the generated graphical data to the heads-up display ## Claims ### Claim 1 A maintenance assistance system, comprising: a camera; a heads-up display; a memory configured to store maintenance task data; and a processor communicatively coupled to the camera, the heads-up display and the memory, the processor configured to: determine a component to be serviced; determine a location of the component based upon data from the camera and the maintenance task data stored in the memory; generate graphical data based upon a maintenance step associated with the component; and output the generated graphical data to the heads-up display. ### Claim 2 (depends on 1) The maintenance assistance system of claim 1, wherein the graphical data includes a location indicator for locating the component. ### Claim 3 (depends on 1) The maintenance assistance system of claim 1, wherein the graphical data includes instructions for servicing the component. ### Claim 4 (depends on 1) The maintenance assistance system of claim 1, wherein the graphical data further includes a model of component. ### Claim 5 (depends on 1) The maintenance assistance system of claim 1, wherein the processor is further configured to determine, based upon data from the camera and the maintenance task data, whether a maintenance step was correctly completed. ### Claim 6 (depends on 5) The maintenance assistance system of claim 5, wherein the processor is further configured to store a video of the maintenance step once the processor determines that the maintenance step was correctly completed. ### Claim 7 (depends on 5) The maintenance assistance system of claim 5, wherein the maintenance task data includes a plurality of images of the component, and the processor is further configured to determine whether the maintenance step was correctly completed by comparing an image from the camera to at least one of the plurality of images of the component stored in the memory. ### Claim 8 (depends on 5) The maintenance assistance system of claim 5, wherein the maintenance task data includes a three-dimensional model of the component, and the processor is further configured to determine whether the maintenance step was correctly completed by comparing an image from the camera to the three-dimensional model. ### Claim 9 A method for operating a maintenance assistance system comprising a camera, a heads-up display, a memory and a processor communicatively coupled to the camera, the heads-up display and the memory, the method comprising: receiving, by the processor, an indication of a component; determining, by the processor, a location of the component based upon data from the camera and maintenance task data stored in the memory; generating, by the processor, graphical data based upon a maintenance step associated with the component; and outputting, by the processor, the generated graphical data to the heads-up display. ### Claim 10 (depends on 9) The method of claim 9, wherein the graphical data includes a location indicator for locating the component. ### Claim 11 (depends on 9) The method of claim 9, wherein the graphical data includes instructions for servicing the component. ### Claim 12 (depends on 9) The method of claim 9, wherein the graphical data further comprises a model of the component. ### Claim 13 (depends on 9) The method of claim 9, further comprising determining, by the processor, whether a maintenance step was correctly completed based upon data from the camera and the maintenance task data. ### Claim 14 (depends on 13) The method of claim 13, wherein the maintenance task data includes a plurality of images of the component, and the method further comprises determining, by the processor, whether the maintenance step was correctly completed by comparing an image from the camera to at least one of the plurality of images of the component stored in the memory. ### Claim 15 (depends on 13) The method of claim 13, wherein the maintenance task data includes a three-dimensional model of the component, and the method further comprises determining, by the processor, whether the maintenance step was correctly completed by comparing an image from the camera to the three-dimensional model. ### Claim 16 A computer-readable medium storing instructions, which when executed by a processor, operate a maintenance assistance system comprising the processor, a camera, a heads-up display, and a memory, the instructions causing the processor to: determine a component to be serviced; determine a location of the component based upon data from the camera and the maintenance task data stored in the memory; generate graphical data based upon a maintenance step associated with the component; and output the generated graphical data to the heads-up display. ### Claim 17 (depends on 16) The computer-readable medium of claim 16, wherein the graphical data includes a location indicator for locating the component. ### Claim 18 (depends on 16) The computer-readable medium of claim 16, wherein the instructions further cause the processor to determine, based upon data from the camera and the maintenance task data, whether a maintenance step was correctly completed. ### Claim 19 (depends on 18) The computer-readable medium of claim 18, wherein the maintenance task data includes a plurality of images of the component, and the instructions further cause the processor to determine whether the maintenance step was correctly completed by comparing an image from the camera to at least one of the plurality of images of the component stored in the memory. ### Claim 20 (depends on 18) The computer-readable medium of claim 18, wherein the maintenance task data includes a three-dimensional model of the component, and the instructions further cause the processor to determine whether the maintenance step was correctly completed by comparing an image from the camera to the three-dimensional model.