# US 2019/0377587 A1 · Sharing and exchanging information onboard displays or instruments and an offboard device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0377587 A1 - Document ID: US20190377587A1 - Status: Published application - Kind: A1 - Application number: 16/005,359 - Filed: 2018-06-11 - Published: 2019-12-12 - Earliest claimed: 2018-06-11 - Assignees: Honeywell International Inc. - Applicants: Honeywell International Inc. - Inventors: Martin Dostal; Aaron J. Gannon - Web page: https://patentgrep.com/patent/US20190377587A1 ## CPC - G06F 9/451: Execution arrangements for user interfaces - G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance - G06F 9/54: Interprogram communication - G06F 16/583: using metadata automatically derived from the content - G06F 16/5838: {using colour} - G06V 20/20: in augmented reality scenes - G06V 30/418: Document matching, e.g. of document images - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method for interfacing between a first electronic system and a second electronic system is provided. The method comprises: capturing data on a graphical user interface state of the first electronic system using one or more sensors of the second electronic system, wherein at least one type of the data captured with the one or more sensors is image data captured using an optical sensor; recognizing at least one component of the graphical user interface state of the first electronic system, wherein the at least one component of the graphical user interface state further comprises structure and content of the graphical user interface; matching the at least one recognized component with one or more entries in a database on the second electronic system; and performing an action on the second electronic system associated with the matched component. ## Claims ### Claim 1 A method for interfacing between a first electronic system and a second electronic system, the method comprising: capturing data on a graphical user interface state of the first electronic system using one or more sensors of the second electronic system, wherein at least one type of the data captured with the one or more sensors is image data captured using an optical sensor; recognizing at least one component of the graphical user interface state of the first electronic system, wherein the at least one component of the graphical user interface state further comprises structure and content of the graphical user interface; matching the at least one recognized component with one or more entries in a database on the second electronic system; and performing an action on the second electronic system associated with the matched component. ### Claim 2 (depends on 1) The method of claim 1, wherein capturing data on graphical user interface state of the first electronic system further comprises using at least one of a sound sensor and an accelerometer. ### Claim 3 (depends on 1) The method of claim 1, wherein the image data captured using an optical sensor is a flow of images. ### Claim 4 (depends on 1) The method of claim 1, wherein the optical sensor is at least one of a camera, a video camera and an optical character recognition (OCR) sensor. ### Claim 5 (depends on 1) The method of claim 1, wherein the structure and content of the graphical user interface further comprises at least one of shape, color, font, position, and text portion of the component on the graphical user interface in the image data captured using the optical sensor. ### Claim 6 (depends on 5) The method of claim 5, wherein matching the at least one recognized component with one or more entries in a database on the second electronic system further comprises matching the at least one recognized component with a set of shapes, colors, positions, fonts and text portions having a similar configuration as the at least one recognized component. ### Claim 7 (depends on 1) The method of claim 1, wherein the first electronic system comprises an on-board device of an aircraft, and wherein the second electronic system comprises an off-board device. ### Claim 8 (depends on 7) The method of claim 7, wherein the off-board device is a portable computing device. ### Claim 9 (depends on 1) The method of claim 1, wherein when the recognized component matches two or more entries in the database, performing an action on the second electronic system associated with the matched component further comprises: resolving ambiguity between the two or more entries in the database to determine a disambiguated matched component by at least one of: prompting for user input to determine one of the two or more entries in the database as the disambiguated matched component; and automatically determining one of the two or more entries in the database as the disambiguated matched component based on additional data on the second electronic system; and performing an action on the second electronic system associated with the matched component. ### Claim 10 (depends on 9) The method of claim 9, wherein prompting for user input further comprises at least one of: prompting for user input to provide additional data such that the processor is able to determine one of the two or more entries in the database based on the received additional data; and prompting for user input to choose one of the two or more matched entries as the matched component. ### Claim 11 A portable computing device for interfacing with an electronic system having a graphical user interface, the portable computing device comprising: one or more sensors configured to capture data on the graphical user interface of the electronic system, wherein at least one of the one or more sensors is an optical sensor configured to capture image data; at least one processor configured to recognize at least one component of the graphical user interface state of the electronic system, wherein the at least one component of the user interface state further comprises structure and content of the graphical user interface; a database coupled to the at least one processor; wherein the processor is further configured to: match the at least one recognized component with one or more entries in the database; and perform an action associated with the matched component. ### Claim 12 (depends on 11) The portable computing device of claim 11, wherein at least one of the one or more sensors is at least one of a sound sensor and an accelerometer. ### Claim 13 (depends on 11) The portable computing device of claim 11, wherein the image data captured using an optical sensor is a flow of images. ### Claim 14 (depends on 11) The portable computing device of claim 11, wherein the structure and content of the graphical user interface further comprises at least one of shape, color and position of the component on the graphical user interface in the image data captured using the optical sensor. ### Claim 15 (depends on 14) The portable computing device of claim 14, wherein the optical sensor is at least one of a camera, a video camera and an optical character recognition (OCR) sensor. ### Claim 16 (depends on 11) The portable computing device of claim 11, wherein when the recognized component matches two or more entries in the database, prior to performing the action, the at least one processor is further configured to resolve ambiguity based on at least one of user input entered upon prompt and additional data received by the processor. ### Claim 17 (depends on 11) The portable computing device of claim 11, wherein the electronic system with which the portable computing device is interfacing comprises an on-board device of an aircraft, and wherein the portable computing device is an off-board device. ### Claim 18 A non-transitory computer readable medium storing a program having instructions stored thereon, executable by a processor, to perform a method for interfacing between first and second electronic systems, the method comprising: capturing data on a graphical user interface state of the first electronic system using one or more sensors of the second electronic system, wherein at least one type of the data captured with the one or more sensors is image data captured using an optical sensor; recognizing at least one component of the graphical user interface state of the first electronic system, wherein the at least one component of the user interface state further comprises structure and content of the graphical user interface; matching the at least one recognized component with one or more entries in a database on the second electronic system; and performing an action on the second electronic system associated with the matched component. ### Claim 19 (depends on 18) The non-transitory computer readable medium of claim 18, wherein when the recognized component matches two or more entries in the database, performing an action on the second electronic system associated with the matched component further comprises: resolving ambiguity between the two or more entries in the database by at least one of: prompting for user input to determine one of the two or more entries in the database as the matched component; and automatically determining one of the two or more entries in the database as the matched component based on additional data on the second electronic system; and performing an action on the second electronic system associated with the matched component. ### Claim 20 (depends on 18) The non-transitory computer readable medium of claim 18, wherein the structure and content of the graphical user interface further comprises at least one of shape, color, font, position, and text portion of the component on the graphical user interface in the image data captured using the optical sensor; and matching the at least one recognized component with one or more entries in a database on the second electronic system further comprises matching the at least one recognized component with a set of shapes, colors, positions, fonts and text portions having a similar configuration as the at least one recognized component.