# US 10,872,238 B2 · Augmented reality system to map and visualize sensor data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,872,238 B2 - Document ID: US10872238B2 - Status: Granted - Kind: B2 - Application number: 16/153,797 - Filed: 2018-10-07 - Published: 2020-04-09 - Granted: 2020-12-22 - Earliest claimed: 2018-10-07 - Assignees: General Electric Company - Applicants: General Electric Company - Inventors: Ratnadeep Paul; Radislav Alexandrovich Potyrailo - Web page: https://patentgrep.com/patent/US10872238B2 ## CPC - G06V 20/20: in augmented reality scenes - G02B 27/017: {Head mounted} - G06F 1/163: {Wearable computers, e.g. on a belt} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G08B 19/00: Alarms responsive to two or more different undesired or abnormal conditions, e.g. burglary and fire, abnormal temperature and abnormal rate of flow - G08B 21/12: responsive to undesired emission of substances, e.g. pollution alarms - G08B 25/00: Alarm systems in which the location of the alarm condition is signalled to a central station, e.g. fire or police telegraphic systems - G08B 25/14: Central alarm receiver or annunciator arrangements - G06T 2207/30108: Industrial image inspection - G08B 25/10: using wireless transmission systems {(G08B25/009 takes precedence)} - G08B 29/186: {Fuzzy logic; neural networks} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS ## Abstract A method and system to receive information from at least one multivariable sensor, each multivariable sensor being deployed in an environment, having internet connectivity to communicate with at least one other device over the internet, and selectively determining at least one attribute of multiple events in its environment; receive an indication of a location of the multivariable sensor; receive an indication of a location of an augmented reality device; determine an alarm based on the received information from the at least one multivariable sensor; determine a location for the alarm and a location of a solution associated with the alarm; and present, in a field of view display on the augmented reality device, a visualization of the determined alarm and at least one of the determined location for the alarm and the determined location for the solution associated with the alarm. ## Claims ### Claim 1 A system comprising: a memory storing processor-executable instructions; and a processor to execute the processor-executable instructions to cause the system to: receive information from at least one multivariable sensor, each multivariable sensor being deployed in an environment, having internet connectivity to communicate with at least one other device over the internet, and selectively determining at least two attributes of multiple events in its environment, the selectively determined at least two attributes being included in the received information, each multivariable sensor responsive to the multiple events in a first dimension and interferences to the multiple events in a second dimension and each multivariable sensor including a gas-selective multidimensional detector having: a sensing material with different responses to different gases; a multivariable transducer to provide independent outputs and to recognize the different responses to the different gases; and data analytics to provide multi-analyte quantification of the different responses to the different gases and rejection of interferences to the different gases; receive an indication of a location of the at least one multivariable sensor; receive an indication of a location of an augmented reality device; determine an alarm based on the received information from the at least one multivariable sensor; determine a location for the alarm based on the location of a specific one of the at least one multivariable sensor and a location of a solution to the determined alarm; and present, contemporaneously in a current field of view display on the augmented reality device, a visualization of the determined alarm and at least one of the determined location for the alarm and the determined location for the solution to the determined alarm. ### Claim 2 (depends on 1) The system of claim 1, wherein the at least one multivariable sensor includes a liquid-selective multidimensional detector comprising: a multivariable transducer to provide independent outputs about electromagnetic properties of the liquid; and data analytics to provide multi-analyte quantification of the different components of interest in the liquid and rejection of response to interferences in the liquid. ### Claim 3 (depends on 1) The system of claim 1, wherein the location of the multivariable sensor and the location of the augmented reality device are referenced with respect to a same coordinate system. ### Claim 4 (depends on 1) The system of claim 1, further comprising presenting, in response to at least the determined alarm and location for the alarm, an actionable solution in the field of view display on the augmented reality device. ### Claim 5 (depends on 1) The system of claim 1, wherein the visualization of the determined alarm and the at least one of the determined location for the alarm and the determined location for the solution to the determined alarm includes at least one of: a directional arrow, a textual description of the alarm, a color-coded indication of an alarm level corresponding to the alarm, a graphical-coded indication of an alarm level corresponding to the alarm, a textual description of a direction to the at least one of the determined location for the alarm and the determined location for the solution to the determined alarm from a current location of the augmented reality device, a textual description of a route to the at least one of the determined location for the alarm and the determined location for the solution to the determined alarm from a current location of the augmented reality device, and a graphical user interface representation of at least one of the determined alarm and the at least one of the determined location for the alarm and the determined location for the solution to the determined alarm. ### Claim 6 (depends on 1) The system of claim 1, wherein the augmented reality device is one of a headset with a display, visor having a display, an eyewear device having a display, a portable computing device with a display, a wearable device with a display, a device in or on a person, a mobile phone device with a display device, on a manned vehicle, and on an unmanned vehicle. ### Claim 7 (depends on 1) The system of claim 1, wherein the field of view, the visualization of the determined alarm, and the at least one of the determined location for the alarm and the determined location for the solution to the alarm determined displayed on the augmented reality device are current contemporaneous versions thereof. ### Claim 8 (depends on 1) The system of claim 1, wherein the augmented reality device is collocated with one of a person and an unmanned mobile device. ### Claim 9 A method comprising: receiving information from at least one multivariable sensor, each multivariable sensor being deployed in an environment, having internet connectivity to communicate with at least one other device over the internet, and selectively determining at least two attributes of multiple events in its environment, the selectively determined at least two attributes being included in the received information, each multivariable sensor responsive to the multiple events in a first dimension and interferences to the multiple events in a second dimension and each multivariable sensor including a gas-selective multidimensional detector having: a sensing material with different responses to different gases; a multivariable transducer to provide independent outputs and to recognize the different responses to the different gases; and data analytics to provide multi-analyte quantification of the different responses to the different gases and rejection of interferences to the different gases; receiving, by a processor, an indication of a location of the at least one multivariable sensor; receiving, by the processor, an indication of a location of an augmented reality device; determining, by the processor, an alarm based on the received information from the at least one multivariable sensor; determining, by the processor, a location for the alarm based on the location of a specific one of the at least one multivariable sensor and a location of a solution to the determined alarm; and presenting, contemporaneously in a current field of view display on the augmented reality device, a visualization of the determined alarm and at least one of the determined location for the alarm and the determined location for the solution to the determined alarm. ### Claim 10 (depends on 9) The method of claim 9, wherein the at least one multivariable sensor includes a liquid-selective multidimensional detector comprising: a multivariable transducer to provide independent outputs about electromagnetic properties of the liquid; and data analytics to provide multi-analyte quantification of the different components of interest in the liquid and rejection of response to interferences in the liquid. ### Claim 11 (depends on 9) The method of claim 9, wherein the location of the multivariable sensor and the location of the augmented reality device are referenced with respect to a same coordinate system. ### Claim 12 (depends on 9) The method of claim 9, further comprising presenting, in response to at least the determined alarm and location for the alarm, an actionable solution in the field of view display on the augmented reality device. ### Claim 13 (depends on 9) The method of claim 9, wherein the visualization of the determined alarm and the at least one of the determined location for the alarm and the determined location for the solution to the determined alarm includes at least one of: a directional arrow, a textual description of the alarm, a color-coded indication of an alarm level corresponding to the alarm, a graphical-coded indication of an alarm level corresponding to the alarm, a textual description of a direction to the at least one of the determined location for the alarm and the determined location for the solution to the determined alarm from a current location of the augmented reality device, a textual description of a route to the at least one of the determined location for the alarm and the determined location for the solution to the determined alarm from a current location of the augmented reality device, and a graphical user interface representation of at least one of the determined alarm and the at least one of the determined location for the alarm and the determined location for the solution to the determined alarm. ### Claim 14 (depends on 9) The method of claim 9, wherein the augmented reality device is one of a headset with a display, visor having a display, an eyewear device having a display, a portable computing device with a display, a wearable device with a display, a device in or on a person, a mobile phone device with a display device, on a manned vehicle, and on an unmanned vehicle. ### Claim 15 (depends on 9) The method of claim 9, wherein the augmented reality device is in air, above ground, underground, above water, underwater, in space, in an industrial setting, a residential setting, environmental setting, military setting, indoor setting, outdoor setting. ### Claim 16 (depends on 9) The method of claim 9, wherein the field of view, the visualization of the determined alarm, and the at least one of the determined location for the alarm and the determined location for the solution to the determined the alarm displayed on the augmented reality device are current contemporaneous versions thereof.