# US 2024/0193064 A1 · Method for monitoring and maintaining health state of electrical device, and electronic device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0193064 A1
- Document ID: US20240193064A1
- Status: Published application
- Kind: A1
- Application number: 18/525,065
- Filed: 2023-11-30
- Published: 2024-06-13
- Earliest claimed: 2022-12-09
- Assignees: ABB Schweiz AG
- Applicants: ABB Schweiz AG
- Inventors: Wei Zheng; Zhui Wang; Longbin Huang; Zhen Chen; Changhui You
- Web page: https://patentgrep.com/patent/US20240193064A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 11/328: {Computer systems status display (G06F11/327 takes precedence)}
- G02B 5/32: Holograms used as optical elements
- G05B 19/4184: {characterised by fault tolerance, reliability of production system}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06Q 10/20: Administration of product repair or maintenance
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G05B 2219/31467: Display of operating conditions of machines, workcells, selected programs
- G05B 2219/32014: Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods and devices for monitoring and maintaining a health state of an electrical device. The methods can include obtaining a first operation state of the electrical device; rendering a holographic image of the electrical device and a first health state of the electrical device indicated by the first operation state by using a mixed reality device; detecting an action of a first user by using the mixed reality device; in response to that an operation performed by the first user on the electrical device has been detected, performing the operation on an image model of the electrical device in the holographic image to obtain a second operation state after the electrical device is operated; and updating the first health state to a second health state indicated by the second operation state.

## Claims

### Claim 1

A method for monitoring and maintaining a health state of an electrical device, comprising:
obtaining a first operation state of the electrical device from a sensing apparatus in the electrical device;
rendering a holographic image of the electrical device and a first health state of the electrical device indicated by the first operation state by using a first display apparatus of a mixed reality device;
detecting an action of a first user by using a first image capturing apparatus of the mixed reality device;
in response to that an operation performed by the first user on the electrical device has been detected, performing the operation on an image model of the electrical device in the holographic image to obtain a second operation state after the electrical device is operated; and
updating the first health state to a second health state indicated by the second operation state.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the performing the operation on the holographic image comprises:
mapping the action of the first user into the holographic image to obtain a virtual trajectory of the action in the holographic image;
determining an operation matched with the virtual trajectory;
simulating a digital model of the electrical device under the operation to obtain simulated data; and
adjusting the holographic image into the second operation state indicated by the simulated data.

### Claim 3 (depends on 2)

The method according to claim 2, further comprising:
in response to a determination that the operation comprises an operation performed on an actuating mechanism of the electrical device, generating an operating instruction; and
sending the operating instruction to the actuating mechanism, so that the actuating mechanism executes the operating instruction.

### Claim 4 (depends on 1)

The method according to claim 1, further comprising:
in response to a determination that the electrical device has an abnormality, generating prompt information which indicates the abnormality;
rendering the prompt information in the holographic image; and
in response to that a selection operation performed by the first user on the prompt information has been detected, rendering, in the holographic image, the abnormal electrical device and operation data associated with the abnormality.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the rendering the abnormal electrical device and operation data comprises:
determining an abnormal component, causing the abnormality, in the electrical device;
rendering a maintenance plan for the abnormal component in the holographic image; and
rendering the abnormal component in a manner that is different from a manner for rendering a normal component.

### Claim 6 (depends on 5)

The method according to claim 5, further comprising:
in response to that a maintenance operation performed by the first user on the abnormal component has been detected, determining a component action trajectory of the abnormal component under the maintenance operation; and
in response to a determination that a deviation between the component action trajectory and a preset trajectory indicated by the maintenance plan is greater than a predetermined threshold, sending information which indicates a wrong operation.

### Claim 7 (depends on 6)

The method according to claim 6, further comprising:
in response to a determination that the deviation between the component action trajectory and the preset trajectory indicated by the maintenance plan is less than the predetermined threshold, sending information which indicates a correct operation.

### Claim 8 (depends on 1)

The method according to claim 1, further comprising:
sending the holographic image to a second display apparatus;
receiving an operation indication from the second display apparatus, wherein the operation indicator is obtained by detecting an action of a second user of the second display apparatus using a second image capturing apparatus; and
rendering the operation indication in the holographic image by using the first display apparatus.

### Claim 9 (depends on 8)

The method according to claim 8, further comprising:
obtaining, by using the first image capturing apparatus, a reality image of a space where the first user is located;
establishing a three-dimensional space model of the space on a basis of the reality image, wherein the three-dimensional space model comprises a three-dimensional device model of the electrical device in the space, and a position and direction of the three-dimensional device model correspond to an actual position and direction of the electrical device in the space;
obtaining operation data of the electrical device; and
rendering a holographic image by the three-dimensional space model and the operation data.

### Claim 10 (depends on 9)

The method according to claim 9, further comprising:
obtaining first posture information of the first user by using the first image capturing apparatus;
receiving second posture information of the second user from the second image capturing apparatus, and determining a visual field of the first user on the basis of the first posture information;
establishing a user model of the second user in the three-dimensional space model on the basis of the second posture information; and
rendering the holographic image of the three-dimensional space model in the visual field by using the first display apparatus, wherein the holographic image comprises a holographic image of the second user.

### Claim 11 (depends on 1)

An electronic device, comprising:
at least one processor; and
at least one memory, comprising instructions stored on the memory, wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform the method according to claim 1.

### Claim 12 (depends on 1)

A computer-readable storage medium, storing instructions, wherein the instructions, when executed by at least one processor, cause the at least one processor to perform the method according to claim 1.

### Claim 13 (depends on 11)

The electronic device according to claim 11, wherein the performing the operation on the holographic image of the method comprises:
mapping the action of the first user into the holographic image to obtain a virtual trajectory of the action in the holographic image;
determining an operation matched with the virtual trajectory;
simulating a digital model of the electrical device under the operation to obtain simulated data; and
adjusting the holographic image into the second operation state indicated by the simulated data.

### Claim 14 (depends on 13)

The electronic device according to claim 13, wherein the method further comprises:
in response to a determination that the operation comprises an operation performed on an actuating mechanism of the electrical device, generating an operating instruction; and
sending the operating instruction to the actuating mechanism, so that the actuating mechanism executes the operating instruction.

### Claim 15 (depends on 11)

The electronic device according to claim 11, wherein the method further comprises:
in response to a determination that the electrical device has an abnormality, generating prompt information which indicates the abnormality;
rendering the prompt information in the holographic image; and
in response to that a selection operation performed by the first user on the prompt information has been detected, rendering, in the holographic image, the abnormal electrical device and operation data associated with the abnormality.

### Claim 16 (depends on 11)

The electronic device according to claim 11, wherein the method further comprises:
sending the holographic image to a second display apparatus;
receiving an operation indication from the second display apparatus, wherein the operation indicator is obtained by detecting an action of a second user of the second display apparatus using a second image capturing apparatus; and
rendering the operation indication in the holographic image by using the first display apparatus.

### Claim 17 (depends on 12)

The computer-readable storage medium according to claim 12, wherein the performing the operation on the holographic image of the method comprises:
mapping the action of the first user into the holographic image to obtain a virtual trajectory of the action in the holographic image;
determining an operation matched with the virtual trajectory;
simulating a digital model of the electrical device under the operation to obtain simulated data; and
adjusting the holographic image into the second operation state indicated by the simulated data.

### Claim 18 (depends on 17)

The computer-readable storage medium according to claim 17, wherein the method further comprises:
in response to a determination that the operation comprises an operation performed on an actuating mechanism of the electrical device, generating an operating instruction; and
sending the operating instruction to the actuating mechanism, so that the actuating mechanism executes the operating instruction.

### Claim 19 (depends on 12)

The computer-readable storage medium according to claim 12, wherein the method further comprises:
in response to a determination that the electrical device has an abnormality, generating prompt information which indicates the abnormality;
rendering the prompt information in the holographic image; and
in response to that a selection operation performed by the first user on the prompt information has been detected, rendering, in the holographic image, the abnormal electrical device and operation data associated with the abnormality.

### Claim 20 (depends on 12)

The computer-readable storage medium according to claim 12, wherein the method further comprises:
sending the holographic image to a second display apparatus;
receiving an operation indication from the second display apparatus, wherein the operation indicator is obtained by detecting an action of a second user of the second display apparatus using a second image capturing apparatus; and
rendering the operation indication in the holographic image by using the first display apparatus.
