# US 12,002,578 B2 · Systems and methods for augmented reality-enhanced field services support

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,578 B2
- Document ID: US12002578B2
- Status: Granted
- Kind: B2
- Application number: 17/292,467
- Filed: 2019-12-02
- Published: 2022-01-20
- Granted: 2024-06-04
- Earliest claimed: 2018-12-10
- Assignees: Koninklijke Philips N.V.
- Applicants: Koninklijke Philips N.V.
- Inventors: Rithesh Sreenivasan; Oladimeji Feyisetan Farri; Sheikh Sadid Al Hasan; Tilak Raj Arora; Vivek Varma Datla
- Web page: https://patentgrep.com/patent/US12002578B2

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G16H 40/63: for local operation
- G02B 27/017: {Head mounted}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G16H 30/40: for processing medical images, e.g. editing
- G16H 40/40: for the management of medical equipment or devices, e.g. scheduling maintenance or upgrades
- G02B 2027/014: {comprising information/image processing systems}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An augmented reality (AR) content generation method includes: acquiring, with a camera of an AR device, one or more images of a component of a medical imaging or medical therapy device; receiving, from a microphone of the AR device, a triggering audio segment; generating one or more query data structures from both the one or more images and the triggering audio segment; retrieving AR instructional content related to the medical imaging or medical therapy device matching the generated one or more query data structures from a database; and outputting the AR instructional content one or more of (i) displayed superimposed on video displayed by the AR device and/or (ii) displayed on a head mounted display of the AR device and/or (iii) as audio content via a loudspeaker of the AR device.

## Claims

### Claim 1

A non-transitory computer-readable medium storing instructions readable and executable by at least one electronic processor to perform an augmented reality (AR) content generation method, the method comprising:
acquiring, with a camera of an AR device, one or more images of a component of a medical imaging or medical therapy device;
processing the acquired one or more images with an image processor to generate one or more processed images;
receiving, from a microphone of the AR device, a triggering audio segment;
processing the received triggering audio segment with a speech-to-text process or a transcription process to generate a processed triggering audio segment;
generating one or more query vectors from both the one or more processed images and the processed triggering audio segment;
retrieving AR instructional content related to the medical imaging or medical therapy device matching the generated one or more query vectors from a database; and
outputting the AR instructional content one or more of (i) displayed superimposed on video displayed by the AR device and/or (ii) displayed on a head mounted display of the AR device and/or (iii) as audio content via a loudspeaker of the AR device.

### Claim 2 (depends on 1)

The non-transitory computer-readable medium of claim 1 wherein the outputting includes one of:
displaying the AR instructional content superimposed on video displayed by the AR device comprising a cellular telephone or electronic tablet; or
displaying the AR instructional content superimposed on a head mounted display of the AR device.

### Claim 3 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein content of the database is indexed by index data structures generated using a same process as is used in generating the one or more query vectors, and
the matching comprises matching the one or more query vectors with the index data structures indexing the retrieved AR instructional content.

### Claim 4 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein content of the database is encoded as vectors using a same process as is used in generating the one or more query vectors, and the retrieving includes:
retrieving encoded AR instructional content from the database; and
decoding the encoded AR instructional content with an auto encoder.

### Claim 5 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the generating of the one or more query vectors data structures includes:
processing the one or more images using a neural network (NN).

### Claim 6 (depends on 5)

The non-transitory computer-readable medium of claim 5, wherein the NN processing the one or more images is a convolutional NN.

### Claim 7 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the generating of the one or more query vectors includes:
processing the triggering audio segment using a neural network (NN).

### Claim 8 (depends on 7)

The non-transitory computer-readable medium of claim 7, wherein the NN processing the triggering audio segment is a recurrent NN.

### Claim 9 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the retrieving uses a cosine similarity metric to match the generated one or more query vectors with the AR instructional content.

### Claim 10 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein outputting the visual instructions further includes one or more of:
outputting a relevant section of a service manual related to the component;
outputting annotations from the service manual related to the component.

### Claim 11

An augmented reality (AR) content generation system, comprising:
an AR device including a camera, a microphone, a display, and a loudspeaker, the camera being configured to acquire one or more images of a component of a medical imaging or medical therapy device;
at least one electronic processor programmed to:
process the acquired one or more images with an image processor to generate one or more processed images;
receive, from the microphone, a triggering audio segment;
process the received triggering audio segment with a speech-to-text process or a transcription process to generate a processed triggering audio segment generate one or more query vectors from both the one or more processed images and the processed triggering audio segment;
retrieve AR instructional content related to the medical imaging or medical therapy device matching the generated one or more query vectors from a database; and
output the AR instructional content one or more of (i) displayed superimposed on video displayed by the AR device and/or (ii) displayed on a head mounted display of the AR device and/or (iii) as audio content via the loudspeaker.

### Claim 12 (depends on 11)

The system of claim 11, wherein the at least one electronic processor is programmed to:
index content of the database by index vectors generated using a same process as is used in generating the one or more query vectors, and
matching the one or more query vectors with the index vectors indexing the retrieved AR instructional content.

### Claim 13 (depends on 11)

The system of claim 11, wherein the at least one electronic processor is programmed to:
encode content of the database as vectors using a same process as is used in generating the one or more query vectors,
retrieve encoded AR instructional content from the database; and
decode the encoded AR instructional content with an auto encoder.

### Claim 14 (depends on 11)

The system of claim 11, wherein the at least one electronic processor is programmed to:
generate the one or more query vectors by processing the one or more images using a convolutional neural network (CNN).

### Claim 15 (depends on 11)

The system of claim 11, wherein the at least one electronic processor is programmed to:
generate the one or more query vectors by processing the triggering audio segment using a recurrent neural network (RNN).

### Claim 16 (depends on 11)

The system of claim 11, wherein the at least one electronic processor is programmed to:
retrieve the AR instructional content with a cosine similarity metric to match the generated one or more query vectors with the AR instructional content.

### Claim 17 (depends on 11)

The system of claim 11, wherein the at least one electronic processor is programmed to output the visual instructions by one or more of:
outputting a relevant section of a service manual related to the component;
outputting annotations from the service manual related to the component.

### Claim 18

An augmented reality (AR) content generation method, comprising:
acquiring, with a camera of an AR device, one or more images of a component of a medical imaging or medical therapy device;
receiving, from a microphone of the AR device, a triggering audio segment;
identifying AR instructional content related to the medical imaging or medical therapy device in a database using both the one or more images of the component of the medical imaging or medical therapy device and the triggering audio segment;
retrieving the identified AR instructional content from the database; and
outputting the retrieved AR instructional content one or more of (i) displayed superimposed on video displayed by the AR device and/or (ii) displayed on a head mounted display of the AR device and/or (iii) as audio content via a loudspeaker of the AR device, the visual instructions including one or more of a relevant section of a service manual related to the component and annotations from the service manual related to the component.

### Claim 19 (depends on 18)

The method of claim 18, wherein the identifying of the AR instructional content includes:
generating a joint vector representation of the one or more images of the component of the medical imaging or medical therapy device and the triggering audio segment, and
matching the structures joint vector representation with index vectors indexing AR instructional content of the database.