# US 2019/0244425 A1 · Augmented Reality Assistant

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244425 A1
- Document ID: US20190244425A1
- Status: Published application
- Kind: A1
- Application number: 15/889,964
- Filed: 2018-02-06
- Published: 2019-08-08
- Earliest claimed: 2018-02-06
- Assignees: ServiceNow, Inc.
- Applicants: ServiceNow, Inc.
- Inventors: Darius Koohmarey
- Web page: https://patentgrep.com/patent/US20190244425A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06Q 30/016: After-sales
- G06V 20/20: in augmented reality scenes
- G06V 40/16: Human faces, e.g. facial parts, sketches or expressions
- H04L 67/12: specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
- H04L 67/125: involving control of end-device applications over a network
- H04L 67/131: Protocols for games, networked simulations or virtual reality
- H04L 67/52: specially adapted for the location of the user terminal
- H04N 5/00: Details of television systems (scanning details or combination thereof with generation of supply voltages H04N3/00)
- H04W 4/02: Services making use of location information
- G06T 2215/16: Using real world measurements to influence rendering
- H04W 4/70: Services for machine-to-machine communication [M2M] or machine type communication [MTC]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

An augmented reality AR device may be communicatively connected to a remote network management platform configured to support a managed network. The AR device may capture an image of a real object in the field of view of an imaging component of the AR device. The real object may be recognized as a known managed object of the managed network. The AR device may also concurrently determine context information indicating a location or physical environment. The AR device may then transmit an identifier of the known managed object and the context information in a message to the management platform. In response, the AR device may receive data associated with the known managed. The AR device may then display a virtual object in a virtual space superimposed on the captured image of the real object, where the virtual object and the virtual space are based on the received management data.

## Claims

### Claim 1

A computer-implemented method carried out by an augmented reality (AR) device communicatively connected to a remote network management platform configured to support a managed network, the method comprising:
capturing an image of a real object in a field of view (FOV) of an imaging component of the AR device;
in real time with capturing the image, recognizing the real object in the captured image as a known managed object of the managed network;
in real time with capturing the image, determining context information indicative of at least one of a location of the AR device, or a physical environment of the AR device;
in real time with capturing the image, transmitting a message to the remote network management platform, the message including both an identifier of the known managed object and the context information;
in response to the transmitted message, receiving from the remote management platform management data associated with the known managed object; and
in real time with capturing the image, displaying in the display component of the AR device a virtual object in a virtual space superimposed on the captured image of the real object, the virtual object and the virtual space being based on the received management data.

### Claim 2 (depends on 1)

The method of claim 1, wherein the real object is a person and the known managed object is a particular customer of the managed network,
and wherein recognizing the real object in the captured image as the known managed object of the managed network comprises applying a facial recognition operation to the captured image to recognize the person to be the particular customer of the managed network.

### Claim 3 (depends on 2)

The method of claim 2, wherein the context information comprises an identification of a customer service interaction that is concurrent with capturing the image,
and wherein transmitting the message to the remote network management platform, the message including both an identifier of the known managed object and the context information, comprises transmitting to the remote network management platform an identity of the particular customer and a request for customer management information associated with the particular customer and selected according to the identified customer service interaction.

### Claim 4 (depends on 2)

The method of claim 2, wherein the management data associated with the known managed object is customer management information associated with the particular customer,
and wherein displaying the virtual object in the virtual space superimposed on the captured image of the real object, the virtual object and the virtual space being based on the received management data, comprises:
displaying a virtual rendering of the customer management information superimposed in the FOV while concurrently displaying the image of the person.

### Claim 5 (depends on 4)

The method of claim 4, wherein the customer management information comprises a customer profile and information indicating a status of one or more customer service issues associated with the particular customer.

### Claim 6 (depends on 1)

The method of claim 1, wherein the real object is a network device and the known managed object is a particular managed device of the managed network,
and wherein recognizing the real object in the captured image as the known managed object of the managed network comprises recognizing an asset identifier on the network device as identifying the particular managed device of the managed network.

### Claim 7 (depends on 6)

The method of claim 6, wherein the context information comprises an identification of a network device field-service operation that is concurrent with capturing the image,
and wherein transmitting the message to the remote network management platform, the message including both an identifier of the known managed object and the context information, comprises transmitting to the remote network management platform an identity of the particular managed device and a request for device management information associated with the particular managed device and selected according to the identified network device field-service operation.

### Claim 8 (depends on 6)

The method of claim 6, wherein the management data associated with the known managed object is device management information associated with the particular managed device,
and wherein displaying the virtual object in the virtual space superimposed on the captured image of the real object, the virtual object and the virtual space being based on the received management data, comprises:
displaying a virtual rendering of the device management information together with a virtual graphical representation of a location of the particular managed device in the managed network superimposed in the FOV while concurrently displaying the image of the network device.

### Claim 9 (depends on 8)

The method of claim 8, wherein the device management information comprises information indicating a status of one or more other devices communicatively connected to the particular managed device.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
receiving a voice prompt at an audio input component of the AR device,
in response to the voice prompt, activating an interface for inputting data to a network management query to the remote network management platform.

### Claim 11

A computer-implemented method carried out by an augmented reality (AR) device communicatively connected to a remote network management platform configured to support a managed network, the method comprising:
in a display component of the AR device, displaying an image of a real object captured in real time in a field of view (FOV) of an imaging component of the AR device;
in real time with displaying the image, receiving from the remote management platform management data associated with a known managed object of the managed network, the known managed object being identified with the real object in the captured image;
in real time with displaying the image, displaying in the display component of the AR device a virtual object in a virtual space superimposed on the captured image of the real object, the virtual object and the virtual space being based on the received management data;
in real time with displaying the image, receiving user input interacting with the virtual object; and
in response to the received user input, initiating a network management operation directed to the real object.

### Claim 12 (depends on 11)

The method of claim 11, wherein displaying the virtual object in the virtual space superimposed on the captured image of the real object, the virtual object and the virtual space being based on the received management data, comprises:
displaying a virtual rendering of the received management data information superimposed in the FOV while concurrently displaying the captured image of the real object.

### Claim 13 (depends on 12)

The method of claim 12, wherein receiving the user input interacting with the virtual object comprises:
detecting a user interaction with the virtual rendering of the received management data information, the detected user interaction being at least one of a voice command including a spoken reference to displayed content of the received management data information, or a hand gesture including virtual manipulation of displayed content of the received management data information,
and wherein initiating a network management operation directed to the real object comprises inputting the displayed content to the data network management operation.

### Claim 14 (depends on 11)

The method of claim 11, wherein the real object is a person and the known managed object is a particular customer of the managed network,
wherein the management data associated with the known managed object is customer management information associated with the particular customer,
and wherein displaying the virtual object in the virtual space superimposed on the captured image of the real object, the virtual object and the virtual space being based on the received management data, comprises:
displaying a virtual rendering of a customer profile and information indicating a status of one or more customer service issues associated with the particular customer superimposed in the FOV while concurrently displaying the image of the person.

### Claim 15 (depends on 14)

The method of claim 14, wherein the customer management information comprises a customer profile and information indicating a status of one or more customer service issues associated with the particular customer,
wherein receiving the user input interacting with the virtual object comprises receiving a voice command including a spoken reference to displayed content of the virtual rendering of the customer management information,
and wherein initiating a network management operation directed to the real object comprises inputting the displayed content to a customer management operation.

### Claim 16 (depends on 11)

The method of claim 11, wherein the real object is a network device and the known managed object is a particular managed device of the managed network,
wherein the management data associated with the known managed object is device management information associated with the particular managed device,
and wherein displaying the virtual object in the virtual space superimposed on the captured image of the real object, the virtual object and the virtual space being based on the received management data, comprises:
displaying a virtual rendering of the device status information together with a virtual graphical representation of a location of the particular managed device in the managed network superimposed in the FOV while concurrently displaying the image of the network device.

### Claim 17 (depends on 16)

The method of claim 16, wherein the virtual graphical representation of a location of the particular managed device in the managed network comprises a graphical network map indicating locations and communicative interconnections of the particular managed device and one or more other devices of the managed network,
wherein receiving the user input interacting with the virtual object comprises detecting a hand gesture selecting one or more of the particular managed device and the other devices of the managed network,
and wherein initiating a network management operation directed to the real object comprises displaying status information of the selected devices, superimposed in the FOV.

### Claim 18

A computing system disposed within a remote network management platform and configured to support a managed network, the computing system comprising:
one or more processors;
memory; and
program instructions, stored in the memory, that upon execution by the one or more processors cause the computing system to perform operations including:
receiving a message transmitted by a remote augmented reality (AR) device via the managed network, the message including information designating an object recognized in a real-time field of view (FOV) captured by the remote AR device, and context information associated with the recognized object;
based on the context information, selecting a category of managed objects of the managed network;
associating the recognized object with a particular managed object in a database of managed objects of the selected category;
retrieving information associated with the particular object from the database of managed objects of the selected category; and
transmitting a reply message to the remote AR device via the managed network, the reply messaging including the retrieved information for display on the remote AR device concurrently with display of the real-time FOV.

### Claim 19 (depends on 18)

The computing system of claim 18, wherein the recognized object is a face of a particular customer of the managed network, and the particular managed object is a particular data record including customer information about the particular customer,
wherein the database of managed objects of the selected category is a database of customers of the managed network,
and wherein associating the recognized object with the particular managed object in the database of managed objects of the selected category comprises identifying the particular data record based on the designating information.

### Claim 20 (depends on 18)

The computing system of claim 18, wherein the recognized object is a printed identifier on a particular network device, and the particular managed object is a particular data record including device information about the particular network device,
wherein the database of managed objects of the selected category is a database of network devices of the managed network,
and wherein associating the recognized object with the particular managed object in the database of managed objects of the selected category comprises identifying the particular data record based on the designating information.
