# US 10,454,943 B2 · Augmented and virtual reality based process oversight > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,454,943 B2 - Document ID: US10454943B2 - Status: Granted - Kind: B2 - Application number: 15/234,786 - Filed: 2016-08-11 - Published: 2017-02-23 - Granted: 2019-10-22 - Earliest claimed: 2015-08-17 - Assignees: The Toronto-Dominion Bank - Applicants: The Toronto-Dominion Bank - Inventors: Christianne Moretti; James Zachary Pryor; Matthew Hamilton; Jonathan K. Barnett; Paul Mon-Wah Chan; Orin Del Vecchio; John Barbon; John Jong-Suk Lee - Web page: https://patentgrep.com/patent/US10454943B2 ## CPC - H04L 63/107: {wherein the security policies are location-dependent, e.g. entities privileges depend on current location or allowing specific operations only from locally connected terminals} - G02B 27/01: Head-up displays - G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units} - G06Q 10/103: {Workflow collaboration or project management} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The present disclosure involves systems and computer implemented methods for generating a virtual or augmented reality-based oversight system of business processes at a location remote from where the business process is executed. In one example, system operations include identifying a process-based operation performed at a physical first location, where the operation is associated with a request for oversight from a remote second location. A current state of the operation at the first location is identified, and at least one input source at the first location is identified. A simulation of the current state of the operation at the first location incorporating input from the input source(s) is generated for presentation at the second location. An action performed at the first location in the current state of the operation is identified, and the second location is then transmitted a request to authorize the identified action. ## Claims ### Claim 1 A system comprising: a memory; at least one hardware processor interoperably coupled with the memory and configured to: identify, at a first system executing a process-based operation performed at a physical first location in association with a customer, a current action of the process-based operation performed by a first user at the first location for the customer, the current action of the process-based operation triggering an automatic request for oversight to be performed by a second system at a remote second location; in response to the identifying, identify a current state associated with a current step of the process-based operation at the first location; determine at least one image or video input source capturing at least a portion of the process-based operation in the current state occurring at the first location; generate, for presentation at a display at the second location, a simulation of the current state of the process-based operation at the first location, the simulation incorporating input from the at least one determined image or video input source and simulating the current state of the process-based operation performing the current action of the process-based operation from a perspective of the first user; identify an action performed at the first location in the current state of the process-based operation, wherein the action performed at the first location seeks authorization from the second location; and transmit a request to the second location to authorize the identified action. ### Claim 2 (depends on 1) The system of claim 1, wherein the physical first location is associated with a first user, and wherein the remote second location is associated with a second user. ### Claim 3 (depends on 2) The system of claim 2, the at least one hardware processor configured to: receive authorization from the second user at the second location; and in response to receiving authorization from the second user at the second location, notify the first user at the first location of the authorization and allowing the process-based operation to continue to a next state at the first location. ### Claim 4 (depends on 2) The system of claim 2, the at least one hardware processor configured to: in response to not receiving authorization from the second user at the second location, suspending execution of the process-based operation at the first location. ### Claim 5 (depends on 4) The system of claim 4, wherein suspending execution of the process-based operation at the first location includes providing control of the process-based operation to the second user at the second location, wherein the second user can perform or revise the identified action within the simulation of the current state, wherein feedback representing the performed or revised actions performed by the second user at the second location are transmitted to a display associated with the first user at the first location. ### Claim 6 (depends on 4) The system of claim 4, wherein suspending execution of the process-based operation at the first location includes: providing control of the process-based operation to the second user at the second location; receiving instructions from the second user related to at least one correction to the identified action; and providing control of the process-based operation to the first user at the first location to revise the identified action based on the received instructions. ### Claim 7 (depends on 2) The system of claim 2, wherein identifying the process-based operation includes: identifying, from a plurality of process-based operations performed at a plurality of physical locations by at least the first user and a third user, a queue of two or more requests for oversight of at least two process-based operations performed by at least one of a second user and a fourth user remote from at least the first user and the third user, wherein the identified process-based operation is associated with the remote second user for oversight. ### Claim 8 (depends on 2) The system of claim 2, wherein identifying the at least one input source associated with the current state of the process-based operation at the first location of the first user includes identifying one of a video or image feed capturing data at the location of the first user, a video or image feed associated with the first user, a video or image feed associated with the process-based actions of the first user at the first location, or a visualization of a display associated with the first user at the first location. ### Claim 9 (depends on 2) The system of claim 2, wherein identifying the at least one input source associated with the current state of the process-based operation at the first location of the first user includes a state indication associated with at least one connected device present at the first location or a state indication associated with at least one connected device involved in the process-based operation. ### Claim 10 (depends on 2) The system of claim 2, wherein the first user is not notified of the automatic request for oversight from the remote second user. ### Claim 11 (depends on 2) The system of claim 2, the at least one hardware processor configured to: display, to the second user at the second location, the generated simulation of the current state of the process-based operation. ### Claim 12 (depends on 11) The system of claim 11, wherein the generated simulation of the current state of the process-based operation is displayed at the second location to the second user via an augmented reality display, wherein the augmented reality display comprises an overlay of at least a portion of input from the at least one input source on real-world content at the second location, and wherein the augmented reality display allows for input from the second user to authorize actions in the process-based operation. ### Claim 13 (depends on 11) The system of claim 11, wherein the generated simulation of the current state of the process-based operation is displayed at the second location to the second user via virtual reality display, wherein the virtual reality display is presented within a virtual reality output device at the second location, and wherein the virtual reality display allows for input from the second user to authorize actions in the process-based operation. ### Claim 14 A non-transitory, computer-readable medium storing computer-readable instructions executable by a computer and configured to: identify, at a first system executing a process-based operation performed at a physical first location in association with a customer, a current action of the process-based operation performed by a first user at the first location for the customer, the current action of the process-based operation triggering an automatic request for oversight to be performed by a second system at a remote second location; in response to the identifying, identify a current state associated with a current step of the process-based operation at the first location; determine at least one image or video input source capturing at least a portion of the process-based operation in the current state occurring at the first location; generate, for presentation at a display at the remote second system, a simulation of the current state of the process-based operation at the first location, the simulation incorporating input from the at least one identified input source; generate, for presentation at a display at the second location, a simulation of the current state of the process-based operation at the first location, the simulation incorporating input from the at least one determined image or video input source and simulating the current state of the process-based operation performing the current action of the process-based operation from a perspective of the first user; identify an action performed at the first location in the current state of the process-based operation, wherein the action performed at the first location requires authorization at the remote second location; and transmit a request to the remote second location to authorize the identified action. ### Claim 15 (depends on 14) The medium of claim 14, wherein the physical first location is associated with a first user, and wherein the remote second location is associated with a second user. ### Claim 16 (depends on 15) The medium of claim 15, the instructions executable by a computer and configured to, in response to not receiving authorization from the second user at the second location, suspending execution of the process-based operation at the first location, wherein suspending execution of the process-based operation at the first location includes providing control of the process-based operation to the second user at the second location, wherein the second user can perform or revise the identified action. ### Claim 17 (depends on 15) The medium of claim 15, wherein identifying the at least one input source associated with the current state of the process-based operation at the first location of the first user includes at least one of: identifying one of a video or image feed capturing data at the first location of the first user, a video or image feed associated with the first user, a video or image feed associated with the process-based actions of the first user, or a visualization of a display associated with the first user at the first location; or identifying the at least one input source associated with the current state of the process-based operation at the first location includes a state indication associated with at least one connected device present at the first location or a state indication associated with at least one connected device involved in the process-based operation. ### Claim 18 (depends on 15) The medium of claim 15, the instructions executable by a computer and configured to display, to the second user at the second location, the generated simulation of the current state of the process-based operation, wherein the generated simulation of the current state of the process-based operation is displayed to the second user at the second location via an augmented reality display, wherein the augmented reality display comprises an overlay of at least a portion of input from the at least one input source on real-world content at the second location to the second user. ### Claim 19 (depends on 15) The medium of claim 15, the instructions executable by a computer and configured to display, to the second user at the second location, the generated simulation of the current state of the process-based operation, wherein the generated simulation of the current state of the process-based operation is displayed to the second user at the second location via virtual reality display, wherein the virtual reality display is presented within a virtual reality output device at the second location to the second user. ### Claim 20 A computerized method performed by one or more processors, the method comprising: identifying, by at least one processor and at a first system executing a process-based operation performed at a physical first location in association with a customer, a current action of the process-based operation performed by a first user at the first location for the customer, the current action of the process-based operation triggering an automatic request for oversight to be performed by a second system at a remote second location; in response to the identifying, identifying, by at least one processor, a current state associated with a current step of the process-based operation at the first location; determining, by at least one processor, at least one image or video input source capturing at least a portion of the process-based operation in the current state occurring at the first location; generating, by at least one processor and for presentation at a display at the second location, a simulation of the current state of the process-based operation at the first location, the simulation incorporating input from the at least one determined image or video input source and simulating the current state of the process-based operation performing the current action of the process-based operation from a perspective of the first user; identifying, by at least one processor, an action performed at the first location in the current state of the process-based operation, wherein the action performed at the first location requires authorization by the second location; and transmitting, by at least one processor, a request to the second location to authorize the identified action.