# US 2025/0085784 A1 · Multi-dimensional extrasensory deciphering of human gestures for digital authentication and related events

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0085784 A1
- Document ID: US20250085784A1
- Status: Published application
- Kind: A1
- Application number: 18/755,536
- Filed: 2024-06-26
- Published: 2025-03-13
- Earliest claimed: 2023-09-07
- Assignees: Bank of America Corporation
- Applicants: Bank of America Corporation
- Inventors: Subburathinam Krishnan; Abhishek Arya; Prabhakaran Balasubramanian; Durga Prasad P. Khuttumolu
- Web page: https://patentgrep.com/patent/US20250085784A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 21/31: User authentication
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Virtual assistant devices and related methods are described that allow users having special needs to communicate with the virtual assistant device. The virtual assistant devices are augmented with additional intelligent sensors, including, but not limited to, three-dimensional optical sensors that are used to capture optical signals of user's non-verbal communication (e.g., sign language, lip movements, gestures or the like). Additionally, the virtual assistant device includes an Artificial Intelligence (AI)-based engine that includes one or more Machine Learning (ML) models trained on user-specific data and used to determine the non-verbal communication of the user (i.e., sign language, lip movements or other gestures) based on inputs derived from the sensors. Moreover, the virtual assistant devices may be configured to generate and display visual information, such as three-dimensional floating avatars of the user and/or the virtual assistant.

## Claims

### Claim 1

A system for multi-dimensional extrasensory deciphering of non-verbal communication of a user, the system comprising:
a virtual assistant device including a memory, one or more computing processing devices in communication with the memory and a plurality of sensors in communication with at least one of the one or more computing processing devices, wherein the plurality of sensors include at least one three-dimensional optical sensor configured for capturing, at least, optical signals that detect motion of the user;
an artificial intelligence (AI)-based engine stored in the memory, executable by at least one of the one or more computing processing devices, and including at least one first machine-learning (ML) model trained at least on user-specific data including historical resource exchange event data for purposes of determining non-verbal communication of the user, wherein the AI-based engine is configured to:
receive, at one or more of the first ML models, first inputs derived from signals captured from the plurality of sensors including the optical signals captured from the at least one three-dimensional optical sensor,
based on the first inputs, execute the one or more of the first ML models to determine a first transcript of the non-verbal communication of the user, and
communicate the first transcript of the non-verbal communication to a response system associated with the virtual assistant device, wherein the response system processes the non-verbal communication to determine a responsive communication to provide to the user; and
a floating image-generating and communication engine stored in the memory, executable by at least one of the one or more computing processing devices and configured to:
generate an avatar that is a three-dimensional floating image of a virtual assistant,
display the avatar in a three-dimensional-enabled display system, wherein the avatar is configured to perform movements that provide the responsive communication to the user in a non-verbal communication format, and
adjust the movements performed by the avatar based on the detected movements of the avatar in comparison to the responsive communication.

### Claim 2 (depends on 1)

The system of claim 1, wherein the floating image-generating and communication engine is further configured to display the avatar in the three-dimensional-enabled display, wherein the avatar is configured to perform movements that replicate detected motion of the user.

### Claim 3 (depends on 2)

The system of claim 2, wherein the least one three-dimensional optical sensor are configured for detecting movements of the avatar, and
wherein the floating image-generating and communication engine further comprises a quality assurance sub-engine configured to adjust the movements performed by the avatar based on the detected movements of the avatar in comparison to the detected motion of the user.

### Claim 4 (depends on 1)

The system of claim 1, wherein the floating image-generating and communication engine is further configured to display the avatar in the three-dimensional-enabled display in at least one chosen from group consisting of (i) an active mode in which the avatar is displayed as a real-time response to capturing the optical signals associated with detected motion of the user and (ii) a passive mode in which the avatar is displayed in response to the virtual assistant device determining preconfigured communication that has been requested by the user.

### Claim 5 (depends on 1)

The system of claim 1, wherein the AI-based engine further comprises at least one second machine-learning (ML) model trained at least on user-specific data including historical resource exchange event emotional state data for purposes of determining an emotional state the user, and wherein the AI-based engine is configured to:
receive, at one or more of the second ML models, second inputs derived from signals captured from the plurality of sensors including the optical signals captured from the at least one three-dimensional optical sensor,
based on the second inputs, execute the one or more of the second ML models to determine an emotional state of the user, and
communicate the emotional state of the user to the response system, wherein the response system determines the responsive communication based further on the emotional state of the user.

### Claim 6 (depends on 1)

The system of claim 1, wherein the AI-based engine further comprises a user authentication sub-engine comprising at least one second machine-learning (ML) model trained at least on user-specific data including historical authentication credential input data for purposes of determining non-verbal authentication credential communication from the user, wherein the user authentication sub-engine is configured to:
receive, at one or more of the second ML models, second inputs derived from signals captured from the plurality of sensors including the optical signals captured from the at least one three-dimensional optical sensor,
based on the second inputs, execute the one or more of the second ML models to determine a second transcript of the non-verbal authentication credential communication of the user, and
communicate the second transcript of the non-verbal authentication credential communication to an authenticating entity, wherein the authenticating entity processes the non-verbal authentication credential communication to authenticate the user for conducting one or more resource exchange events.

### Claim 7 (depends on 1)

The system of claim 1, wherein the at least one three-dimensional optical sensor is further configured for detecting a preconfigured visual cue of the user, wherein detection of the preconfigured visual cue triggers further continual capturing of the optical signals.

### Claim 8

A computer-implemented method for multi-dimensional extrasensory deciphering of non-verbal communication of a user, at least a portion of the method being executable by one or more computing device processors and comprising:
capturing optical signals from at least one three-dimensional optical sensor disposed within a virtual assistant device;
receiving, at one or more first ML models, first inputs derived from the optical signals captured from the at least one three-dimensional optical sensor;
based on the first inputs, executing the one or more first ML models to determine a first transcript of the non-verbal communication of the user;
communicating first transcript of the non-verbal communication to a response system associated with the virtual assistant device, wherein the response system processes the non-verbal communication to determine a responsive communication to provide to the user;
generating an avatar that is a three-dimensional floating image of a virtual assistant;
displaying the avatar in a three-dimensional-enabled display system, wherein the avatar is configured to perform movements that provide the responsive communication to the user in a non-verbal communication format; and
adjusting the movements performed by the avatar based on the detected movements of the avatar in comparison to the responsive communication.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, further comprising:
generating at least one chosen from the group consisting of (i) a first avatar that is a three-dimensional floating image of the user and (ii) a second avatar that is a three-dimensional floating image of a virtual assistant; and
displaying the first avatar or second avatar in a three-dimensional-enabled display that is in communication with the virtual assistant device, wherein, upon display, the first avatar is configured to perform movements that replicate detected motion of the user and the second avatar is configured to perform movements that provide the responsive communication to the user in a non-verbal communication format.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, further comprising
detecting from the optical signals movements of at least one chosen from the group consisting of (i) the first avatar and (ii) the second avatar; and
adjusting the movements performed by at least one chosen from the group consisting of (i) the first avatar and (ii) the second avatar based on at least one chosen from the group consisting of (i) the detected movements of the first avatar in comparison to the detected motion of the user and (ii) the detected movements of the avatar in comparison to the responsive communication.

### Claim 11 (depends on 9)

The computer-implemented method of claim 9, wherein displaying the second avatar in the three-dimensional-enabled display further comprises displaying the second avatar in the three-dimensional-enabled display in at least one chosen from group consisting of (i) an active mode in which the avatar is displayed as a real-time response to capturing the optical signals associated with detected motion of the user and (ii) a passive mode in which the avatar is displayed in response to the virtual assistant device determining preconfigured communication that has been requested by the user.

### Claim 12 (depends on 8)

The computer-implemented method of claim 8, further comprising:
capturing signals from a plurality of sensors included within the virtual assistant device;
receiving, at one or more second ML models, second inputs derived from signals captured from the plurality of sensors including the optical signals captured from the at least one three-dimensional optical sensor;
based on the second inputs, executing the one or more second ML models to determine an emotional state of the user; and
communicating the emotional state of the user to the response system, wherein the response system determines the responsive communication based further on the emotional state of the user.

### Claim 13 (depends on 8)

The computer-implemented method of claim 8, further comprising:
capturing signals from a plurality of sensors included within the virtual assistant device;
receiving, at one or more second ML models, second inputs derived from signals captured from the plurality of sensors including the optical signals captured from the at least one three-dimensional optical sensor,
based on the second inputs, executing the one or more second ML models to determine a second transcript of a non-verbal authentication credential communication of the user; and
communicating the second transcript of a non-verbal authentication credential communication to an authenticating entity, wherein the authenticating entity processes the second transcript of a non-verbal authentication credential communication to authenticate the user for conducting one or more resource exchange events.

### Claim 14

A computer program product comprising:
a non-transitory computer-readable medium comprising sets of codes for causing one or more computing devices to:
capture optical signals from at least one three-dimensional optical sensor disposed within a virtual assistant device, wherein the optical signals detect motion of a user;
receive, at one or more first ML models, first inputs derived from the optical signals captured from the at least one three-dimensional optical sensor;
based on the first inputs, (i) execute the one or more first ML models to determine a first transcript of the non-verbal communication of the user based on the detected motion;
communicate the first transcript of the non-verbal communication to a response system associated with the virtual assistant device, wherein the response system processes the non-verbal communication to determine a responsive communication to provide to the user;
generate an avatar that is a three-dimensional floating image of a virtual assistant;
display the avatar in a three-dimensional-enabled display system, wherein the avatar is configured to perform movements that provide the responsive communication to the user in a non-verbal communication format; and
adjust the movements performed by the avatar based on the detected movements of the avatar in comparison to the responsive communication.

### Claim 15 (depends on 14)

The computer program product of claim 14, wherein the sets of codes further cause the one or more computing devices to:
generate at least one chosen from the group consisting of (i) a first avatar that is a three-dimensional floating image of the user and (ii) a second avatar that is a three-dimensional floating image of a virtual assistant; and
display the first avatar or second avatar in the three-dimensional-enabled display that is in communication with the virtual assistant device, wherein, upon display, the first avatar is configured to perform movements that replicate detected visual cues of the user and the second avatar is configured to perform movements that provide the responsive communication to the user in a non-verbal communication format.

### Claim 16 (depends on 15)

The computer program product of claim 15, wherein the sets of codes further cause the one or more computing devices to:
detect from the optical signals movements of at least one chosen from the group consisting of (i) the first avatar and (ii) the second avatar; and
adjust the movements performed by at least one chosen from the group consisting of (i) the first avatar and (ii) the second avatar based on at least one chosen from the group consisting of (i) the detected movements of the first avatar in comparison to the detected visual cues of the user and (ii) the detected movements of the avatar in comparison to the responsive communication.

### Claim 17 (depends on 15)

The computer program product of claim 15, wherein the sets of codes further cause the one or more computing devices to:
capture signals from a plurality of sensors included within the virtual assistant device;
receive, at one or more second ML models, second inputs derived from signals captured from the plurality of sensors including the optical signals captured from the at least one three-dimensional optical sensor;
based on the second inputs, execute the one or more second ML models to determine an emotional state of the user; and
communicate the emotional state of the user to the response system, wherein the response system determines the responsive communication based further on the emotional state of the user.

### Claim 18 (depends on 15)

The computer program product of claim 15, wherein the sets of codes further cause the one or more computing devices to:
capture signals from a plurality of sensors included within the virtual assistant device;
receive, at one or more second ML models, second inputs derived from signals captured from the plurality of sensors including the optical signals captured from the at least one three-dimensional optical sensor,
based on the second inputs, execute the one or more second ML models to determine a second transcript of a non-verbal authentication credential communication of the user; and
communicate the second transcript of the non-verbal authentication credential communication to an authenticating entity, wherein the authenticating entity processes the non-verbal authentication credential communication to authenticate the user for conducting one or more resource exchange events.
