# US 12,001,752 B1 · Voice assistance in virtual reality environments and augmented reality environments > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,001,752 B1 - Document ID: US12001752B1 - Status: Granted - Kind: B1 - Application number: 18/165,795 - Filed: 2023-02-07 - Granted: 2024-06-04 - Earliest claimed: 2023-02-07 - Assignees: Capital One Services, LLC - Applicants: Capital One Services, LLC - Inventors: Julian Duque; Tania Cruz Morales; Kevin M. Lowe; Helen Temesghen-Blount; Luis Catacora - Web page: https://patentgrep.com/patent/US12001752B1 ## CPC - G06F 3/16: Sound input; Sound output (speech processing G10L) - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 40/56: Natural language generation - G10L 13/00: Speech synthesis; Text to speech systems - G10L 13/027: Concept to speech synthesisers; Generation of natural phrases from machine-based concepts (generation of parameters for speech synthesis out of text G10L13/08) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract In some implementations, a voice assistance device may receive an indication of a visualization change associated with the virtual or augmented reality environment. The voice assistance device may determine that a change value, calculated based on the indication of the visualization change, satisfies a relevance threshold. The voice assistance device may apply, based on the indicator of change satisfying the relevance threshold, a description model to determine at least one sentence describing the virtual or augmented reality environment. The voice assistance device may apply a text-to-speech model to the at least one sentence to generate a digital sound signal and may transmit, to an output component, the digital sound signal. ## Claims ### Claim 1 A system for providing voice assistance in a virtual or augmented reality environment, the system comprising: one or more memories; and one or more processors, communicatively coupled to the one or more memories, configured to: receive a first indication of a first visualization associated with the virtual or augmented reality environment; receive a second indication of a second visualization, subsequent to the first visualization, associated with the virtual or augmented reality environment; determine that an indicator of change from the first visualization to the second visualization satisfies a relevance threshold; apply an image model, based on the indicator of change satisfying the relevance threshold, to determine one or more string descriptors for the second visualization; apply a language model to the one or more string descriptors to generate at least one sentence; apply a text-to-speech model to the at least one sentence to generate a digital sound signal; transmit the digital sound signal to an output component; receive a third indication of a third visualization, subsequent to the second visualization, associated with the virtual or augmented reality environment; determine that an additional indicator of change from the second visualization to the third visualization fails to satisfy the relevance threshold; receive a command to provide voice assistance; apply the language model, based on the command to provide voice assistance, to generate at least one additional sentence; apply the text-to-speech model to the at least one additional sentence to generate an additional digital sound signal; and transmit the additional digital sound signal to the output component. ### Claim 2 (depends on 1) The system of claim 1, wherein the one or more processors, to receive the first indication of the first visualization, are configured to: receive the first indication from a server associated with the virtual or augmented reality environment. ### Claim 3 (depends on 1) The system of claim 1, wherein the one or more processors, to determine the indicator of change, are configured to: calculate the indicator of change as a measurement based on pixel changes from the first visualization to the second visualization. ### Claim 4 (depends on 1) The system of claim 1, wherein the first indication of the first visualization is a first I frame, and the second indication of the second visualization is a second I frame. ### Claim 5 (depends on 1) The system of claim 1, wherein the one or more processors, to determine the indicator of change, are configured to: apply a motion model to the first visualization and the second visualization to generate the indicator of change. ### Claim 6 (depends on 1) The system of claim 1, wherein the one or more processors, to determine the indicator of change, are configured to: receive a tag associated with the second visualization that includes the indicator of change. ### Claim 7 (depends on 1) The system of claim 1, wherein the one or more processors, to determine the indicator of change, are configured to: apply the image model to the first visualization to generate one or more first string descriptors; apply the image model to the second visualization to generate one or more second string descriptors; and calculate the indicator of change based on the one or more first string descriptors and the one or more second string descriptors. ### Claim 8 A method of providing voice assistance in a virtual or augmented reality environment, comprising: receiving, at a device, an indication of a visualization change associated with the virtual or augmented reality environment; determining, at the device, that a change value, calculated based on the indication of the visualization change, satisfies a relevance threshold; applying, at the device and based on the indicator of change satisfying the relevance threshold, a description model to determine at least one sentence describing the virtual or augmented reality environment; applying, at the device, a text-to-speech model to the at least one sentence to generate a digital sound signal; transmitting, at the device and to an output component, the digital sound signal; receiving, at the device, an additional indication of a visualization change associated with the virtual or augmented reality environment; determining, at the device, that an additional change value, calculated based on the additional indication of visualization change, fails to satisfy the relevance threshold; receiving, at the device, a command to provide voice assistance; applying, at the device and based on the command to provide voice assistance, the description model to determine at least one additional sentence describing the virtual or augmented reality environment; applying, at the device, the text-to-speech model to the at least one additional sentence to generate an additional digital sound signal; and transmitting, at the device and to the output component, the additional digital sound signal. ### Claim 9 (depends on 8) The method of claim 8, wherein receiving the indication of the visualization change comprises: determining a first visualization for the virtual or augmented reality environment; receiving input associated with interaction with the virtual or augmented reality environment; determining a second visualization for the virtual or augmented reality environment based on the input; and generating the indication of the visualization change based on determining the second visualization. ### Claim 10 (depends on 8) The method of claim 8, wherein the change value comprises an average over a plurality of frames within a time window. ### Claim 11 (depends on 8) The method of claim 8, further comprising: receiving an indication of a setting associated with a level of voice assistance; and determining the relevance threshold based on the setting. ### Claim 12 (depends on 11) The method of claim 11, wherein determining the relevance threshold comprises: selecting the relevance threshold, from a plurality of possible relevance thresholds, based on the setting. ### Claim 13 (depends on 8) The method of claim 8, wherein applying the text-to-speech model to the at least one sentence comprises: transmitting, to a cloud service implementing the text-to-speech model, the at least one sentence; and receiving, from the cloud service, the digital sound signal based on the at least one sentence. ### Claim 14 A non-transitory computer-readable medium storing a set of instructions for providing voice assistance in a virtual or augmented reality environment, the set of instructions comprising: one or more instructions that, when executed by one or more processors of a device, cause the device to: display a first visualization associated with the virtual or augmented reality environment; display a second visualization, subsequent to the first visualization, associated with the virtual or augmented reality environment; determine that an indicator of change from the first visualization to the second visualization satisfies a relevance threshold; apply a description model, based on the indicator of change satisfying the relevance threshold, to determine at least one sentence describing the virtual or augmented reality environment; output, to a user and using an output component, sound based on applying a text-to-speech model to the at least one sentence; display a third visualization, subsequent to the second visualization, associated with the virtual or augmented reality environment; determine that an additional indicator of change from the second visualization to the third visualization fails to satisfy the relevance threshold; receive, from the user and using an input component, a command to provide voice assistance; apply the description model, based on the command to provide voice assistance, to determine at least one additional sentence describing the virtual or augmented reality environment; and output, to the user and using the output component, additional sound based on applying the text-to-speech model to the at least one additional sentence. ### Claim 15 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein the one or more instructions, when executed by the one or more processors, further cause the device to: receive, from the user and using an input component, feedback associated with the sound; and update the description model based on the feedback. ### Claim 16 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein the indicator of change is determined using a change model, and wherein the one or more instructions, when executed by the one or more processors, further cause the device to: receive, from the user and using an input component, feedback associated with the sound; and update the change model based on the feedback. ### Claim 17 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein the one or more instructions, that cause the device to determine that the indicator of change satisfies the relevance threshold, cause the device to: receive a location associated with the second visualization, wherein the indicator of change is further based on the location. ### Claim 18 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein the one or more instructions, when executed by the one or more processors, further cause the device to: receive, from the user and using an input component, a command to provide voice assistance; and calculate the indicator of change based on the command to provide voice assistance. ### Claim 19 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein the one or more instructions, that cause the device to display the first visualization, cause the device to: receive data encoding the first visualization from a remote server; and transmit the data encoding the first visualization to a display. ### Claim 20 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein the virtual or augmented reality environment is a virtual reality environment.