# US 12,008,048 B2 · Drill back to original audio clip in virtual assistant initiated lists and reminders > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,048 B2 - Document ID: US12008048B2 - Status: Granted - Kind: B2 - Application number: 17/140,783 - Filed: 2021-01-04 - Published: 2022-07-07 - Granted: 2024-06-11 - Earliest claimed: 2021-01-04 - Assignees: Oracle International Corporation - Applicants: Oracle International Corporation - Inventors: Michael Patrick Rodgers - Web page: https://patentgrep.com/patent/US12008048B2 ## CPC - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 16/9038: Presentation of query results - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 2015/0631: {Creating reference templates; Clustering} - G10L 2015/223: {Execution procedure of a spoken command} - G10L 2015/228: {of application context} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A system drills back to an original audio clip in lists and reminders initiated by a virtual assistant. The system may receive audio input comprising a first request. Based on the first request, the system may schedule an action to be performed by the virtual assistant platform. The system stores at least a portion of the audio input and a mapping between the action and at least the portion of the audio input. The system performs the action. Subsequent to performing the action, the system receives a second request for audio playback of the first request corresponding to the action. The system retrieves at least the portion of the audio input based on the mapping between the action and at least the portion of the audio input, and plays at least the portion of the audio input comprising the first request. ## Claims ### Claim 1 A non-transitory computer readable medium comprising instructions which, when executed by one or more hardware processors, causes performance of operations comprising: receiving, by a virtual assistant platform, a first audio input comprising a first request; based on the first request, scheduling an action to be performed by the virtual assistant platform; determining that at least a portion of the first audio input includes one or more phonemes associated with a first confidence level below a threshold confidence level; responsive to determining that at least the portion of the first audio input includes one or more phonemes associated with the first confidence level below the threshold confidence level: storing (a) at least the portion of the first audio input and (b) a mapping between the action and at least the portion of the first audio input; performing, by the virtual assistant platform, the action; subsequent to performing the action: receiving, by the virtual assistant platform, a second request for audio playback of the first request corresponding to the action performed by the virtual assistant platform; retrieving at least the portion of the first audio input based on the mapping between the action and at least the portion of the first audio input; playing, by the virtual assistant platform, at least the portion of the first audio input comprising the first request. ### Claim 2 (depends on 1) The non-transitory computer readable medium of claim 1, the operations further comprising: storing (a) context information from a time the first request was received and (b) a mapping between the action and the context information, wherein the context information comprises one or more of (i) a user location when the first request was received, (ii) the time at which the first request was received, and (iii) a list of applications running at the time the first request was received; retrieving the context information based on the mapping between the action and the context information; and presenting, by the virtual assistant platform, at least a portion of the context information. ### Claim 3 (depends on 1) The non-transitory computer readable medium of claim 1, wherein the storing the at least the portion of the first audio input and the mapping between the action and at least the portion of the first audio input is performed in response to evaluating one or more characteristics of the first audio input; wherein the operations further comprise: receiving, by the virtual assistant platform, a second audio input comprising a third request; based on the third request, scheduling a second action to be performed by the virtual assistant platform; evaluating one or more characteristics of the second audio input; and refraining from storing any portion of the second audio input. ### Claim 4 (depends on 1) The non-transitory computer readable medium of claim 1, the operations further comprising: determining a second confidence level that associated with scheduling the action to be performed based on the first request; responsive to the second confidence level being below a confidence threshold, prompting a user to confirm the action. ### Claim 5 (depends on 1) The non-transitory computer readable medium of claim 1, wherein storing at least the portion of the first audio input comprises storing an entirety of the first audio input. ### Claim 6 (depends on 1) The non-transitory computer readable medium of claim 1, wherein playing at least the portion of the first audio input comprises playing an entirety of the first audio input. ### Claim 7 (depends on 1) The non-transitory computer readable medium of claim 1, the operations further comprising: parsing the first audio input to divide the first audio input into a plurality of sections; determining a particular section of the plurality of sections of the first audio input, that includes the first request; wherein storing at least the portion of the first audio input comprises storing the particular section. ### Claim 8 (depends on 1) The non-transitory computer readable medium of claim 1, the operations further comprising: parsing the first audio input to divide the first audio input into a plurality of sections; generating, for each section of the plurality of sections of the first audio input, a transcript of audio content contained in the section; determining, for each section of the plurality of sections of the first audio input, a likelihood that the transcript corresponding to the section includes any transcription error; wherein storing at least the portion of the first audio input comprises storing at least a particular section associated with the greatest likelihood that a particular transcript corresponding to the particular section includes any transcription error. ### Claim 9 (depends on 1) The non-transitory computer readable medium of claim 1, the operations further comprising: responsive to receiving the second request for audio playback of the first request corresponding to the action performed by the virtual assistant platform: marking at least the portion of the first audio input for inclusion in a training data set. ### Claim 10 (depends on 1) The non-transitory computer readable medium of claim 1, wherein the operations further comprise: storing context information at a time the first request was received, wherein the context information comprises a list of applications running at the time the first request was received; responsive to receiving the second request for audio playback of the first request: retrieving the list of applications running at the time the first request was received; and presenting, by the virtual assistant platform, at least a portion of the list of applications running at the time the first request was received. ### Claim 11 (depends on 1) The non-transitory computer readable medium of claim 1, wherein the operations further comprise: receiving, by the virtual assistant platform, a second audio input comprising a third request; determining that no portion of the second audio input includes any phoneme associated with any confidence level below the threshold confidence level; responsive to determining that no portion of the second audio input includes any phoneme associated with any confidence level below the threshold confidence level: refraining from storing any portion of the second audio input. ### Claim 12 A method comprising: receiving, by a virtual assistant platform, a first audio input comprising a first request; based on the first request, scheduling an action to be performed by the virtual assistant platform; determining that at least a portion of the first audio input includes one or more phonemes associated with a first confidence level below a threshold confidence level; responsive to determining that at least the portion of the first audio input includes one or more phonemes associated with the first confidence level below the threshold confidence level: storing (a) at least the portion of the first audio input and (b) a mapping between the action and at least the portion of the first audio input; performing, by the virtual assistant platform, the action; subsequent to performing the action: receiving, by the virtual assistant platform, a second request for audio playback of the first request corresponding to the action performed by the virtual assistant platform; retrieving at least the portion of the first audio input based on the mapping between the action and at least the portion of the first audio input; playing, by the virtual assistant platform, at least the portion of the first audio input comprising the first request; wherein the method is performed by at least one device including a hardware processor. ### Claim 13 (depends on 12) The method of claim 12, further comprising: storing (a) context information from a time the first request was received and (b) a mapping between the action and the context information, wherein the context information comprises one or more of (i) a user location when the first request was received, (ii) the time at which the first request was received, and (iii) a list of applications running at the time the first request was received; retrieving the context information based on the mapping between the action and the context information; and presenting, by the virtual assistant platform, at least a portion of the context information. ### Claim 14 (depends on 12) The method of claim 12, wherein the storing the at least a portion of the first audio input and the mapping between the action and at least the portion of the first audio input is performed in response to evaluating one or more characteristics of the first audio input; wherein the method further comprises: receiving, by the virtual assistant platform, a second audio input comprising a third request; based on the third request, scheduling a second action to be performed by the virtual assistant platform; evaluating one or more characteristics of the second audio input; and responsive to evaluating the one or more characteristics: refraining from storing any portion of the second audio input. ### Claim 15 (depends on 12) The method of claim 12, further comprising: determining a second confidence level that associated with scheduling the action to be performed based on the first request; responsive to the second confidence level being below a confidence threshold, prompting a user to confirm the action. ### Claim 16 (depends on 12) The method of claim 12, wherein storing at least the portion of the first audio input comprises storing all of the first audio input. ### Claim 17 (depends on 12) The method of claim 12, further comprising: parsing the first audio input to divide the first audio input into a plurality of sections; determining a particular section of the plurality of sections of the first audio input, that includes the first request; wherein storing at least the portion of the first audio input comprises storing the particular section. ### Claim 18 (depends on 12) The method of claim 12, further comprising: parsing the first audio input to divide the first audio input into a plurality of sections; generating, for each section of the plurality of sections of the first audio input, a transcript of audio content contained in the section; determining, for each section of the plurality of sections of the first audio input, a likelihood that the transcript of the audio content contained in the section includes any transcription error; wherein storing at least the portion of the first audio input comprises storing at least a particular section, among the plurality of sections, associated with the greatest likelihood that the transcript of the audio content contained in the section includes any transcription error. ### Claim 19 (depends on 12) The method of claim 12, further comprising: responsive to receiving the second request for audio playback of the first request corresponding to the action performed by the virtual assistant platform: marking at least the portion of the first audio input for inclusion in a training data set. ### Claim 20 A system comprising: at least one device including a hardware processor; the system being configured to perform operations comprising: receiving, by a virtual assistant platform, a first audio input comprising a first request; determining that at least a portion of the first audio input includes one or more phonemes associated with a first confidence level below a threshold confidence level; responsive to determining that at least the portion of the first audio input includes one or more phonemes associated with the first confidence level below the threshold confidence level: based on the first request, scheduling an action to be performed by the virtual assistant platform; storing (a) at least a portion of the first audio input and (b) a mapping between the action and at least the portion of the first audio input; performing, by the virtual assistant platform, the action; subsequent to performing the action: receiving, by the virtual assistant platform, a second request for audio playback of the first request corresponding to the action performed by the virtual assistant platform; retrieving at least the portion of the first audio input based on the mapping between the action and at least the portion of the first audio input; playing, by the virtual assistant platform, at least the portion of the first audio input comprising the first request. ### Claim 21 (depends on 20) The system of claim 20, the operations further comprising: storing (a) context information from a time the first request was received and (b) a mapping between the action and the context information, wherein the context information comprises one or more of (i) a user location when the first request was received, (ii) the time at which the first request was received, or (iii) a list of applications running at the time the first request was received; retrieving the context information based on the mapping between the action and the context information; and presenting, by the virtual assistant platform, at least a portion of the context information. ### Claim 22 (depends on 20) The system of claim 20, wherein the storing the at least a portion of the first audio input and the mapping between the action and at least the portion of the first audio input is performed in response to evaluating one or more characteristics of the first audio input; wherein the operations further comprise: receiving, by the virtual assistant platform, a second audio input comprising a third request; based on the third request, scheduling a second action to be performed by the virtual assistant platform; evaluating one or more characteristics of the second audio input; and based on evaluating the one or more characteristics: refraining from storing any portion of the second audio input.