# US 10,319,375 B2 · Audio message extraction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,319,375 B2 - Document ID: US10319375B2 - Status: Granted - Kind: B2 - Application number: 15/392,291 - Filed: 2016-12-28 - Published: 2018-06-28 - Granted: 2019-06-11 - Earliest claimed: 2016-12-28 - Assignees: Amazon Technologies, Inc. - Applicants: Amazon Technologies, Inc. - Inventors: Neil Christopher Fritz; Lakshya Bhagat; Scott Southwood; Katelyn Doran; Brett Lounsbury; Christo Frank Devaraj - Web page: https://patentgrep.com/patent/US10319375B2 ## CPC - G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning} - G06F 40/295: Named entity recognition - G10L 15/1822: {Parsing for meaning understanding} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications - H04M 7/0042: {where the data service is a text-based messaging service (H04M7/0054 takes precedence)} - H04W 4/12: Messaging; Mailboxes; Announcements - G10L 2015/088: {Word spotting} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) ## Abstract Audio data, corresponding to an utterance spoken by a person within a detection range of a voice communications device, can include an audio message portion. The audio data can be captured and analyzed to determine the intent to send a message. Based at least in part upon that intent, a remaining portion of the audio data can be analyzed to determine the intended message target or recipient, as well as the portion corresponding to the actual message payload. Once determined, the audio file can be trimmed to the message payload, and the message payload of the audio data can be delivered as an audio message to the target recipient. ## Claims ### Claim 1 A system, comprising: at least one device processor; memory including instructions that, when executed by the at least one device processor, cause the system to: receive audio input data from a voice communications device associated with an account, the audio input data corresponding to an utterance received by a microphone of the voice communications device, wherein a beginning of the utterance is identified by the voice communications device in response to a wakeword being detected by the voice communications device; generate text data from the audio input data by performing automated speech recognition (ASR) on the audio input data; determine, from the text data, a messaging intent by performing natural language processing (NLP) on the text data; determine a slot pattern corresponding to the messaging intent, the slot pattern including at least a target slot and a message payload slot; determine respective portions of the text data that correspond to the target slot and the message payload slot; identify, based upon the text data corresponding to the target recipient slot and a contact list associated with the voice communications device, a recipient identifier; determine a first timestamp associated with the message payload slot; generate, based upon the first timestamp, audio message data including a portion of the audio data corresponding to the text data of the message payload slot; and send the audio message data for playback on an audio playback device associated with the recipient identifier. ### Claim 2 (depends on 1) The system of claim 1, wherein the instructions, when executed further cause the system to: determine a second timestamp corresponding to an end location of the message payload slot; and include only a portion of the audio input data located between a time of the first timestamp and a time of the second timestamp in the audio message data. ### Claim 3 (depends on 1) The system of claim 1, wherein the instructions, when executed further cause the system to: determine message text corresponding to the audio message data; and transmit the message text for access via the recipient identifier, wherein the message text is able to be presented with, or separate from, the audio message data. ### Claim 4 (depends on 1) The system of claim 1, wherein the instructions, when executed further cause the system to: receive a request from the audio playback device to an address corresponding to the audio message data; and send the audio message data for playback on the audio playback device in response to the request. ### Claim 5 A computer-implemented method, comprising: receiving audio input data corresponding to an utterance received by at least one microphone of a voice communications device associated with an account, wherein a beginning of the utterance is identified by the voice communications device in response to a wakeword being detected by the voice communications device; determining a messaging intent represented by the audio input data; determining a slot pattern corresponding to the messaging intent, the slot pattern including at least a target slot and a message payload slot; determining, from the target slot, a recipient identifier represented by the audio input data; determining a message payload portion that corresponds to the message payload slot; determining a first time stamp identifying a beginning of the message payload portion; and generating, for playback on an audio playback device and accessible according to the recipient identifier, audio message data including the message payload portion starting from a location of the first time stamp in the audio input data. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, further comprising: determining a set of time stamps identifying locations of the target slot and the message payload slot, the set of time stamps including the first time stamp; and determining the recipient identifier and the message payload portion based upon the locations of the set of time stamps with respect to the audio input data. ### Claim 7 (depends on 5) The computer-implemented method of claim 5, further comprising: generating tokenized text data from the audio input data by performing automated speech recognition (ASR) on the audio input data. ### Claim 8 (depends on 7) The computer-implemented method of claim 7, further comprising: determining at least the messaging intent, and respective words corresponding to the target slot and the message payload slot, by performing natural language processing (NLP) on the tokenized text data. ### Claim 9 (depends on 5) The computer-implemented method of claim 5, further comprising: determining, for the audio input data, an identity of a user having spoken the utterance; and determining a contact list for the user based upon the identity; and determining the recipient identifier based upon performing a lookup of a target from the target slot against the contact list for with the user. ### Claim 10 (depends on 5) The computer-implemented method of claim 5, further comprising: receiving media input data including the audio input data and corresponding video input data; and extracting the audio input data for determining the messaging intent. ### Claim 11 (depends on 10) The computer-implemented method of claim 10, further comprising: including, for playback, a portion of the video input data corresponding to the message payload portion. ### Claim 12 (depends on 5) The computer-implemented method of claim 5, further comprising: determining that the recipient identifier is unable to be determined with at least a minimum level of confidence based on the audio input data; causing additional audio input data to be received that includes additional identifying information for a target of the messaging intent; and determining the recipient identifier based upon the additional identifying information. ### Claim 13 (depends on 5) The computer-implemented method of claim 5, further comprising: determining message text corresponding to the message payload slot; and providing the message text for access via the recipient identifier, wherein the message text is able to be presented with, or separate from, the audio message data. ### Claim 14 A system, comprising: at least one device processor; memory including instructions that, when executed by the at least one device processor, cause the system to: receive media input data corresponding to an utterance received by a communications device associated with an account, wherein a beginning of the utterance is identified by the communications device in response to a wakeword being detected by the communications device; extract audio input data from the media input data; determine a messaging intent represented by the audio input data; determine a slot pattern corresponding to the messaging intent, the slot pattern including at least a target slot and a message payload slot; determine a recipient identifier represented by the audio input data; determine a message payload portion that corresponds to the message payload slot; determine a first time stamp identifying a beginning of the message payload portion; and generate, for playback on a playback device and accessible according to the recipient identifier, media message data including the message payload portion starting from a location of the first time stamp in the audio input data. ### Claim 15 (depends on 14) The system of claim 14, wherein the instructions, when executed further cause the system to: determine a set of time stamps identifying locations of the target slot and the message payload slot in the audio input data; and determine the recipient identifier and the message payload portion based upon the locations of the set of time stamps with respect to the audio input data. ### Claim 16 (depends on 14) The system of claim 14, wherein the instructions, when executed further cause the system to: generate tokenized text data from the audio input data by performing automated speech recognition (ASR) on the audio input data; and determine at least the messaging intent, and respective words corresponding to the target slot and the message payload slot, by performing natural language processing (NLP) on the tokenized text data. ### Claim 17 (depends on 14) The system of claim 14, wherein the instructions, when executed further cause the system to: determine, for the media input data, an identity of a user having spoken the utterance; and determine a contact list for the user based upon the identity; and determine the recipient identifier based upon performing a lookup of a target from the target slot against the contact list for with the user. ### Claim 18 (depends on 14) The system of claim 14, wherein the instructions, when executed further cause the system to: determine that the recipient identifier is unable to be determined with at least a minimum level of confidence based on the audio input data; cause additional audio input data to be received that includes additional identifying information for a target of the messaging intent; and determine the recipient identifier based upon the additional identifying information. ### Claim 19 (depends on 14) The system of claim 14, wherein the instructions, when executed further cause the system to: extract video input data from the media input data; and include, for playback on the playback device, a portion of the video input data corresponding to the message payload portion.