# US 9,436,951 B1 · Facilitating presentation by mobile device of additional content for a word or phrase upon utterance thereof > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,436,951 B1 - Document ID: US9436951B1 - Status: Granted - Kind: B1 - Application number: 12/198,116 - Filed: 2008-08-25 - Granted: 2016-09-06 - Earliest claimed: 2007-08-22 - Assignees: Amazon Technologies, Inc. - Applicants: Victor Roman Jablokov; Igor Roditis Jablokov; James Richard Terrell, II; Scott Edward Paden - Inventors: Victor Roman Jablokov; Igor Roditis Jablokov; James Richard Terrell, II; Scott Edward Paden - Web page: https://patentgrep.com/patent/US9436951B1 ## CPC - G06Q 30/0269: {based on user profile or attribute} - G06F 16/632: Query formulation - G06F 16/683: using metadata automatically derived from the content - G06F 16/951: Indexing; Web crawling techniques - G06F 16/9538: Presentation of query results - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 21/00: Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility (G10L19/00 takes precedence) - H04L 51/216: Handling conversation history, e.g. grouping of messages in sessions or threads - H04W 4/12: Messaging; Mailboxes; Announcements - H04W 4/21: for social networking applications - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 16/9535: Search customisation based on user profiles and personalisation - G06F 16/957: Browsing optimisation, e.g. caching or content distillation - G06F 16/9577: {Optimising the visualization of content, e.g. distillation of HTML documents} - G06Q 10/107: Computer-aided management of electronic mailing [e-mailing] - G06Q 30/02: Marketing; Price estimation or determination; Fundraising - G06Q 30/0256: {User search} - G06Q 30/0271: {Personalized advertisement} - G06Q 30/0631: {Recommending goods or services} - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP] - H04L 69/329: in the application layer [OSI layer 7] - H04M 1/7243: with interactive means for internal management of messages - H04M 3/42382: {Text-based messaging services in telephone networks such as PSTN/ISDN, e.g. User-to-User Signalling or Short Message Service for fixed networks} - H04M 3/4878: {Advertisement messages} - H04M 3/4938: {comprising a voice browser which renders and interprets, e.g. VoiceXML} - H04M 7/0036: {where the data service is an information service} - H04M 2201/39: using speech synthesis - H04M 2201/40: using speech recognition - H04M 2203/357: Autocues for dialog assistance - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) ## Abstract A method for presenting additional content for a word that is part of a message, and that is presented by a mobile communication device, includes the steps of: presenting the message, including emphasizing one or more words for which respective additional content is available for presenting by the mobile communication device; receiving an utterance that includes an emphasized word for which additional content is available for presenting by the mobile communication device; and presenting the additional content for the emphasized word included in the utterance received by the mobile communication device. These steps are performed by the mobile communication device. ## Claims ### Claim 1 A computer-implemented method of presenting additional content for a term presented by a first mobile communication device, the computer-implemented method comprising: receiving, by the first mobile communication device, a first utterance; transmitting, by the first mobile communication device, a first identifier of the first mobile communication device and the first utterance to a server; receiving, by the first mobile communication device from the server, text representing a transcription of the first utterance; receiving, by the first mobile communication device from the server, an indicator that first additional content is available for a term identified within the text by the indicator, wherein the term is associated at the server with the first identifier of the first mobile communication device and a second identifier of a second mobile communication device, and wherein the first additional content for the term is associated with the first identifier and the second identifier; presenting, on the first mobile communication device, the text with an emphasis on the term identified by the indicator; after presenting the text on the first mobile communication device, receiving, by the first mobile communication device, a second utterance that includes the term; transmitting, by the first mobile communication device, the first identifier and the second utterance to the server; receiving, by the first mobile communication device from the server, in response to transmitting the second utterance and the first identifier, the first additional content; presenting, on the first mobile communication device the first additional content for the term; transmitting, by the first mobile communication device, the second identifier to the server, the server configured to send the text as well as the indicator that first additional content is available for the term to the second mobile communication device using the second identifier of the second mobile communication device; and receiving, by the first mobile communication device from the server, a message including a transcribed third utterance received by the second communication device in response to the text. ### Claim 2 (depends on 1) The method of claim 1, wherein the first mobile communication device is a phone. ### Claim 3 (depends on 1) The method of claim 1, wherein the first additional content is presented by playing audio over a speaker of the first mobile communication device. ### Claim 4 (depends on 1) The method of claim 1, wherein presenting the text on the first mobile communication device, including emphasizing the term comprises playing audio of the text on the first mobile communication device, wherein the audio includes pronunciation emphasis for the term. ### Claim 5 (depends on 1) The method of claim 1, wherein presenting the text on the first mobile communication device with an emphasis on the term identified by the indicator comprises playing audio of the text on the first mobile communication device, wherein the audio includes aural cueing for the term. ### Claim 6 (depends on 1) The method of claim 1, wherein presenting the text on the first mobile communication device with an emphasis on the term identified by the indicator comprises displaying the text on the first mobile communication device such that the term is highlighted. ### Claim 7 (depends on 6) The method of claim 6, wherein the first additional content for the term is presented by expanding the displayed text to include the respective additional content in conjunction with the term. ### Claim 8 (depends on 1) The method of claim 1, wherein the first additional content comprises at least one of a web address, an email address, or geospatial information. ### Claim 9 (depends on 1) The method of claim 1, wherein the first additional content is presented by displaying a video on the first mobile communication device. ### Claim 10 (depends on 1) The method of claim 1, wherein the first additional content is presented in a web browser of the first mobile communication device. ### Claim 11 (depends on 1) The method of claim 1, wherein the first additional content is presented in a popup window of the first mobile communication device. ### Claim 12 (depends on 1) The method of claim 1, wherein the term consists of a single word. ### Claim 13 (depends on 1) The method of claim 1, wherein the term comprises a phrase. ### Claim 14 (depends on 1) The method of claim 1, wherein the term comprises an alphanumeric string. ### Claim 15 (depends on 1) The method of claim 1, wherein the second utterance is transmitted with a cue indicating that the second utterance includes the term. ### Claim 16 (depends on 1) The method of claim 1, wherein multiple alternative first additional content exists for the term and are available for presentation by the first mobile communication device, and wherein the method further comprises presenting, on the first mobile communication device, the first additional content of each alternative first additional content of the multiple alternative first additional content. ### Claim 17 (depends on 1) The method of claim 1, further comprising receiving non-verbal input by the first mobile communication device, which input represents a selection of term that is emphasized; and presenting, on the first mobile communication device, the first additional content for the term of the selection, whereby the first additional content may be presented during times when the first mobile communication device is unable to successfully receive an utterance. ### Claim 18 (depends on 1) The method of claim 1, wherein multiple alternative first additional content exists for the term and are available for presentation by the first mobile communication device, and wherein the method further comprises selecting and presenting at least one alternative first additional content of the multiple alternative first additional content. ### Claim 19 (depends on 18) The method of claim 18, wherein selecting the at least one alternative first additional content of the multiple alternative first additional content is performed by a user of the first mobile communication device. ### Claim 20 (depends on 18) The method of claim 18, wherein selecting the at least one alternative first additional content of the multiple alternative first additional content is performed in accordance with a computer algorithm. ### Claim 21 (depends on 1) The method of claim 1, wherein receiving the text representing a transcription of the first utterance includes receiving a plurality of terms associated with the first identifier of the first mobile communication device. ### Claim 22 A non-transitory computer-readable medium whose contents configure one or more computing systems to perform a method of presenting additional content for a term presented by a first mobile communication device, the method comprising: receiving, by the first mobile communication device, a first utterance; transmitting a first identifier of the first mobile communication device and the first utterance from the first mobile communication device to a computing device; receiving, by the first mobile communication device from the computing device, text representing a transcription of the first utterance; receiving, by the first mobile communication device from the computing device, an indicator that first additional content is available for a term identified within the text by the indicator, wherein the term is associated at the computing device with the first identifier of the first mobile communication device and a second identifier of a second mobile communication device, and wherein the first additional content for the term is associated with the first identifier and the second identifier; presenting, on the first mobile communications device, the text with an emphasis on the term identified by the indicator; after presenting the text on the first mobile communication device, receiving, by the first mobile communication device, a second utterance that includes the term; transmitting, by the first mobile communication device, the first identifier and the second utterance to the computing device; receiving, by the first mobile communication device from the computing device, in response to transmitting the second utterance and the first identifier, the first additional content; presenting, on the first mobile communication device, the first additional content for the term; transmitting, by the first mobile communication device, the second identifier to the computing device, the computing device configured to send the text as well as the indicator that first additional content is available for the term to the second mobile communication device using the second identifier; and receiving, by the first mobile communication device from the computing device, a message including a transcribed third utterance received by the second communication device in response to the text. ### Claim 23 (depends on 22) The non-transitory computer-readable medium of claim 22, wherein the first mobile communication device is at least one of a phone, a PDS, a tablet notebook or a two-way pager. ### Claim 24 (depends on 22) The non-transitory computer-readable medium of claim 22, wherein presenting the text on the first mobile communication device with an emphasis on the term identified by the indicator comprises playing audio of the text on the first mobile communication device, wherein the audio includes one or more of pronunciation emphasis for the term or aural cueing for the term. ### Claim 25 (depends on 22) The non-transitory computer-readable medium of claim 22, wherein presenting the text on the first mobile communication device with an emphasis on the term identified by the indicator comprises displaying the text on the first mobile communication device such that the term is highlighted. ### Claim 26 (depends on 22) The non-transitory computer-readable medium of claim 22, wherein presenting the first additional content for the term on the first mobile communication device comprises one or more of playing audio over a speaker of the first mobile communication device, expanding the displayed text to include the respective first additional content in conjunction with the term, or displaying a video on the first mobile communication device. ### Claim 27 (depends on 22) The non-transitory computer-readable medium of claim 22, wherein the first additional content comprises one or more of a web address, an email address, or geospatial information. ### Claim 28 (depends on 22) The non-transitory computer-readable medium of claim 22, wherein the first additional content is presented in one or more of a web browser of the first mobile communication device or a popup window of the first mobile communication device. ### Claim 29 (depends on 22) The non-transitory computer-readable medium of claim 22, wherein the term comprises one of more of a single word, a phrase, or an alphanumeric string. ### Claim 30 (depends on 22) The non-transitory computer-readable medium of claim 22, wherein the second utterance is transmitted with a cue indicating that the second utterance includes an emphasized term. ### Claim 31 (depends on 22) The non-transitory computer-readable medium of claim 22, wherein multiple alternative first additional content exists for the term and are available for presentation by the first mobile communication device, and wherein the method further comprises presenting by the first mobile communication device the first additional content of each such alternative. ### Claim 32 (depends on 22) The non-transitory computer-readable medium of claim 22, wherein the method further comprises receiving non-verbal input by the first mobile communication device, which input represents a selection of an emphasized term; and presenting, by the first mobile communication device, the first additional content for the selected term, whereby the first additional content may be presented during times when the first mobile communication device is unable to successfully receive an utterance. ### Claim 33 (depends on 22) The non-transitory computer-readable medium of claim 22, wherein multiple alternative first additional content exists for the term and are available for presentation by the first mobile communication device, and wherein the method further comprises presenting, on the first mobile communication device, the multiple alternative first additional content. ### Claim 34 (depends on 33) The non-transitory computer-readable medium of claim 33, wherein selecting the first additional content of one of the alternatives is performed by a user of the first mobile communication device or in accordance with a computer algorithm. ### Claim 35 (depends on 22) The non-transitory computer-readable medium of claim 22, wherein receiving the text from the first computing device includes receiving a plurality of terms that have been selected by the first computing device from the transcription and associated with the identifier of the first mobile communication device. ### Claim 36 A system for presenting additional content for a term presented by a first mobile communication device, the system comprising one or more processors executing instructions configured to: receive, by the first mobile communication device, a first utterance; transmit, by the first mobile communication device, a first identifier of the first mobile communication device and the first utterance to a computing device; receive, by the first mobile communication device from the computing device, text representing a transcription of the first utterance; receive, by the first mobile communication device from the computing device, an indicator that first additional content is available for a term identified within the text by the indicator, wherein the term is associated at the computing device with the first identifier of the first mobile communication device and a second identifier of a second mobile communication device, and wherein the first additional content for the term is associated with the first identifier and the second identifier; present, on the first mobile communication device, the text with an emphasis on the term identified by the indicator; after presenting the text on the first mobile communication device, receive, by the first mobile communication device, a second utterance that includes the term; transmit, by the first mobile communication device, the first identifier and the second utterance to the computing device; receive, by the first mobile communication device from the computing device, in response to transmitting the second utterance and the first identifier, the first additional content; present, on the first mobile communication device, the first additional content for the term; transmit, by the first mobile communication device, the second identifier to the computing device, the computing device configured to send the text as well as the indicator that first additional content is available for the term to the second mobile communication device using the second identifier; and receive, by the first mobile communication device from the computing device, a message including a transcribed third utterance received by the second communication device in response to the text. ### Claim 37 (depends on 36) The system of claim 36, wherein the one or more processors executing the instructions are configured to present the text on the first mobile communication device by playing audio of the text on the first mobile communication device, wherein the audio includes one or more of pronunciation emphasis for the term or aural cueing for the term. ### Claim 38 (depends on 36) The system of claim 36, wherein the one or more processors executing the instructions are configured to present the text representing the transcription on the first mobile communication device by displaying the text representing the transcription on the first mobile communication device such that within the text representing the transcription the term is highlighted. ### Claim 39 (depends on 36) The system of claim 36, wherein the one or more processors executing the instructions are configured to present the first additional content for the term on the first mobile communication device by at least one of playing audio over a speaker of the first mobile communication device, expanding the presented text representing the transcription to include the respective first additional content in conjunction with the term, or displaying a video on the first mobile communication device. ### Claim 40 (depends on 36) The system of claim 36, wherein the first additional content comprises one or more of a web address, an email address, or geospatial information. ### Claim 41 (depends on 36) The system of claim 36, wherein multiple alternative first additional content exists for the term and are available for presentation by the first mobile communication device, and wherein the one or more processors executing the instructions are further configured to present, on the first mobile communication device, the multiple alternative first. ### Claim 42 (depends on 36) The system of claim 36, wherein the one or more processors executing the instructions are further configured to receive non-verbal input by the first mobile communication device, which input represents a selection of the term that is emphasized; and present, on the first mobile communication device, the first additional content for the term, whereby the first additional content may be presented during times when the first mobile communication device is unable to successfully receive an utterance. ### Claim 43 (depends on 36) The system of claim 36, wherein multiple alternative first additional content exists for the term and are available for presentation by the first mobile communication device, and wherein the one or more processors executing the instructions are configured to present at least one alternative first additional content of the multiple alternative first additional content. ### Claim 44 (depends on 43) The system of claim 43, wherein the one or more processors executing the instructions are configured to select the at least one alternative first additional content of the multiple alternative first additional content by input from a user of the first mobile communication device or in accordance with a computer algorithm. ### Claim 45 (depends on 36) The system of claim 36, wherein to receive the text representing a transcription of the first utterance, the one or more processors executing the instructions are configured to receive a plurality of terms associated with the identifier of the first mobile communication device.