# US 2017/0256258 A1 · Speech Recognition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2017/0256258 A1 - Document ID: US20170256258A1 - Status: Published application - Kind: A1 - Application number: 15/229,854 - Filed: 2016-08-05 - Published: 2017-09-07 - Earliest claimed: 2016-03-01 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Raymond J. Froelich - Web page: https://patentgrep.com/patent/US20170256258A1 ## CPC - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit - G10L 15/063: {Training} - G10L 15/183: using context dependencies, e.g. language models - G10L 25/87: Detection of discrete points within a voice signal - H04N 7/157: {defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827)} - G10L 15/19: Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules - G10L 15/222: {Barge in, i.e. overridable guidance for interrupting prompts} - G10L 2015/0635: {updating or merging of old and new templates; Mean values; Weighting} - G10L 2015/088: {Word spotting} - G10L 2015/223: {Execution procedure of a spoken command} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Voice input is received from a user. An ASR system generates in memory a set of words it has identified in the voice input, and update the set each time it identifies a new word in the voice input to add the new word to the set, during at least one interval of speech activity. Information is pre-retrieved whilst the speech activity interval is still ongoing, for conveying in a response to be outputted at the end of the speech activity interval. ## Claims ### Claim 1 A computer system comprising: an input configured to receive voice input from a user; an ASR system for identifying individual words in the voice input, wherein the ASR system is configured to generate in memory during at least one interval of speech activity in the voice input a set of one or more words it has identified in the voice input, and update the set in the memory each time it identifies a new word in the voice input to add the new word to the set; a lookup module configured to retrieve during the interval of speech activity in the voice input at least one word from the set in the memory at a first time whilst the speech activity interval is still ongoing, and perform a lookup whilst the speech activity interval is still ongoing to pre-retrieve information associated with the at least one word; and a response generation module configured to detect the end of the speech activity interval at a later time, the set having been updated by the ASR system at least once between the first time and the later time, and to generate based thereon a response for outputting to the user, wherein the response conveys the information pre-retrieved by the lookup module. ### Claim 2 (depends on 1) A computer system according to claim 1, wherein the response is generated based on a confidence value assigned to at least one of the individual words by the ASR system and/or a confidence value assigned to the set of words by a language model of the computer system. ### Claim 3 (depends on 1) A computer system according to claim 1, wherein the information is pre-retrieved from an Internet search engine. ### Claim 4 (depends on 1) A computer system according to claim 1, wherein the user is a user of a communication system and the pre-retrieved information is information about another user in the communication system. ### Claim 5 (depends on 3) A computer system according to claim 3, wherein the pre-retrieved information comprises contact details for the other user. ### Claim 6 (depends on 1) A computer system according to claim 1, comprising a speech detection module configured monitor the set of identified words in the memory as it is updated by the ASR system, wherein the response generation module is configured to detect the end of the speech activity interval based on said monitoring of the identified set of words. ### Claim 7 (depends on 6) A computer system according to claim 6, further comprising a language model, wherein detecting the end of the speech activity interval comprises detecting, by the speech detection module, when the set of identified words forms a grammatically complete sentence according to the language model. ### Claim 8 (depends on 6) A computer system according to claim 6, wherein detecting the end of the speech activity interval comprises detecting, by the speech detection module, that no new words have been identified by the ASR system for a pre-determined duration. ### Claim 9 (depends on 1) A computer system according to claim 1, wherein the response is an audio response for playing out to the user in audible form. ### Claim 10 (depends on 1) A computer system according to claim 1, comprising a video generation module configured to generate and output to the user a moving image of an avatar, which conveys when the outputting of the response is about to begin by performing a visual action. ### Claim 11 (depends on 6) A computer system according to claim 6, further comprising: a speech detection module; a response delivery module configured to begin outputting the audio response to the user when the speech activity interval has ended, wherein the outputting of the audio response is terminated before it has completed in response to the speech detection module detecting the start of a subsequent speech activity interval in the voice input. ### Claim 12 (depends on 11) A computer system according to claim 11, wherein the speech detection module is configured to detect the start of subsequent speech activity interval by detecting an identification of another word in the voice input by the ASR system, the speech activity interval commencing with the detection of the other word. ### Claim 13 A computer-implemented method of effecting communication between a user and an artificial intelligence software agent executed on a computer, the method comprising: receiving at an ASR system voice input from the user; identifying by the ASR system individual words in the voice input, wherein the ASR system generates in memory during at least one interval of speech activity in the voice input a set of one or more words it has identified in the voice input, and updates the set in the memory each time it identifies a new word in the voice input to add the new word to the set; retrieving during the interval of speech activity in the voice input at least one word from the set in the memory at a first time whilst the speech activity interval is still ongoing; performing a lookup whilst the speech activity interval is still ongoing to pre-retrieve information associated with the at least one word; detecting the end of the speech activity interval at a later time, the set having been updated by the ASR system at least once between the first time and the later time; and generating based thereon a response for outputting to the user, by the software agent, wherein the response conveys the information pre-retrieved by the lookup module. ### Claim 14 (depends on 13) A method according to claim 13, wherein the voice input is received from a user device via a communication network, wherein the outputting step is performed by the software agent transmitting the response to the user device via the network so as to cause the user device to output the response to the user. ### Claim 15 (depends on 14) A method according to claim 14, wherein the voice input is received from and the response outputted to the user in real-time, thereby effecting a real-time communication event between the user and the software agent via the network. ### Claim 16 (depends on 13) A method according to claim 13, the method being implemented in a communication system, wherein the communication system comprises a user account database storing, for each of a plurality of users of the communication system, a user identifier that uniquely identifies that user within the communication system; wherein a user identifier of the software agent is also stored in the user account database so that the software agents appears as another user of the communication system. ### Claim 17 (depends on 13) A method according to claim 13, comprising monitoring the set of identified words in the memory as it is updated by the ASR system, the end of the speech activity interval being detected based on said monitoring of the identified set of words. ### Claim 18 (depends on 13) A method according to claim 13, wherein detecting the end of the speech activity interval comprises detecting, by the speech detection module, when the set of identified words forms a grammatically complete sentence according to a language model. ### Claim 19 A computer program product comprising an artificial intelligence software agent stored on a computer readable storage medium, the software agent for communicating with a user based on the output of an ASR system, the ASR system for receiving voice input from the user and identifying individual words in the voice input during at least one interval of speech activity in the voice input, the software agent being configured when executed to perform operations of: retrieving during the interval of speech activity in the voice input at least one word from the set at a first time whilst the speech activity interval is still ongoing; performing a lookup whilst the speech activity interval is still ongoing to pre-retrieve information associated with the at least one word; detecting the end of the speech activity interval at a later time, the set having been updated by the ASR system at least once between the first time and the later time; and generating based thereon a response for outputting to the user, by the software agent, wherein the response conveys the information pre-retrieved by the lookup module. ### Claim 20 (depends on 19) A computer program product according to claim 19, wherein the response is outputted to the user by transmitting it to a user device available to the user via a network so as to cause the user device to output the response to the user.