# US 10,319,383 B1 · Customizing actions based on contextual data and voice-based inputs > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,319,383 B1 - Document ID: US10319383B1 - Status: Granted - Kind: B1 - Application number: 16/112,578 - Filed: 2018-08-24 - Granted: 2019-06-11 - Earliest claimed: 2013-03-01 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Zoltan Stekkelpak; Gyula Simonyi - Web page: https://patentgrep.com/patent/US10319383B1 ## CPC - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 15/07: to the speaker - G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning} - G10L 2015/223: {Execution procedure of a spoken command} - G10L 2015/228: {of application context} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Methods and systems are provided for customizing an action. In some implementations, voice input is received from a user and a context is determined from the voice input. Potential contextual data is identified based on the context and the voice input. A level of confidence is determined for an association of the potential contextual data and the context. An action is performed based on the voice input, the potential contextual data, and the level of confidence. The potential contextual data is used to customize the action. ## Claims ### Claim 1 A method comprising: receiving, at data processing hardware, a voice input command from a user to perform a specific action; processing, by the data processing hardware, the voice input command and the specific action to identify potential contextual data necessary to perform the specific action, the potential contextual data comprising a current event that is related to the user and currently occurring relative to a time when the voice input command was received; presenting, by the data processing hardware, a refining prompt prompting the user to provide a confirmation response that confirms the potential contextual data is associated with the specified action; in response to receiving the confirmation response from the user, supplementing the voice input command based on the potential contextual data; and performing, by the data processing hardware, the specific action based on the supplemented voice input command. ### Claim 2 (depends on 1) The method of claim 1, wherein processing the voice input command and the specific action to identify the potential contextual data comprises: processing the voice input command and the specific action to determine whether there is sufficient information to perform the specific action; determining one or more missing parameters from the voice input command when there is not sufficient information; and identifying the potential contextual data based on the one or more missing parameters, the voice input command, and the specific action. ### Claim 3 (depends on 2) The method of claim 2, wherein presenting the refining prompt comprises: determining one or more possible options for the one or more missing parameters based on data associated with the current event that is related to the user and currently occurring relative to the time when the voice input command was received; and presenting the refining prompt to allow the user to select one of the one or more possible options for the one or more missing parameters, wherein the confirmation response received from the user comprises the selected one of the one or more possible options. ### Claim 4 (depends on 3) The method of claim 3, wherein supplementing the voice input command comprises supplementing the voice input command using the selected one of the one or more possible options received in the confirmation response from the user. ### Claim 5 (depends on 2) The method of claim 2, wherein identifying the potential contextual data comprises: identifying trends in user data associated with the user over time; and determining information indicative of the one or more missing parameters based on the trends. ### Claim 6 (depends on 1) The method of claim 1, wherein the identified potential contextual data further comprises predictive preference data associated with the specific action and indicative of trends in user data associated with the user over time. ### Claim 7 (depends on 1) The method of claim 1, wherein the identified potential contextual data further comprises predictive preference data associated with the specific action and indicative of trends in aggregate data associated with a pool of other users, the pool of other users sharing similar characteristics as the user. ### Claim 8 (depends on 1) The method of claim 1, wherein presenting the refining prompt comprises presenting the refining prompt as an audio output from a speaker of a user device. ### Claim 9 (depends on 1) The method of claim 1, wherein presenting the refining prompt comprises displaying the refining prompt on a display of a user device. ### Claim 10 (depends on 1) The method of claim 1, wherein the specific action is associated with a plurality of parameters, each parameter specifying information necessary to perform the specific action. ### Claim 11 A system comprising: data processing hardware; and memory hardware in communication with the data processing and storing instructions that when executed by the data processing hardware cause the data processing hardware to perform operations comprising: receiving a voice input command from a user to perform a specific action; processing the voice input command and the specific action to identify potential contextual data necessary to perform the specific action, the potential contextual data comprising a current event that is related to the user and currently occurring relative to a time when the voice input command was received; presenting a refining prompt prompting the user to provide a confirmation response that confirms the potential contextual data is associated with the specified action; in response to receiving the confirmation response from the user, supplementing the voice input command based on the potential contextual data; and performing the specific action based on the supplemented voice input command. ### Claim 12 (depends on 11) The system of claim 11, wherein processing the voice input command and the specific action to identify the potential contextual data comprises: processing the voice input command and the specific action to determine whether there is sufficient information to perform the specific action; determining one or more missing parameters from the voice input command when there is not sufficient information; and identifying the potential contextual data based on the one or more missing parameters, the voice input command, and the specific action. ### Claim 13 (depends on 12) The system of claim 12, wherein presenting the refining prompt comprises: determining one or more possible options for the one or more missing parameters based on data associated with the current event that is related to the user and currently occurring relative to the time when the voice input command was received; and presenting the refining prompt to allow the user to select one of the one or more possible options for the one or more missing parameters, wherein the confirmation response received from the user comprises the selected one of the one or more possible options. ### Claim 14 (depends on 13) The system of claim 13, wherein supplementing the voice input command comprises supplementing the voice input command using the selected one of the one or more possible options received in the confirmation response from the user. ### Claim 15 (depends on 12) The system of claim 12, wherein identifying the potential contextual data comprises: identifying trends in user data associated with the user over time; and determining information indicative of the one or more missing parameters based on the trends. ### Claim 16 (depends on 11) The system of claim 11, wherein the identified potential contextual data further comprises predictive preference data associated with the specific action and indicative of trends in user data associated with the user over time. ### Claim 17 (depends on 11) The system of claim 11, wherein the identified potential contextual data further comprises predictive preference data associated with the specific action and indicative of trends in aggregate data associated with a pool of other users, the pool of other users sharing similar characteristics as the user. ### Claim 18 (depends on 11) The system of claim 11, wherein presenting the refining prompt comprises presenting the refining prompt as an audio output from a speaker of a user device. ### Claim 19 (depends on 11) The system of claim 11, wherein presenting the refining prompt comprises displaying the refining prompt on a display of a user device. ### Claim 20 (depends on 11) The system of claim 11, wherein the specific action is associated with a plurality of parameters, each parameter specifying information necessary to perform the specific action.