# US 11,908,452 B1 · Alternative input representations for speech inputs > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,452 B1 - Document ID: US11908452B1 - Status: Granted - Kind: B1 - Application number: 17/325,523 - Filed: 2021-05-20 - Granted: 2024-02-20 - Earliest claimed: 2021-05-20 - Assignees: Amazon Technologies, Inc. - Applicants: Amazon Technologies, Inc. - Inventors: Sixing Lu; Chengyuan Ma; Chenlei Guo; Fangfu Li - Web page: https://patentgrep.com/patent/US11908452B1 ## CPC - G10L 15/01: Assessment or evaluation of speech recognition systems - G06F 40/30: Semantic analysis - G10L 15/005: {Language recognition} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 2015/221: {Announcement of recognition results} - G10L 2015/223: {Execution procedure of a spoken command} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Techniques for presenting an alternative input representation to a user for testing and collecting processing data are described. A system may determine that a received spoken input triggers an alternative input representation for presenting. The system may output data corresponding to the alternative input representation in response to the received spoken input, and the system may receive user feedback from the user. The system may store the user feedback and processing data corresponding to processing of the alternative input representation, which may be later used to update an alternative input component configured to determine alternative input representations for spoken inputs. ## Claims ### Claim 1 A computer-implemented method comprising: receiving, from a device, first audio data representing a first spoken natural language input; determining that the first spoken natural language input corresponds to a first alternative input representation for which processing data is to be collected; determining a first number of times the first alternative input representation is to be tested; determining a counter value for the first alternative input representation, the counter value representing a second number of times the first alternative input representation has been presented for collecting processing data; determining the first number of times is different than the second number of times; in response to the first number of times being different than the second number of times, generating first output data requesting confirmation to perform an action using the first alternative input representation; sending the first output data to the device; receiving, from the device, second audio data representing user feedback corresponding to the first output data; and storing the first alternative input representation and the user feedback for updating an alternative input component configured to determine alternative input representations for spoken natural language inputs. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising: determining natural language understanding (NLU) data corresponding to the first spoken natural language input; determining a domain of the first alternative input representation; determining that the NLU data satisfies the domain; and wherein generating the first output data is further in response to determining that the NLU data satisfies the domain. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising: based on sending the first output data to the device, generating an updated counter value for the first alternative input representation; determining that the updated counter value is equal to the first number of times; and based on determining that the updated counter value is equal to the first number of times, setting a testing state, associated with the first alternative input representation, to indicate testing of the first alternative input representation has ended. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, further comprising: receiving a plurality of alternative input representations for which processing data is to be collected; determining a second alternative input representation, from the plurality of alternative input representations, corresponds to the first spoken natural language input; determining that a testing period for the second alternative input representation has expired; and based on determining that the testing period has expired, setting a testing state, associated with the second alternative input representation, to indicate testing of the second alternative input representation has ended. ### Claim 5 A computer-implemented method comprising: receiving first audio data representing a spoken natural language input; determining, based at least in part on the spoken natural language input, a first alternative input representation is to be presented to a user for collecting processing data corresponding to the first alternative input representation; determining a counter value representing a number of times the first alternative input representation has been presented for collecting processing data; determining the counter value is less than a threshold value; generating first output data using the first alternative input representation; causing presentation of the first output data; receiving user feedback corresponding to the first output data; and storing the first alternative input representation and the user feedback for updating an alternative input component. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, further comprising: determining that a time period for the first alternative input representation is unexpired; and generating the first output data based on the time period being unexpired. ### Claim 7 (depends on 5) The computer-implemented method of claim 5, further comprising: based on causing presentation of the first output data, generating an updated counter value representing the number of times the first alternative input representation has been presented for collecting processing data. ### Claim 8 (depends on 5) The computer-implemented method of claim 5, further comprising: determining natural language understanding (NLU) data corresponding to the first alternative input representation; determining, using the NLU data, second output data responsive to the first alternative input representation; and storing the NLU data and the second output data for later use in updating the alternative input component. ### Claim 9 (depends on 5) The computer-implemented method of claim 5, further comprising: identifying criteria associated with the first alternative input representation, the criteria including at least one user profile feature; determining profile data corresponding to the first audio data; determining that the at least one user profile feature is represented in the profile data; and determining the first alternative input representation is to be presented based at least in part on the at least one user profile feature being represented in the profile data. ### Claim 10 (depends on 5) The computer-implemented method of claim 5, further comprising: identifying criteria associated with the first alternative input representation, the criteria including at least a device type; determining a device associated with the first audio data corresponds to the device type; and determining the first alternative input representation is to be presented based at least in part on the device corresponding to the device type. ### Claim 11 (depends on 5) The computer-implemented method of claim 5, further comprising: receiving a plurality of alternative input representations for presenting; determining a second alternative input representation, from the plurality of alternative input representations, corresponding to the spoken natural language input; determining that a time period for presenting the second alternative input representation has expired; and setting data, associated with the second alternative input representation, to indicate that presentation of the second alternative input representation for collecting processing data corresponding to the second alternative input representation has been completed. ### Claim 12 (depends on 5) The computer-implemented method of claim 5, further comprising: after causing presentation of the first output data, determining that the counter value meets or exceeds the threshold value; and setting data, associated with the first alternative input representation, to indicate that presentation of the first alternative input representation for collecting processing data corresponding to the first alternative input representation has been completed. ### Claim 13 A system comprising: at least one processor; and at least one memory including instructions that, when executed by the at least one processor, cause the system to: receive first audio data representing a spoken natural language input; determine, based at least in part on the spoken natural language input, a first alternative input representation is to be presented to a user for collecting processing data corresponding to the first alternative input representation; determine a counter value representing a number of times the first alternative input representation has been presented for collecting processing data; determine the counter value is less than a threshold value; generate first output data using the first alternative input representation; cause presentation of the first output data; receive user feedback corresponding to the first output data; and store the first alternative input representation and the user feedback for updating an alternative input component. ### Claim 14 (depends on 13) The system of claim 13, wherein the at least one memory includes further instructions that, when executed by the at least one processor, further cause the system to: determine that a time period for the first alternative input representation is unexpired; and generate the first output data based on the time period being unexpired. ### Claim 15 (depends on 13) The system of claim 13, wherein the at least one memory includes further instructions that, when executed by the at least one processor, further cause the system to: based on causing presentation of the first output data, generating an updated counter value representing the number of times the first alternative input representation has been presented for collecting processing data. ### Claim 16 (depends on 13) The system of claim 13, wherein the at least one memory includes further instructions that, when executed by the at least one processor, further cause the system to: determine natural language understanding (NLU) data corresponding to the first alternative input representation; determine, using the NLU data, second output data responsive to the first alternative input representation; and store the NLU data and the second output data for later use in updating the alternative input component. ### Claim 17 (depends on 13) The system of claim 13, wherein the at least one memory includes further instructions that, when executed by the at least one processor, further cause the system to: identify criteria associated with the first alternative input representation, the criteria including at least one user profile feature; determine profile data corresponding to the first audio data; determine that the at least one user profile feature is represented in the profile data; and determine the first alternative input representation is to be presented based at least in part on the at least one user profile feature being represented in the profile data. ### Claim 18 (depends on 13) The system of claim 13, wherein the at least one memory includes further instructions that, when executed by the at least one processor, further cause the system to: identify criteria associated with the first alternative input representation, the criteria including at least a device type; determine a device associated with the first audio data corresponds to the device type; and determine the first alternative input representation is to be presented based at least in part on the device corresponding to the device type. ### Claim 19 (depends on 13) The system of claim 13, wherein the at least one memory includes further instructions that, when executed by the at least one processor, further cause the system to: receive a plurality of alternative input representations for presenting; determine a second alternative input representation, from the plurality of alternative input representations, corresponding to the spoken natural language input; determine that a time period for presenting the second alternative input representation has expired; and set data, associated with the second alternative input representation, to indicate that presentation of the second alternative input representation for collecting processing data corresponding to the second alternative input representation has been completed. ### Claim 20 (depends on 13) The system of claim 13, wherein the at least one memory includes further instructions that, when executed by the at least one processor, further cause the system to: after causing presentation of the first output data, determine that the counter value meets or exceeds the threshold value; and set data, associated with the first alternative input representation, to indicate that presentation of the first alternative input representation for collecting processing data corresponding to the first alternative input representation has been completed.