# US 12,118,981 B2 · Determining multilingual content in responses to a query > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,981 B2 - Document ID: US12118981B2 - Status: Granted - Kind: B2 - Application number: 17/475,897 - Filed: 2021-09-15 - Published: 2023-03-16 - Granted: 2024-10-15 - Earliest claimed: 2021-09-15 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Wangqing Yuan; Bryan Christopher Horling; David Kogan - Web page: https://patentgrep.com/patent/US12118981B2 ## CPC - G10L 13/086: {Detection of language} - G06F 40/35: Discourse or dialogue representation - G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation - G09B 5/065: {Combinations of audio and video presentations, e.g. videotapes, videodiscs, television systems} - G09B 19/06: Foreign languages (with audible presentation of material to be studied G09B5/04) - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 2015/223: {Execution procedure of a spoken command} - G10L 2015/225: {Feedback of the input speech} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) ## Abstract Implementations relate to determining multilingual content to render at an interface in response to a user submitted query. Those implementations further relate to determining a first language response and a second language response to a query that is submitted to an automated assistant. Some of those implementations relate to determining multilingual content that includes a response to the query in both the first and second languages. Other implementations relate to determining multilingual content that includes a query suggestion in the first language and a query suggestion in a second language. Some of those implementations relate to pre-fetching results for the query suggestions prior to rendering the multilingual content. ## Claims ### Claim 1 A method implemented by one or more processors, the method comprising: receiving audio data that captures a spoken query of a user that is in a first language, wherein the spoken query is provided via an automated assistant interface of a client device, and wherein the first language is specified as a primary language for the user; generating, based on processing the audio data, a first language response to the spoken query, wherein the first language response is in the first language; generating a second language response, to the spoken query, that is in the second language, wherein the second language is specified as a secondary language of interest to the user; determining, based on verification data provided with or derived from the audio data, to render multilingual content in response to the spoken query, the multilingual content including the first language response and the second language response; in response to determining to render the multilingual content: causing the multilingual content to be rendered at the assistant interface of the client device and in response to the spoken query; determining a query suggestion, wherein determining the query suggestion is based on the spoken query, the first language response, and/or the second language response; in response to determining to render the multilingual content: causing a first language version of the query suggestion and a second language version of the query suggestion to be rendered at the assistant interface of the client device in response to the spoken query; prior to receiving any selection of the first language version of the query suggestion or the second language version of the query suggestion: generating a first response to the query suggestion in the first language; generating a second response to the query suggestion in the second language; and causing the first response and the second response to be cached; receiving a selection of the first language version of the query suggestion; and in response to receiving the selection of first language version of the query suggestion: causing the cached first response to the query suggestion to be audibly rendered; and causing the cached second response to the query suggestion to be audibly rendered subsequent to causing the cached first response to the query suggestion to be audibly rendered. ### Claim 2 (depends on 1) The method of claim 1, further comprising: causing one or more actions to be performed in response to receiving the audio data. ### Claim 3 (depends on 1) The method of claim 1, wherein causing the multilingual content to be rendered at the assistant interface of the client device comprises: causing the first language response to be audibly rendered as first synthesized speech output and then causing the second language response to be audibly rendered as second synthesized speech output. ### Claim 4 (depends on 1) The method of claim 1, further comprising: generating a second language query by translating first language recognized text, of the spoken query, to the second language; and in response to determining to render the multilingual content: causing the second language query to be rendered at the assistant interface of the client device and in response to the spoken query. ### Claim 5 (depends on 4) The method of claim 4, further comprising: causing the second language query to be visually rendered with a selectable audible rendering interface element, wherein the audible rendering interface element, when selected, causes the second language query to be audibly rendered as synthesized speech output. ### Claim 6 (depends on 1) The method of claim 1, wherein generating the second language response is performed in response to determining to render the multilingual content. ### Claim 7 (depends on 1) The method of claim 1, further comprising: receiving a selection of the second language version of the query suggestion; and in response to receiving the selection of second language version of the query suggestion: causing the cached second response to the query suggestion to be audibly rendered; and causing the cached first response to the query suggestion to be audibly rendered subsequent to causing the cached second response to the query suggestion to be audibly rendered. ### Claim 8 (depends on 1) The method of claim 1, further comprising: determining a user proficiency measure that is specific to the user and that is specific to the second language, wherein determining to render the multilingual content is further based on the user proficiency measure. ### Claim 9 (depends on 8) The method of claim 8, further comprising: determining a complexity measure of the second language response, wherein determining to render the multilingual content based on the user proficiency measure comprises: determining to render the multilingual content based on comparing the user proficiency measure to the complexity measure of the second language response. ### Claim 10 (depends on 9) The method of claim 9, wherein determining the complexity measure comprises: determining, based on the terms of the second language response, a comprehension level for the second language response, wherein the comprehension level is indicative of level of skill in the second language that is sufficient to comprehend the second language response. ### Claim 11 (depends on 8) The method of claim 8, wherein determining to render the multilingual content based on the user proficiency measure comprises determining that the user proficiency measure satisfies a threshold. ### Claim 12 (depends on 1) The method of claim 1, further comprising: determining a user interest measure indicative of user interest in being provided with content in the second language, wherein determining to render the multilingual content is further based on the user interest measure. ### Claim 13 (depends on 1) The method of claim 1, wherein the verification data includes an identifier of the user that provided the audio data, wherein the user has previously indicated an interest in being provided multilingual content. ### Claim 14 A method implemented by one or more processors, the method comprising: receiving audio data that captures a spoken query of a user that is in a first language, wherein the spoken query is provided via an automated assistant interface of a client device, and wherein the first language is specified as a primary language for the user; generating, based on processing the audio data, a first language response to the spoken query, wherein the first language response is in the first language; generating a second language response, to the spoken query, that is in the second language, wherein the second language is specified as a secondary language of interest to the user; determining, based on verification data provided with or derived from the audio data, to render multilingual content in response to the spoken query, the multilingual content including the first language response and the second language response; in response to determining to render the multilingual content: causing the multilingual content to be rendered at the assistant interface of the client device and in response to the spoken query; determining a query suggestion, wherein determining the query suggestion is based on the spoken query, the first language response, and/or the second language response; in response to determining to render the multilingual content: causing a first language version of the query suggestion and a second language version of the query suggestion to be rendered at the assistant interface of the client device in response to the spoken query; prior to receiving any selection of the first language version of the query suggestion or the second language version of the query suggestion: generating a first response to the query suggestion in the first language; generating a second response to the query suggestion in the second language; and causing the first response and the second response to be cached; receiving a selection of the second language version of the query suggestion; and in response to receiving the selection of second language version: causing the cached second response to the query suggestion to be audibly rendered; and causing the cached first response to the query suggestion to be audibly rendered subsequent to causing the cached second response to the query suggestion to be audibly rendered. ### Claim 15 (depends on 14) The method of claim 14, further comprising: causing one or more actions to be performed in response to receiving the audio data. ### Claim 16 (depends on 14) The method of claim 14, wherein causing the multilingual content to be rendered at the assistant interface of the client device comprises: causing the first language response to be audibly rendered as first synthesized speech output and then causing the second language response to be audibly rendered as second synthesized speech output. ### Claim 17 A method implemented by one or more processors, the method comprising: receiving audio data that captures a spoken query of a user that is in a first language, wherein the spoken query is provided via an automated assistant interface of a client device, and wherein the first language is specified as a primary language for the user; generating, based on processing the audio data, a first language response to the spoken query, wherein the first language response is in the first language; generating a second language response, to the spoken query, that is in the second language, wherein the second language is specified as a secondary language of interest to the user; determining, based on verification data provided with or derived from the audio data, to render multilingual content in response to the spoken query, the multilingual content including the first language response and the second language response; in response to determining to render the multilingual content: causing the multilingual content to be rendered at the assistant interface of the client device and in response to the spoken query; determining a query suggestion, wherein determining the query suggestion is based on the spoken query, the first language response, and/or the second language response; prior to receiving any selection of a first language version of the query suggestion or a second language version of the query suggestion: generating a first response to the query suggestion in the first language; generating a second response to the query suggestion in the second language; and causing the first response and the second response to be cached; and in response to determining to render the multilingual content: causing the first language version of the query suggestion and the second language version of the query suggestion to be rendered at the assistant interface of the client device in response to the spoken query; wherein selection of the first language version of the query suggestion causes the cached first response to the query suggestion to be audibly rendered, and causes the cached second response to the query suggestion to be audibly rendered subsequent to causing the cached first response to the query suggestion to be audibly rendered, and wherein selection of the cached second version of the query suggestion causes the cached second response to the query suggestion to be audibly rendered, and causes the cached first response to the query suggestion to be audibly rendered subsequent to causing the cached second response to the query suggestion to be audibly rendered. ### Claim 18 (depends on 17) The method of claim 17, further comprising: causing one or more actions to be performed in response to receiving the audio data. ### Claim 19 (depends on 17) The method of claim 17, wherein causing the multilingual content to be rendered at the assistant interface of the client device comprises: causing the first language response to be audibly rendered as first synthesized speech output and then causing the second language response to be audibly rendered as second synthesized speech output.