# US 11,907,276 B2 · Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,276 B2 - Document ID: US11907276B2 - Status: Granted - Kind: B2 - Application number: 17/964,343 - Filed: 2022-10-12 - Published: 2023-02-02 - Granted: 2024-02-20 - Earliest claimed: 2016-05-17 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Maryam Garrett; Wan Fen Nicole Quah; Bryan Horling; Ruijie He - Web page: https://patentgrep.com/patent/US11907276B2 ## CPC - G06F 16/338: Presentation of query results - G06F 16/3344: {using natural language analysis} - G06F 16/907: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually - G06F 16/909: using geographical or spatial information, e.g. location (spatiotemporally dependent retrieval from the web G06F16/9537) - G06F 40/103: Formatting, i.e. changing of presentation of documents (automatic justification G06F40/189; automatic line break hyphenation G06F40/191) - G06F 40/169: Annotation, e.g. comment data or footnotes - G06F 40/30: Semantic analysis - H04W 4/02: Services making use of location information - H04W 4/029: Location-based management or tracking services - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) ## Abstract Some implementations are directed to generating a personal database entry for a user based on free-form natural language input formulated by the user via one or more user interface input devices of a computing device of the user. The generated personal database entry may include one or more terms of the natural language input and descriptive metadata determined based on one or more terms of the natural language input and/or based on contextual features associated with receiving the natural language input. Some implementations are directed to generating, based on one or more personal database entries of a user, output that is responsive to further free-form natural language input of the user. For example, one or more entries that are responsive to further natural language input of the user can be identified based on matching content of those entries to one or more search parameters determined based on the further input. ## Claims ### Claim 1 A method performed by one or more processors, comprising: receiving first natural language input, wherein the first natural language input is free-form input formulated by a user via a user interface input device of a computing device of the user; generating an entry for the first natural language input in a personal database of the user stored in one or more computer readable media, wherein generating the entry comprises: generating descriptive metadata for the entry, wherein generating the descriptive metadata for the entry comprises: generating a semantic label associated with an object based on one or more terms of the first natural language input, and generating an additional semantic label associated with a location based on one or more of the terms of the first natural language input; and storing, in the entry, the descriptive metadata; subsequent to receiving the first natural language input, receiving second natural language input, wherein the second natural language input is free-form input formulated by the user via the user interface input device or an additional user interface input device of an additional computing device of the user, and wherein one or more additional terms of the second natural language input include at least the semantic label associated with the object; determining, based on the second natural language input, at least one search parameter, wherein the at least one search parameter includes the object; searching the personal database based on the at least one search parameter; determining, based on the searching, that the entry is responsive to the second natural language input, wherein determining that the entry is responsive to the second natural language input is based at least in part on matching the at least one search parameter to at least some of the descriptive metadata; and in response to determining that the entry is responsive to the second natural language input: generating natural language output that includes one or more natural language output terms that are based on the entry; and providing, for audible presentation to the user via a user interface output device of the computing device, the natural language output. ### Claim 2 (depends on 1) The method of claim 1, wherein generating the semantic label associated with the object based on one or more of the terms of the first natural language input comprises: processing the first natural language input to classify one or more of the terms of the first natural language input; and generating the semantic label associated with an object based on a classification of one or more of the terms of the first natural language input. ### Claim 3 (depends on 2) The method of claim 2, wherein the semantic label associated with the object includes a reference to the object. ### Claim 4 (depends on 2) The method of claim 2, wherein generating the additional semantic label associated with the location based on one or more of the terms of the first natural language input comprises: generating the additional semantic label associated with the location based on the classification of one or more of the terms of the first natural language input. ### Claim 5 (depends on 1) The method of claim 1, wherein the first natural language input is received as part of a dialog session with an automated personal assistant, wherein the second natural language input is received as part of a subsequent dialog session with the automated personal assistant, and wherein the subsequent dialog session is subsequent to the dialog session. ### Claim 6 (depends on 1) The method of claim 1, further comprising: in response to determining that the entry is responsive to the second natural language input: providing, for visual presentation to the user via an additional user interface output device of the computing device, a representation of the natural language output. ### Claim 7 (depends on 6) The method of claim 6, wherein the representation of the natural language output includes a transcription corresponding to the natural language output. ### Claim 8 (depends on 1) The method of claim 1, further comprising: determining, based on the searching, that an additional entry is responsive to the second natural language input, wherein determining that the additional entry is responsive to the second natural language input is based at least in part on matching the at least one search parameter to at least some of the descriptive metadata. ### Claim 9 (depends on 8) The method of claim 8, wherein generating the natural language output further comprises including one or more additional natural language output terms that are based on the additional entry. ### Claim 10 (depends on 8) The method of claim 8, wherein generating the descriptive metadata for the entry further comprises generating temporal metadata that indicates one or more of: a day that the first natural language input was received, or a time that the first natural language input was received. ### Claim 11 (depends on 10) The method of claim 10, further comprising: in response to determining that both the entry and the additional entry are responsive to the second natural language input: selecting the entry as responsive to the second natural language input based on the temporal metadata. ### Claim 12 (depends on 11) The method of claim 11, wherein selecting the entry as responsive to the second natural language input based on the temporal metadata comprises: determining a current day or a current time that the second natural language input is received conforms with one or more of: the day that the first natural language input was received, or the time that the first natural language input was received. ### Claim 13 A system comprising: at least one processor; and at least one memory storing instructions that, when executed, cause the at least one processor to: receive first natural language input, wherein the first natural language input is free-form input formulated by a user via a user interface input device of a computing device of the user; generate an entry for the first natural language input in a personal database of the user stored in one or more computer readable media, wherein the instructions to generate the entry cause the at least one processor to: generate descriptive metadata for the entry, wherein the instructions to generate the descriptive metadata for the entry cause the at least one processor to: generate a semantic label associated with an object based on one or more terms of the first natural language input, and generate an additional semantic label associated with a location based on one or more of the terms of the first natural language input; and store, in the entry, the descriptive metadata; subsequent to receiving the first natural language input, receive second natural language input, wherein the second natural language input is free-form input formulated by the user via the user interface input device or an additional user interface input device of an additional computing device of the user, and wherein one or more additional terms of the second natural language input include at least the semantic label associated with the object; determine, based on the second natural language input, at least one search parameter, wherein the at least one search parameter includes the object; search the personal database based on the at least one search parameter; determine, based on the searching, that the entry is responsive to the second natural language input, wherein determining that the entry is responsive to the second natural language input is based at least in part on matching the at least one search parameter to at least some of the descriptive metadata; and in response to determining that the entry is responsive to the second natural language input: generate natural language output that includes one or more natural language output terms that are based on the entry; and provide, for audible presentation to the user via a user interface output device of the computing device, the natural language output. ### Claim 14 (depends on 13) The system of claim 13, wherein the instructions to generate the semantic label associated with the object based on one or more of the terms of the first natural language input comprise instructions to: process the first natural language input to classify one or more of the terms of the first natural language input; and generate the semantic label associated with an object based on a classification of one or more of the terms of the first natural language input. ### Claim 15 (depends on 14) The system of claim 14, wherein the semantic label associated with the object includes a reference to the object. ### Claim 16 (depends on 14) The system of claim 14, wherein the instructions to generate the additional semantic label associated with the location based on one or more of the terms of the first natural language input comprise instructions to: generate the additional semantic label associated with the location based on the classification of one or more of the terms of the first natural language input. ### Claim 17 A non-transitory computer readable storage medium storing instructions that, when executed, cause at least one processor to perform operations, the operations comprising: receiving first natural language input, wherein the first natural language input is free-form input formulated by a user via a user interface input device of a computing device of the user; generating an entry for the first natural language input in a personal database of the user stored in one or more computer readable media, wherein generating the entry comprises: generating descriptive metadata for the entry, wherein generating the descriptive metadata for the entry comprises: generating a semantic label associated with an object based on one or more terms of the first natural language input, and generating an additional semantic label associated with a location based on one or more of the terms of the first natural language input; and storing, in the entry, the descriptive metadata; subsequent to receiving the first natural language input, receiving second natural language input, wherein the second natural language input is free-form input formulated by the user via the user interface input device or an additional user interface input device of an additional computing device of the user, and wherein one or more additional terms of the second natural language input include at least the semantic label associated with the object; determining, based on the second natural language input, at least one search parameter, wherein the at least one search parameter includes the object; searching the personal database based on the at least one search parameter; determining, based on the searching, that the entry is responsive to the second natural language input, wherein determining that the entry is responsive to the second natural language input is based at least in part on matching the at least one search parameter to at least some of the descriptive metadata; and in response to determining that the entry is responsive to the second natural language input: generating natural language output that includes one or more natural language output terms that are based on the entry; and providing, for audible presentation to the user via a user interface output device of the computing device, the natural language output. ### Claim 18 (depends on 17) The non-transitory computer readable storage medium of claim 17, wherein generating the semantic label associated with the object based on one or more of the terms of the first natural language input comprises: processing the first natural language input to classify one or more of the terms of the first natural language input; and generating the semantic label associated with an object based on a classification of one or more of the terms of the first natural language input. ### Claim 19 (depends on 18) The non-transitory computer readable storage medium of claim 18, wherein the semantic label associated with the object includes a reference to the object. ### Claim 20 (depends on 18) The non-transitory computer readable storage medium of claim 18, wherein generating the additional semantic label associated with the location based on one or more of the terms of the first natural language input comprises: generating the additional semantic label associated with the location based on the classification of one or more of the terms of the first natural language input.