# US 12,001,798 B2 · Generation of training data for machine learning based models for named entity recognition for natural language processing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,001,798 B2 - Document ID: US12001798B2 - Status: Granted - Kind: B2 - Application number: 17/202,188 - Filed: 2021-03-15 - Published: 2022-07-14 - Granted: 2024-06-04 - Earliest claimed: 2021-01-13 - Assignees: Salesforce, Inc. - Applicants: Salesforce, Inc. - Inventors: Jingyuan Liu; Abhishek Sharma; Suhail Sanjiv Barot; Gurkirat Singh; Mridul Gupta; Shiva Kumar Pentyala; Ankit Chadha - Web page: https://patentgrep.com/patent/US12001798B2 ## CPC - G06F 40/295: Named entity recognition - H04L 51/18: Commands or executable codes - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 40/247: Thesauruses; Synonyms - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06F 40/35: Discourse or dialogue representation - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - G06F 40/166: Editing, e.g. inserting or deleting - G06F 40/253: Grammatical analysis; Style critique - G06F 40/289: Phrasal analysis, e.g. finite state techniques or chunking - G06F 40/56: Natural language generation - H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A system performs named entity recognition for performing natural language processing, for example, for conversation engines. The system uses context information in named entity recognition. The system includes the context of a sentence during model training and execution. The system generates high quality contextual data for training NER models. The system utilizes labeled and unlabeled contextual data for training NER models. The system provides NER models for execution in production environments. The system uses heuristics to determine whether to use a context-based NER model or a simple NER model that does not use context information. This allows the system to use simple NER models when the likelihood of improving the accuracy of prediction based on context is low. ## Claims ### Claim 1 A computer implemented method for generating a training dataset for machine learning based models for natural language processing, the method comprising: receiving a set of natural language phrases; generating a plurality of user responses from the set of natural language phrases, the generating comprising: selecting a natural language phrase from the set of natural language phrases; processing the selected natural language phrase to extract an entity of a particular named entity type; generating a user response associated with the particular named entity type based on the extracted entity; and determining whether to collect a plurality of context strings, each context string associated with a named entity type, wherein the determining is based on whether a length of a natural language phrase associated with the named entity type is shorter than a threshold length; receiving, based on determining whether to collect the plurality of context strings, the plurality of context strings; generating the training dataset by repeating for a named entity type: selecting a context string associated with the named entity type from the plurality of context strings; selecting a user response associated with the named entity type from the plurality of user responses, wherein the selecting is random in response to an amount of context strings in the plurality of context strings and an amount of user responses in the plurality of user responses being above a threshold amount, and wherein the selecting is according to a pattern such that each context string in the plurality of context strings is selected with each user response in the plurality of user responses in response to the amount of context strings in the plurality of context strings and the amount of user responses in the plurality of user responses being below a threshold amount; and adding a pair of the selected context string and the selected user response to the training dataset; and using the training dataset for training a machine learning based model for use in named entity recognition. ### Claim 2 (depends on 1) The computer implemented method of claim 1, wherein the machine learning based model is configured to receive as input, a combined string obtained by combining a context string and a user response, wherein different portions of the combined string are identified using separator tokens. ### Claim 3 (depends on 1) The computer implemented method of claim 1, further comprising, generating context sentences by performing: receiving a set of seed context sentences, each seed context associated with a named entity type; and for each of one or more seed context sentences, generating a plurality of context sentences by modifying the seed context sentence. ### Claim 4 (depends on 3) The computer implemented method of claim 3, wherein modifying the seed context sentence comprises converting a first letter of multiple keywords of the seed context sentence to uppercase. ### Claim 5 (depends on 3) The computer implemented method of claim 3, wherein modifying the seed context sentence comprises adding one or more punctuation marks to the seed context sentence. ### Claim 6 (depends on 3) The computer implemented method of claim 3, wherein modifying the seed context sentence comprises switches one or more words of the seed context with other words that could occur in that context. ### Claim 7 (depends on 3) The computer implemented method of claim 3, wherein modifying the seed context sentence comprises generating an alternative statement that has equivalent meaning using different words. ### Claim 8 (depends on 1) The computer implemented method of claim 1, wherein the machine learning based model is used for generating responses in a conversation engine in response to a prediction of a pretrained binary intent model that a context of a conversation of the conversation engine is relevant. ### Claim 9 A non-transitory computer readable storage medium storing instructions that when executed by a computer processor, cause the computer processor to perform steps comprising: receiving a set of natural language phrases; generating a plurality of user responses from the set of natural language phrases, the generating comprising: selecting a natural language phrase from the set of natural language phrases; processing the selected natural language phrase to extract an entity of a particular named entity type; generating a user response associated with the particular named entity type based on the extracted entity; and determining whether to collect a plurality of context strings, each context string associated with a named entity type, wherein the determining is based whether on a length of a natural language phrase associated with the named entity type is shorter than a threshold length; receiving, based on determining whether to collect the plurality of context strings, the plurality of context strings; generating a training dataset by repeating for a named entity type: selecting a context string associated with the named entity type from the plurality of context strings; selecting a user response associated with the named entity type from the plurality of user responses, wherein the selecting is random in response to an amount of context strings in the plurality of context strings and an amount of user responses in the plurality of user responses being above a threshold amount, and wherein the selecting is according to a pattern such that each context string in the plurality of context strings is selected with each user response in the plurality of user responses in response to the amount of context strings in the plurality of context strings and the amount of user responses in the plurality of user responses being below a threshold amount; and adding a pair of the selected context string and the selected user response to the training dataset; and using the training dataset for training a machine learning based model for use in named entity recognition. ### Claim 10 (depends on 9) The non-transitory computer readable storage medium of claim 9, wherein the machine learning based model is configured to receive as input, a combined string obtained by combining a context string and a user response, wherein different portions of the combined string are identified using separator tokens. ### Claim 11 (depends on 9) The non-transitory computer readable storage medium of claim 9, further comprising, generating context sentences by performing: receiving a set of seed context sentences, each seed context associated with a named entity type; and for each of one or more seed context sentences, generating a plurality of context sentences by modifying the seed context sentence. ### Claim 12 (depends on 11) The non-transitory computer readable storage medium of claim 11, wherein modifying the seed context sentence comprises converting a first letter of multiple keywords of the seed context sentence to uppercase. ### Claim 13 (depends on 11) The non-transitory computer readable storage medium of claim 11, wherein modifying the seed context sentence comprises adding one or more punctuation marks to the seed context sentence. ### Claim 14 (depends on 11) The non-transitory computer readable storage medium of claim 11, wherein modifying the seed context sentence comprises switches one or more words of the seed context with other words that could occur in that context. ### Claim 15 (depends on 11) The non-transitory computer readable storage medium of claim 11, wherein modifying the seed context sentence comprises generating an alternative statement that has equivalent meaning using different words. ### Claim 16 (depends on 11) The non-transitory computer readable storage medium of claim 11, wherein the machine learning based model is used for generating responses in a conversation engine. ### Claim 17 A computer system comprising: a computer processor; and a non-transitory computer readable storage medium storing instructions that when executed by a computer processor, cause the computer processor to perform steps comprising: receiving a set of natural language phrases; generating a plurality of user responses from the set of natural language phrases, the generating comprising: selecting a natural language phrase from the set of natural language phrases; processing the selected natural language phrase to extract an entity of a particular named entity type; generating a user response associated with the particular named entity type based on the extracted entity; and determining whether to collect a plurality of context strings, each context string associated with a named entity type, wherein the determining is based on whether a length of a natural language phrase associated with the named entity type; receiving, based on determining whether to collect the plurality of context strings, the plurality of context strings; generating a training dataset by repeating for a named entity type: selecting a context string associated with the named entity type from the plurality of context strings; selecting a user response associated with the named entity type from the plurality of user responses, wherein the selecting is random in response to an amount of context strings in the plurality of context strings and an amount of user responses in the plurality of user responses being above a threshold amount, and wherein the selecting is according to a pattern such that each context string in the plurality of context strings is selected with each user response in the plurality of user responses in response to the amount of context strings in the plurality of context strings and the amount of user responses in the plurality of user responses being below a threshold amount; and adding a pair of the selected context string and the selected user response to the training dataset; and using the training dataset for training a machine learn-ing based model for use in named entity recognition. ### Claim 18 (depends on 17) The computer system of claim 17, wherein the machine learning based model is configured to receive as input, a combined string obtained by combining a context string and a user response, wherein different portions of the combined string are identified using separator tokens. ### Claim 19 (depends on 17) The computer system of claim 17, further comprising, generating context sentences by performing: receiving a set of seed context sentences, each seed context associated with a named entity type; and for each of one or more seed context sentences, generating a plurality of context sentences by modifying the seed context sentence. ### Claim 20 (depends on 17) The computer system of claim 17, wherein the machine learning based model is used for generating responses in a conversation engine.