# US 2022/0277142 A1 · Methods and apparatus for natural language understanding in conversational systems using machine learning processes > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0277142 A1 - Document ID: US20220277142A1 - Status: Published application - Kind: A1 - Application number: 17/187,764 - Filed: 2021-02-27 - Published: 2022-09-01 - Earliest claimed: 2021-02-27 - Assignees: Walmart Apollo, LLC - Applicants: Walmart Apollo, LLC - Inventors: Deepa Mohan - Web page: https://patentgrep.com/patent/US20220277142A1 ## CPC - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06F 16/3329: Natural language query formulation - G06F 16/3347: {using vector based model} - G06F 16/90332: {Natural language query formulation or dialogue systems} - G06F 40/295: Named entity recognition - G06F 40/35: Discourse or dialogue representation - G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/0499: Feedforward networks - G06N 3/09: Supervised learning - G06N 3/045: Combinations of networks - G06N 3/08: Learning methods - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract This application relates to apparatus and methods for natural language understanding in conversational systems using machine learning processes. In some examples, a computing device receives a request that identifies textual data. The computing device applies a natural language model to the textual data to generate first embeddings. In some examples, the natural language model is trained on retail data, such as item descriptions and chat session data. The computing device also applies a dependency based model to the textual data to generate second embeddings. Further, the computing device concatenates the first and second embeddings, and applies an intent and entity classifier to the concatenated embeddings to determine entities, and an intent, for the request. The computing device may generate a response to the request based on the determined intent and entities. ## Claims ### Claim 1 A system comprising: a database; and a computing device communicatively coupled to the database and configured to: receive input data comprising a plurality of characters; generate word embeddings based on the plurality of characters; apply a natural language model to the word embeddings to generate first output embeddings; apply a linear layer to the first output embeddings to generate second output embeddings; and store the second output embeddings in the database. ### Claim 2 (depends on 1) The system of claim 1, wherein the plurality of characters comprise a previous context and a current context. ### Claim 3 (depends on 2) The system of claim 2, wherein the previous context is based on a previous chat session, and the current context is based on an inquiry. ### Claim 4 (depends on 1) The system of claim 1, wherein the computing device is further configured to apply a neural network to previously generated intent data to generate third output embeddings, where the linear layer is applied to the first output embeddings and the third output embeddings to generate the second output embeddings. ### Claim 5 (depends on 4) The system of claim 4, wherein the computing device is further configured to concatenate the first output embeddings and the third output embeddings to generate concatenated embeddings, wherein the linear layer is applied to the concatenated embeddings. ### Claim 6 (depends on 5) The system of claim 5, wherein the computing device is further configured to normalize the concatenated embeddings, and generate the second output embeddings based on the normalized concatenated embeddings. ### Claim 7 (depends on 1) The system of claim 1, wherein the computing device is further configured to tokenize the input data into a plurality of tokens, wherein the natural language model is applied to the tokenized input data. ### Claim 8 (depends on 1) The system of claim 1, wherein the second output embeddings characterize an intent of the plurality of characters. ### Claim 9 (depends on 1) The system of claim 1, wherein the plurality of characters identify a first item title, and the second output embeddings characterize a second item title that is shorter than the first item title. ### Claim 10 (depends on 1) The system of claim 1, wherein the computing device is further configured to: apply a dependency based model to the input data to generate dependency based embeddings; apply a second linear layer to the dependency based embeddings to generate third output embeddings; and concatenate the second output embeddings with the third output embeddings. ### Claim 11 (depends on 1) The system of claim 1, wherein the input data is received in a request from a second computing device, and wherein the computing device is further configured to generate a response to the request based on the output values. ### Claim 12 (depends on 1) The system of claim 1, wherein the natural language model is a BERT model. ### Claim 13 (depends on 8) The system of claim 8, the computing device is further configured to train the BERT model based on catalog data and chat session data. ### Claim 14 A method comprising: receiving input data comprising a plurality of characters; generating word embeddings based on the plurality of characters; applying a natural language model to the word embeddings to generate first output embeddings; applying a linear layer to the first output embeddings to generate second output embeddings; and storing the second output embeddings in the database. ### Claim 15 (depends on 14) The method of claim 14 further comprising applying a neural network to previously generated intent data to generate third output embeddings, where the linear layer is applied to the first output embeddings and the third output embeddings to generate the second output embeddings. ### Claim 16 (depends on 15) The method of claim 15, further comprising concatenating the first output embeddings and the third output embeddings to generate concatenated embeddings, wherein the linear layer is applied to the concatenated embeddings. ### Claim 17 (depends on 14) The method of claim 14 further comprising tokenizing the input data into a plurality of tokens, wherein the natural language model is applied to the tokenized input data. ### Claim 18 (depends on 14) The method of claim 14 wherein the second output embeddings characterize an intent of the plurality of characters. ### Claim 19 (depends on 14) The method of claim 14 wherein the plurality of characters identify a first item title, and the second output embeddings characterize a second item title that is shorter than the first item title. ### Claim 20 A non-transitory computer readable medium having instructions stored thereon, wherein the instructions, when executed by at least one processor, cause a device to perform operations comprising: receiving input data comprising a plurality of characters; generating word embeddings based on the plurality of characters; applying a natural language model to the word embeddings to generate first output embeddings; applying a linear layer to the first output embeddings to generate second output embeddings; and storing the second output embeddings in the database.