# US 12,182,136 B2 · Global confidence classifier for information retrieval in contact centers > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,136 B2 - Document ID: US12182136B2 - Status: Granted - Kind: B2 - Application number: 17/564,783 - Filed: 2021-12-29 - Published: 2023-06-29 - Granted: 2024-12-31 - Earliest claimed: 2021-12-29 - Assignees: Genesys Cloud Services, Inc. - Applicants: Genesys Cloud Services, Inc. - Inventors: Veera Raghavendra Elluru; Ramasubramanian Sundaram; Basil George; Naresh Kumar Elluru; Pavan Kumar Buduguppa - Web page: https://patentgrep.com/patent/US12182136B2 ## CPC - G06F 16/24578: {using ranking} - G06F 16/3344: {using natural language analysis} - G06F 16/3346: {using probabilistic model} - G06F 16/3347: {using vector based model} - G06F 16/383: using metadata automatically derived from the content - G06F 16/93: Document management systems - G06N 3/045: Combinations of networks - G06N 20/00: Machine learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method of leveraging a global confidence classifier for information retrieval in contact centers according to an embodiment includes receiving, by a computing system, a user query from a contact center client communication with a knowledge-only bot of the computing system, performing, by the computing system, feature extraction on the user query by converting query words of the user query into a numerical vector representation of the user query, identifying, by the computing system, a subset of documents most likely to be responsive to the user query, and re-ranking, by the computing system, the subset of documents most likely to be responsive to the user query based on a global confidence classifier model. ## Claims ### Claim 1 A method of leveraging a global confidence classifier for information retrieval of knowledge-only bots in contact centers, the method comprising: receiving, by a computing system, a user query from a contact center client's communication with a knowledge-only bot of the computing system; performing, by the computing system, feature extraction on the user query by converting query words of the user query into a numerical vector representation of the user query; identifying, by the computing system using a document classifier machine learning model, a subset of documents most likely to be responsive to the user query; and re-ranking, by the computing system using a global confidence classifier machine learning model, the subset of documents identified to be most likely to be responsive to the user query based on the document classifier machine learning model; wherein the global confidence classifier machine learning model is trained based on a confidence classifier that uses at least an alignment score, a similarity score, a document score, and a word overlap score; wherein the alignment score is determined based on the subset of documents, a matrix of word embedding vectors for each word in the user query, and a similarity matrix between words in the user query and words in the subset of documents; wherein the similarity score is a cosine similarity on average word embedding vectors of a respective document of the subset of documents and the user query; wherein the document score is a score generated by the document classifier machine learning model; and wherein the word overlap score is indicative of a number of n-grams that are similar between the user query and the respective document of the subset of documents. ### Claim 2 (depends on 1) The method of claim 1, further comprising providing, by the computing system, a response to the contact center client based on the re-ranked subset of documents via the knowledge-only bot in response to re-ranking the subset of documents identified to be most likely to be responsive to the user query. ### Claim 3 (depends on 1) The method of claim 1, wherein the global confidence classifier machine learning model is trained based on document sets for a plurality of organizations. ### Claim 4 (depends on 3) The method of claim 3, wherein the global confidence classifier machine learning model is trained based on document sets in a plurality of human languages. ### Claim 5 (depends on 1) The method of claim 1, wherein the global confidence classifier machine learning model is trained based on a confidence classifier that uses at least five input features. ### Claim 6 A method of leveraging a global confidence classifier for information retrieval of knowledge-only bots in contact centers, the method comprising: receiving, by a computing system, a user query from a contact center client communication with a knowledge-only bot of the computing system; performing, by the computing system, feature extraction on the user query by converting query words of the user query into a numerical vector representation of the user query; identifying, by the computing system, a subset of documents most likely to be responsive to the user query using a first classifier machine learning model; and re-ranking, by the computing system using a global confidence classifier machine learning model, the subset of documents identified to be most likely to be responsive to the user query based on the first classifier machine learning model; wherein the global confidence classifier machine learning model is trained based on a confidence classifier that uses at least five input features; and wherein the at least five input features comprise a first alignment score between the user query and a corresponding document, a second alignment score between the corresponding document and the user query, a cosine similarity score between the user query and the corresponding document, a document score from a document classifier, and a word overlap score that calculates a number of n-grams that are similar between the user query and the corresponding document. ### Claim 7 (depends on 1) The method of claim 1, wherein performing the feature extraction on the user query comprises performing the feature extraction on the user query using custom embeddings. ### Claim 8 (depends on 1) The method of claim 1, further comprising natural language processing, by the computing system, the user query to generate processed data, wherein the natural language processing comprises at least one of cleaning or segmenting data of the user query. ### Claim 9 A system for leveraging a global confidence classifier for information retrieval of knowledge-only bots in contact centers, the system comprising: at least one processor; and at least one memory comprising a plurality of instructions stored thereon that, in response to execution by the at least one processor, causes the system to: receive a user query from a contact center client communicating with a knowledge-only bot of the system; perform feature extraction on the user query by converting query words of the user query into a numerical vector representation of the user query; identify, using a document classifier machine learning model, a subset of documents most likely to be responsive to the user query; re-rank, using a global confidence classifier machine learning model, the subset of documents identified to be most likely to be responsive to the user query based on the feature extraction and the document classifier machine learning model; wherein the global confidence classifier machine learning model is trained based on a confidence classifier that uses at least five input features; and wherein the at least five input features comprise a first alignment score between the user query and a corresponding document, a second alignment score between the corresponding document and the user query, a cosine similarity score between the user query and the corresponding document, a document score from a document classifier, and a word overlap score that calculates a number of n-grams that are similar between the user query and the corresponding document. ### Claim 10 (depends on 9) The system of claim 9, wherein the plurality of instructions further causes the system to provide a response to the contact center client based on the re-ranked subset of documents via the knowledge-only bot in response to re-ranking the subset of documents identified to be most likely to be responsive to the user query. ### Claim 11 (depends on 9) The system of claim 9, wherein the global confidence classifier machine learning model is trained based on document sets for a plurality of organizations in a plurality of human languages. ### Claim 12 (depends on 9) The system of claim 9, wherein to perform the feature extraction on the user query comprises to perform the feature extraction on the user query using custom embeddings. ### Claim 13 (depends on 9) The system of claim 9, wherein the plurality of instructions further causes the system to apply natural language processing to the user query to generate processed data, wherein the natural language processing comprises at least one of cleaning or segmenting data of the user query; wherein to perform the feature extraction on the user query comprises perform feature extraction on the processed data; and wherein to identify the subset of documents most likely to be responsive to the user query comprises to identify the subset of documents based on the processed data. ### Claim 14 (depends on 1) The method of claim 1, wherein the document classifier machine learning model comprises a linear document classifier model. ### Claim 15 (depends on 1) The method of claim 1, wherein identifying the subset of documents most likely to be responsive to the user query comprises calculating term frequency-inverse document frequency (TF-IDF) scores for the user query. ### Claim 16 (depends on 1) The method of claim 1, wherein re-ranking the subset of documents identified to be most likely to be responsive to the user query based on the document classifier machine learning model comprises generating a confidence score between the user query and each possible document. ### Claim 17 (depends on 1) The method of claim 1, wherein the knowledge-only bot retrieves data from at least one knowledge base; and wherein the global confidence classifier machine learning model is trained based on a plurality of knowledge bases including another knowledge base different from the at least one knowledge base. ### Claim 18 (depends on 1) The method of claim 1, wherein the alignment score is indicative of a similarity of the respective document of the subset of documents to the user query.