# US 2018/0322370 A1 · Method and apparatus of discovering bad case based on artificial intelligence, device and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0322370 A1 - Document ID: US20180322370A1 - Status: Published application - Kind: A1 - Application number: 15/961,721 - Filed: 2018-04-24 - Published: 2018-11-08 - Earliest claimed: 2017-05-05 - Assignees: Baidu Online Network Technology (Beijing) Co., Ltd. - Applicants: Baidu Online Network Technology (Beijing) Co., Ltd. - Inventors: Xiaoxiong SUN - Web page: https://patentgrep.com/patent/US20180322370A1 ## CPC - G06N 20/00: Machine learning - G06F 15/76: Architectures of general purpose stored program computers (with program plugboard G06F15/08; multicomputers G06F15/16) - G06F 18/2185: {the supervisor being an automated module, e.g. intelligent oracle} - G06F 18/2193: {based on specific statistical tests} - G06F 40/295: Named entity recognition - G06V 30/1985: {Syntactic analysis, e.g. using a grammatical approach (syntactic image representation G06V30/18171)} - G06F 40/205: Parsing - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The present disclosure provides a method and apparatus of discovering a bad case based on artificial intelligence, a device and a storage medium, wherein the method comprises: performing named entity recognition for a to-be-recognized query, and respectively obtaining a confidence level of each character in the query; respectively obtaining a probability value of each character of forming a word with a neighboring character in the query; determining whether there is a bad case according to the confidence level and the probability value. The solution of the present disclosure may be applied to save man power costs, and improve the processing efficiency and enhance a discovery rate of bad cases. ## Claims ### Claim 1 A method of discovering a bad case based on artificial intelligence, wherein the method comprises: performing named entity recognition for a to-be-recognized query, and respectively obtaining a confidence level of each character in the query; respectively obtaining a probability value of each character of forming a word with a neighboring character in the query; determining whether there is a bad case according to the confidence level and the probability value. ### Claim 2 (depends on 1) The method according to claim 1, wherein before the performing named entity recognition for a to-be-recognized query, the method further comprises: training to obtain a probability value evaluating model; the respectively obtaining a probability value of each character of forming a word with a neighboring character in the query comprises: according to the probability value evaluating model, respectively determining the probability value of each character of forming a word with a neighboring character in the query. ### Claim 3 (depends on 2) The method according to claim 2, wherein the probability value evaluating model comprises a word embedding model. ### Claim 4 (depends on 2) The method according to claim 2, wherein the according to the probability value evaluating model, respectively determining the probability value of each character of forming a word with a neighboring character in the query comprises: considering each character in the query as a candidate character, and respectively performing the following processing for each candidate character: determining a character which is spaced apart from a candidate character by less than or equal to M characters in the query as a neighboring character of the candidate character, M being a natural number; segmenting the query to obtain a segment which comprises the candidate word and at least one neighboring character; regarding each segment, determining a similar word similar to the segment and a similar probability value of each similar word according to the probability value evaluating model; selecting a similar probability value with a maximum value as a probability value of the candidate character forming a word with the neighboring character. ### Claim 5 (depends on 4) The method according to claim 4, wherein the segmenting the query to obtain a segment comprises: for each neighboring word, determining a location of the neighboring character; if the neighboring character is located before the candidate character, segmenting the query to obtain a segment starting from the neighboring character and ending at the candidate character; if the neighboring character is located behind the candidate character, segmenting the query to obtain a segment starting from the candidate character and ending at the neighboring character. ### Claim 6 (depends on 1) The method according to claim 1, wherein the determining whether there is a bad case according to the confidence level and the probability value comprises: considering each character in the query as a candidate character, and respectively performing the following processing for each candidate character: calculating a difference between the confidence level of the candidate character and a preset first threshold to obtain a first difference, and determining a first parameter according to the first difference; calculating a difference between a preset second threshold and the probability value corresponding to the candidate character to obtain a second difference, and determining a second parameter according to the second difference; determining a third parameter according to the first parameter and the second parameter; summating the third parameter corresponding to each candidate character in the query, comparing a sum with a preset third threshold, and determining a bad case if the sum is larger than the third threshold. ### Claim 7 (depends on 6) The method according to claim 6, wherein the determining a first parameter according to the first difference comprises: if the first difference is larger than 0, setting a value of the first parameter as 1; if the first difference is equal to 0, setting the value of the first parameter as 0; if the first difference is smaller than 0, setting the value of the first parameter as −1; the determining a second parameter according to the second difference comprises: if the second difference is larger than 0, setting the value of the second parameter as 1; if the second difference is equal to 0, setting the value of the second parameter as 0; if the second difference is smaller than 0, setting the value of the second parameter as −1. ### Claim 8 (depends on 7) The method according to claim 7, wherein the determining the third parameter according to the first parameter and the second parameter comprises: if both the first parameter and second parameter are smaller than 0, setting a value of the third parameter as 1, otherwise as 0; a value of the third threshold is 1. ### Claim 9 A computer device, comprising a memory, a processor and a computer program which is stored on the memory and runs on the processor, wherein the processor, upon executing the program, implements the following operation: performing named entity recognition for a to-be-recognized query, and respectively obtaining a confidence level of each character in the query; respectively obtaining a probability value of each character of forming a word with a neighboring character in the query; determining whether there is a bad case according to the confidence level and the probability value. ### Claim 10 (depends on 9) The computer device according to claim 9, wherein before the performing named entity recognition for a to-be-recognized query, the operation further comprises: training to obtain a probability value evaluating model; the respectively obtaining a probability value of each character of forming a word with a neighboring character in the query comprises: according to the probability value evaluating model, respectively determining the probability value of each character of forming a word with a neighboring character in the query. ### Claim 11 (depends on 10) The computer device according to claim 10, wherein the probability value evaluating model comprises a word embedding model. ### Claim 12 (depends on 10) The computer device according to claim 10, wherein the according to the probability value evaluating model, respectively determining the probability value of each character of forming a word with a neighboring character in the query comprises: considering each character in the query as a candidate character, and respectively performing the following processing for each candidate character: determining a character which is spaced apart from a candidate character by less than or equal to M characters in the query as a neighboring character of the candidate character, M being a natural number; segmenting the query to obtain a segment which comprises the candidate word and at least one neighboring character; regarding each segment, determining a similar word similar to the segment and a similar probability value of each similar word according to the probability value evaluating model; selecting a similar probability value with a maximum value as a probability value of the candidate character forming a word with the neighboring character. ### Claim 13 (depends on 12) The computer device according to claim 12, wherein the segmenting the query to obtain a segment comprises: for each neighboring word, determining a location of the neighboring character; if the neighboring character is located before the candidate character, segmenting the query to obtain a segment starting from the neighboring character and ending at the candidate character; if the neighboring character is located behind the candidate character, segmenting the query to obtain a segment starting from the candidate character and ending at the neighboring character. ### Claim 14 (depends on 9) The computer device according to claim 9, wherein the determining whether there is a bad case according to the confidence level and the probability value comprises: considering each character in the query as a candidate character, and respectively performing the following processing for each candidate character: calculating a difference between the confidence level of the candidate character and a preset first threshold to obtain a first difference, and determining a first parameter according to the first difference; calculating a difference between a preset second threshold and the probability value corresponding to the candidate character to obtain a second difference, and determining a second parameter according to the second difference; determining a third parameter according to the first parameter and the second parameter; summating the third parameter corresponding to each candidate character in the query, comparing a sum with a preset third threshold, and determining a bad case if the sum is larger than the third threshold. ### Claim 15 (depends on 14) The computer device according to claim 14, wherein the determining a first parameter according to the first difference comprises: if the first difference is larger than 0, setting a value of the first parameter as 1; if the first difference is equal to 0, setting the value of the first parameter as 0; if the first difference is smaller than 0, setting the value of the first parameter as −1; the determining a second parameter according to the second difference comprises: if the second difference is larger than 0, setting the value of the second parameter as 1; if the second difference is equal to 0, setting the value of the second parameter as 0; if the second difference is smaller than 0, setting the value of the second parameter as −1. ### Claim 16 (depends on 15) The computer device according to claim 15, wherein the determining the third parameter according to the first parameter and the second parameter comprises: if both the first parameter and second parameter are smaller than 0, setting a value of the third parameter as 1, otherwise as 0; a value of the third threshold is 1. ### Claim 17 A computer-readable storage medium on which a computer program is stored, wherein the program, when executed by a processor, implements the following operation: performing named entity recognition for a to-be-recognized query, and respectively obtaining a confidence level of each character in the query; respectively obtaining a probability value of each character of forming a word with a neighboring character in the query; determining whether there is a bad case according to the confidence level and the probability value. ### Claim 18 (depends on 17) The computer-readable storage medium according to claim 17, wherein before the performing named entity recognition for a to-be-recognized query, the operation further comprises: training to obtain a probability value evaluating model; the respectively obtaining a probability value of each character of forming a word with a neighboring character in the query comprises: according to the probability value evaluating model, respectively determining the probability value of each character of forming a word with a neighboring character in the query. ### Claim 19 (depends on 18) The computer-readable storage medium according to claim 18, wherein the probability value evaluating model comprises a word embedding model. ### Claim 20 (depends on 18) The computer-readable storage medium according to claim 18, wherein the according to the probability value evaluating model, respectively determining the probability value of each character of forming a word with a neighboring character in the query comprises: considering each character in the query as a candidate character, and respectively performing the following processing for each candidate character: determining a character which is spaced apart from a candidate character by less than or equal to M characters in the query as a neighboring character of the candidate character, M being a natural number; segmenting the query to obtain a segment which comprises the candidate word and at least one neighboring character; regarding each segment, determining a similar word similar to the segment and a similar probability value of each similar word according to the probability value evaluating model; selecting a similar probability value with a maximum value as a probability value of the candidate character forming a word with the neighboring character. ### Claim 21 (depends on 20) The computer-readable storage medium according to claim 20, wherein the segmenting the query to obtain a segment comprises: for each neighboring word, determining a location of the neighboring character; if the neighboring character is located before the candidate character, segmenting the query to obtain a segment starting from the neighboring character and ending at the candidate character; if the neighboring character is located behind the candidate character, segmenting the query to obtain a segment starting from the candidate character and ending at the neighboring character. ### Claim 22 (depends on 17) The computer-readable storage medium according to claim 17, wherein the determining whether there is a bad case according to the confidence level and the probability value comprises: considering each character in the query as a candidate character, and respectively performing the following processing for each candidate character: calculating a difference between the confidence level of the candidate character and a preset first threshold to obtain a first difference, and determining a first parameter according to the first difference; calculating a difference between a preset second threshold and the probability value corresponding to the candidate character to obtain a second difference, and determining a second parameter according to the second difference; determining a third parameter according to the first parameter and the second parameter; summating the third parameter corresponding to each candidate character in the query, comparing a sum with a preset third threshold, and determining a bad case if the sum is larger than the third threshold. ### Claim 23 (depends on 22) The computer-readable storage medium according to claim 22, wherein the determining a first parameter according to the first difference comprises: if the first difference is larger than 0, setting a value of the first parameter as 1; if the first difference is equal to 0, setting the value of the first parameter as 0; if the first difference is smaller than 0, setting the value of the first parameter as −1; the determining a second parameter according to the second difference comprises: if the second difference is larger than 0, setting the value of the second parameter as 1; if the second difference is equal to 0, setting the value of the second parameter as 0; if the second difference is smaller than 0, setting the value of the second parameter as −1. ### Claim 24 (depends on 23) The computer-readable storage medium according to claim 23, wherein the determining the third parameter according to the first parameter and the second parameter comprises: if both the first parameter and second parameter are smaller than 0, setting a value of the third parameter as 1, otherwise as 0; a value of the third threshold is 1.