# US 2018/0322114 A1 · System and method for detecting portability of sentiment analysis system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0322114 A1 - Document ID: US20180322114A1 - Status: Published application - Kind: A1 - Application number: 15/588,503 - Filed: 2017-05-05 - Published: 2018-11-08 - Earliest claimed: 2017-05-05 - Assignees: Medallia, Inc. - Applicants: Medallia, Inc. - Inventors: Tzu-Ting Kuo; Ji Fang - Web page: https://patentgrep.com/patent/US20180322114A1 ## CPC - G06F 40/30: Semantic analysis - G06Q 10/067: Enterprise or organisation modelling - G06Q 30/0282: Rating or review of business operators or products - G06N 3/126: Evolutionary algorithms, e.g. genetic algorithms or genetic programming - G06N 5/041: {Abduction} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Embodiments of the present invention provide a system that that can be used to determine whether a sentiment analysis model is portable between two data sets. During operation, the system analyzes the text of a respective review in a data set (e.g., a set of reviews) using the sentiment analysis model to determine a sentiment expressed in the review. The system then computes a confidence score, which indicates the accuracy of a respective sentiment. The system subsequently determines a confidence score distribution for various sentiments, as determined by the sentiment analysis model. The system determines the significance of changes between the confidence score distribution and a benchmark confidence score distribution, which is associated with a benchmark data set for which the sentiment analysis model yields a high accuracy. The system can then determine whether the sentiment analysis model is portable to the data set based on the significance of changes. ## Claims ### Claim 1 A computer-implemented method for facilitating portability analysis for a sentiment analysis model, the method comprising: analyzing, by a computer, text of a first review to determine a sentiment expressed in the text of the first review, wherein the first review is in a first data set comprising a plurality of reviews; computing a first confidence score that indicates an accuracy of the determined sentiment; determining a first confidence score distribution for the first data set, wherein the first confidence score distribution incorporates the first confidence score; determining a significance of changes between the first confidence score distribution and a benchmark confidence score distribution, wherein the benchmark confidence score distribution is associated with a benchmark data set for which the sentiment analysis model yields a high accuracy; determining whether the sentiment analysis model is portable to the first data set based on the significance of changes; and displaying, via a display device, a recommendation indicating the portability of the sentiment analysis model. ### Claim 2 (depends on 1) The method of claim 1, wherein analyzing the text of the first review comprises applying the sentiment analysis model. ### Claim 3 (depends on 1) The method of claim 1, wherein the benchmark data set comprises a set of reviews over a first period of time for a business entity, and wherein the first data set is one of: a set of reviews over a second period of time for the same business entity; a set of reviews for a second business entity in a same industry as the business entity; and a set of reviews for a third business entity in a different industry than the business entity. ### Claim 4 (depends on 1) The method of claim 1, wherein determining whether the sentiment analysis model is portable to the first data set comprises: determining whether the significance of changes is less than a portability threshold; and in response to the significance of changes being greater than or equal to the portability threshold, determining that the sentiment analysis model is portable to the first data set. ### Claim 5 (depends on 4) The method of claim 4, wherein determining whether the sentiment analysis model is portable to the first data set further comprises, in response to the significance of changes being less than the portability threshold, determining that the sentiment analysis model needs retraining for the first data set. ### Claim 6 (depends on 4) The method of claim 4, wherein determining whether the significance of changes is less than the portability threshold comprises: determining a recall of a set of p-values obtained from applying a Kolmogorov-Smirnov (K-S) test to the first confidence score distribution and the benchmark confidence score distribution; and comparing the recall with the portability threshold. ### Claim 7 (depends on 1) The method of claim 1, further comprising training the sentiment analysis model using the benchmark data set. ### Claim 8 (depends on 1) The method of claim 1, further comprising: determining a first median score for the first confidence score distribution and a second median score for the benchmark confidence score distribution; and in response to the first median score being higher than the second median score, setting the first data set as a benchmark data set for subsequent computing. ### Claim 9 A computer system for non-parametric correlation analysis, the system comprising: a processor; and a storage device storing instructions that when executed by the processor cause the processor to perform a method, the method comprising: analyzing text of a first review to determine a sentiment expressed in the text of the first review, wherein the first review is in a first data set comprising a plurality of reviews; computing a first confidence score that indicates an accuracy of the determined sentiment; determining a first confidence score distribution for the first data set, wherein the first confidence score distribution incorporates the first confidence score; determining a significance of changes between the first confidence score distribution and a benchmark confidence score distribution, wherein the benchmark confidence score distribution is associated with a benchmark data set for which the sentiment analysis model yields a high accuracy; determining whether the sentiment analysis model is portable to the first data set based on the significance of changes; and displaying, via a display device, a recommendation indicating the portability of the sentiment analysis model. ### Claim 10 (depends on 9) The computer system of claim 9, wherein analyzing the text of the first review comprises applying the sentiment analysis model. ### Claim 11 (depends on 9) The computer system of claim 9, wherein the benchmark data set comprises a set of reviews over a first period of time for a business entity, and wherein the first data set is one of: a set of reviews over a second period of time for the same business entity; a set of reviews for a second business entity in a same industry as the business entity; and a set of reviews for a third business entity in a different industry than the business entity. ### Claim 12 (depends on 9) The computer system of claim 9, wherein determining whether the sentiment analysis model is portable to the first data set comprises: determining whether the significance of changes is less than a portability threshold; and in response to the significance of changes being greater than or equal to the portability threshold, determining that the sentiment analysis model is portable to the first data set. ### Claim 13 (depends on 12) The computer system of claim 12, wherein determining whether the sentiment analysis model is portable to the first data set further comprises, in response to the significance of changes being less than the portability threshold, determining that the sentiment analysis model needs retraining for the first data set. ### Claim 14 (depends on 12) The computer system of claim 12, wherein determining whether the significance of changes is less than the portability threshold comprises: determining a recall of a set of p-values obtained from applying a Kolmogorov-Smirnov (K-S) test to the first confidence score distribution and the benchmark confidence score distribution; and comparing the recall with the portability threshold. ### Claim 15 (depends on 9) The computer system of claim 9, wherein the method further comprises training the sentiment analysis model using the benchmark data set. ### Claim 16 (depends on 9) The computer system of claim 9, wherein the method further comprises: determining a first median score for the first confidence score distribution and a second median score for the benchmark confidence score distribution; and in response to the first median score being higher than the second median score, setting the first data set as a benchmark data set for subsequent computing. ### Claim 17 A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method, the method comprising: analyzing text of a first review to determine a sentiment expressed in the text of the first review, wherein the first review is in a first data set comprising a plurality of reviews; computing a first confidence score that indicates an accuracy of the determined sentiment; determining a first confidence score distribution for the first data set, wherein the first confidence score distribution incorporates the first confidence score; determining a significance of changes between the first confidence score distribution and a benchmark confidence score distribution, wherein the benchmark confidence score distribution is associated with a benchmark data set for which the sentiment analysis model yields a high accuracy; determining whether the sentiment analysis model is portable to the first data set based on the significance of changes; and displaying, via a display device, a recommendation indicating the portability of the sentiment analysis model. ### Claim 18 (depends on 17) The storage medium of claim 17, wherein the benchmark data set comprises a set of reviews over a first period of time for a business entity, and wherein the first data set is one of: a set of reviews over a second period of time for the same business entity; a set of reviews for a second business entity in a same industry as the business entity; and a set of reviews for a third business entity in a different industry than the business entity. ### Claim 19 (depends on 17) The storage medium of claim 17, wherein determining whether the sentiment analysis model is portable to the first data set comprises: determining whether the significance of changes is less than a portability threshold; in response to the significance of changes being greater than or equal to the portability threshold, determining that the sentiment analysis model is portable to the first data set; and in response to the significance of changes being less than the portability threshold, determining that the sentiment analysis model needs retraining for the first data set. ### Claim 20 (depends on 17) The storage medium of claim 17, wherein the method further comprises: determining a first median score for the first confidence score distribution and a second median score for the benchmark confidence score distribution; and in response to the first median score being higher than the second median score, setting the first data set as a benchmark data set for subsequent computing.