# US 11,029,814 B1 · Visualization of a machine learning confidence score and rationale > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,029,814 B1 - Document ID: US11029814B1 - Status: Granted - Kind: B1 - Application number: 16/919,580 - Filed: 2020-07-02 - Granted: 2021-06-08 - Earliest claimed: 2019-03-12 - Assignees: Bottomline Technologies, Inc. - Applicants: Bottomline Technologies (de) Inc. - Inventors: Flora Kidani; John Canneto; Anne Baron; Jonathan Hewitt; William Cashman; Michael Marcinelli - Web page: https://patentgrep.com/patent/US11029814B1 ## CPC - G06F 3/04817: using icons (graphical or visual programming using iconic symbols G06F8/34) - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 40/109: Font handling; Temporal or kinetic typography - G06F 40/205: Parsing - G06F 40/216: using statistical methods - G06N 20/00: Machine learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A unique user interface for improving machine learning algorithms is described herein. The user interface comprises an icon with multiple visual indicators displaying the machine learning confidence score. When a mouse hovers over the icon, a set of icons are displayed to accept the teaching user's input. In addition, the words that drove the machine learning confidence score are highlighted with formatting so that the teaching user can understand what drove the machine learning confidence score. ## Claims ### Claim 1 A special purpose computer implemented method of visualizing a machine learning confidence score, the method comprising: determining the machine learning confidence score on an invoice comprising: parsing a narrative section of the invoice into a table words; converting the words in the table of into stems; looking up each stem in a machine learning model; placing a weight for the stem, as found in the machine learning model, in the table of the words; determining a confidence score by averaging the weights for each word in the table of the words; and looking through the narrative section and changing formatting of the words based on a relative weight in the table of the words to create a word cloud; displaying an indication of the machine learning confidence score on a display screen along with the narrative section; detecting a user input; changing the indication of the machine learning confidence score based on the user input; and displaying the word cloud in response to detecting the user input. ### Claim 2 (depends on 1) The method of claim 1 wherein the formatting is a bold format. ### Claim 3 (depends on 1) The method of claim 1 wherein the formatting is a color format. ### Claim 4 (depends on 1) The method of claim 1 wherein the formatting is a font size. ### Claim 5 (depends on 1) The method of claim 1 wherein the user input is a mouse-over. ### Claim 6 (depends on 1) The method of claim 1 wherein the user input is a touch on a touch screen. ### Claim 7 (depends on 1) The method of claim 1 wherein the changing of the indication of the confidence score displays a thumb icon. ### Claim 8 (depends on 7) The method of claim 7 wherein the thumb icon is a thumb up icon. ### Claim 9 (depends on 1) The method of claim 1 further comprising accepting a user indication of the acceptance of the confidence score. ### Claim 10 (depends on 9) The method of claim 9 further comprising changing the display screen to reflect that the confidence score has been accepted. ### Claim 11 An apparatus for visualization of a machine learning confidence score, the apparatus comprising: a display screen; a special purpose computer electrically connected to the display screen; a large capacity data storage facility with a machine learning training data set; a user interface display algorithm operates on the special purpose computer and displays an indication of the machine learning confidence score on the display screen; a natural language processing algorithm operates on the special purpose computer and interfaces with the user interface display algorithm to convert a textual description into a table of word stems; a machine learning algorithm operates on the special purpose computer and interfaces with the natural language processing algorithm to convert the table of the word stems into the machine learning confidence score using a machine learning model built using the machine learning training data set; a scan back algorithm operates on the special purpose computer and interfaces to the machine learning algorithm to insert format instructions into the textual description using the table of word stems to format at least one word in the textual description that impacted the machine learning confidence score; and a user interface input algorithm operates on the special purpose computer and interfaces with the scan back algorithm to display the textual description with the format instructions and to accept user input regarding the machine learning confidence score and to add the user input to the machine learning training data set, wherein the textual description with the format instructions is only displayed on the display screen when a mouse location is over a variable icon that varies depending on a magnitude of the machine learning confidence score. ### Claim 12 (depends on 11) The apparatus of claim 11 wherein the format instructions bold a portion of the textual description. ### Claim 13 (depends on 11) The apparatus of claim 11 wherein the format instructions increase a font on a portion of the textual description. ### Claim 14 (depends on 11) The apparatus of claim 11 wherein the indication of the machine learning confidence score on the display screen is in a form of the variable icon that varies depending on the magnitude of the machine learning confidence score. ### Claim 15 (depends on 14) The apparatus of claim 14 wherein the variable icon displays a different number of items depending upon the magnitude of the machine learning confidence score. ### Claim 16 (depends on 15) The apparatus of claim 15 wherein the variable icon displays three rays off of a lightbulb based on one range of the confidence score, and one ray off of the lightbulb based on another range of the machine learning confidence score. ### Claim 17 (depends on 11) The apparatus of claim 11 wherein the user input modifies the machine learning model in real time. ### Claim 18 A special purpose computer implemented method of visualizing a machine learning confidence score, the method comprising: displaying an indication of the machine learning confidence score on a display screen along with a narrative section; detecting that a mouse location is over the indication of the machine learning confidence score on the display screen; changing the indication of the machine learning confidence score based on the mouse over the indication of the machine learning confidence score; and changing formatting of the narrative section based on the mouse over the indication of the machine learning confidence score. ### Claim 19 (depends on 18) The method of claim 18 wherein the indication of the machine learning confidence score is a light bulb icon. ### Claim 20 (depends on 19) The method of claim 19 wherein the light bulb icon varies a number of rays with the light bulb icon based on the machine learning confidence score.