# US 2022/0277192 A1 · Visual Analytics System to Assess, Understand, and Improve Deep Neural Networks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0277192 A1 - Document ID: US20220277192A1 - Status: Published application - Kind: A1 - Application number: 17/186,392 - Filed: 2021-02-26 - Published: 2022-09-01 - Earliest claimed: 2021-02-26 - Assignees: Robert Bosch GmbH - Applicants: Robert Bosch GmbH - Inventors: Liang Gou; Lincan Zou; Wenbin He; Liu Ren - Web page: https://patentgrep.com/patent/US20220277192A1 ## CPC - G06N 3/08: Learning methods - G06N 3/10: Interfaces, programming languages or software development kits, e.g. for simulating neural networks - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0475: Generative networks - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06N 3/045: Combinations of networks - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A visual analytics workflow and system are disclosed for assessing, understanding, and improving deep neural networks. The visual analytics workflow advantageously enables interpretation and improvement of the performance of a neural network model, for example an image-based objection detection and classification model, with minimal human-in-the-loop interaction. A data representation component extracts semantic features of input image data, such as colors, brightness, background, rotation, etc. of the images or objects in the images. The input image data are passed through the neural network to obtain prediction results, such as object detection and classification results. An interactive visualization component transforms the prediction results and semantic features into interactive and human-friendly visualizations, in which graphical elements encoding the prediction results are visually arranged depending on the extracted semantic features of input image data. ## Claims ### Claim 1 A method for visualizing an operation of a neural network model, the method comprising: receiving, with a processor, a plurality of images; receiving, with the processor, a plurality of outputs, each output in the plurality of outputs having been outputted by the neural network model responsive to a corresponding image in the plurality of images; receiving, with the processor, a plurality of sets of visual features, each set of visual features having been extracted from a corresponding image in the plurality of images; and displaying, on a display screen, a graphical user interface including a graphical depiction of the plurality of outputs, the graphical depiction including a plurality of graphical elements encoding the plurality of outputs which are visually arranged depending on the plurality of sets of visual features. ### Claim 2 (depends on 1) The method of claim 1 further comprising: receiving, via a user interface, a user input; and adjusting, with the processor, based on the user input, the graphical depiction to arrange the plurality of graphical elements in a different manner according to the plurality of sets of visual features. ### Claim 3 (depends on 1) The method of claim 1, the receiving the plurality of outputs further comprising: determining, with the processor, each output in the plurality of outputs based on the corresponding image in the plurality of images using the neural network model. ### Claim 4 (depends on 1) The method of claim 1, wherein each output in the plurality of outputs is a classification of at least one of (i) a presence of a particular object in the corresponding image in the plurality of images and (ii) state of the particular object in the corresponding image in the plurality of images. ### Claim 5 (depends on 1) The method of claim 1, the receiving the plurality of sets of visual features further comprising: determining, with the processor, each set of visual features in the plurality of sets of visual features based on the corresponding image in the plurality of images. ### Claim 6 (depends on 1) The method of claim 1, the displaying the graphical user interface including the graphical depiction further comprising: sorting, with the processor, the plurality of images into a plurality of bins based on at least one visual feature in the corresponding sets of visual features, each bin of the plurality of bins defining a range of values for the at least one visual feature; and generating, with the processor, the graphical depiction with each graphical element in the plurality graphical elements encoding outputs in the plurality of outputs corresponding to images in the plurality of images that are sorted into a corresponding bin of the plurality of bins. ### Claim 7 (depends on 6) The method of claim 6, the generating the graphical depiction further comprising: generating, with the processor, the graphical depiction with each graphical element in the plurality graphical elements arranged along at least one axis of the graphical depiction depending on the respective ranges of values for the at least one visual feature defined by the corresponding bins of the plurality of bins. ### Claim 8 (depends on 6) The method of claim 6, the sorting the plurality of images into the plurality of bins further comprising: sorting, with the processor, the plurality of images into a plurality of bins based on a first visual feature and a second visual feature of the corresponding sets of visual features, each bin of the plurality of bins defining a range of values for each of the first visual feature and the second visual feature. ### Claim 9 (depends on 8) The method of claim 8, the generating the graphical depiction further comprising: generating, with the processor, the graphical depiction with each graphical element in the plurality graphical elements arranged in a grid formation having a first axis and a second axis, the plurality graphical elements being arranged along the first axis depending on the respective ranges of values for the first feature defined by the corresponding bins of the plurality of bins, the plurality graphical elements being arranged along the second axis depending on the respective ranges of values for the second feature defined by the corresponding bins of the plurality of bins. ### Claim 10 (depends on 6) The method of claim 6, the generating the graphical depiction further comprising: generating, with the processor, the graphical depiction with some of the graphical elements in the plurality graphical elements being thumbnails of representative images of the plurality of images that are sorted into the corresponding bins of the plurality of bins. ### Claim 11 (depends on 6) The method of claim 6, the generating the graphical depiction further comprising: generating, with the processor, the graphical depiction with some of the graphical element in the plurality graphical elements encoding a performance metric of the neural network model with respect to images of the plurality of images that are sorted into the corresponding bins of the plurality of bins. ### Claim 12 (depends on 11) The method of claim 11, wherein at least one of: a color of some of graphical elements in the plurality graphical elements encodes the performance metric; and at least one of a (i) size and (ii) shape of some of graphical elements in the plurality graphical elements encodes a total number of images sorted into the corresponding bins of the plurality of bins. ### Claim 13 (depends on 6) The method of claim 6 further comprising: receiving, via a user interface, a user input selecting the at least one visual feature in the corresponding sets of visual features, based on which the plurality of images are sorted into the plurality of bins. ### Claim 14 (depends on 6) The method of claim 6, the generating the graphical depiction further comprising: generating, with the processor, a first graphical depiction having a first plurality of graphical elements, each graphical element in the first plurality graphical elements encoding outputs in the plurality of outputs corresponding to images in the plurality of images that are sorted into a corresponding bin of the plurality of bins. generating, with the processor, a second graphical depiction having a second plurality of graphical elements, each graphical element in the second plurality graphical elements encoding outputs in the plurality of outputs corresponding to images in the plurality of images that are sorted into a corresponding bin of the plurality of bins. ### Claim 15 (depends on 14) The method of claim 14 further comprising: receiving, via a user interface, a user input selecting a subset of the first plurality of graphical elements; and adjusting, with the processor, the second graphical depiction depending on the selected subset of the first plurality of graphical elements. ### Claim 16 (depends on 15) The method of claim 15, the adjusting the second graphical depiction further comprising: adjusting, with the processor, the second graphical depiction to highlight a subset of the second plurality of graphical elements corresponding to bins in the plurality of bins defining ranges of values for the at least one visual feature that correlate to the ranges of values for the at least one visual feature defined by the bins in the plurality of bins that correspond to the selected subset of the first plurality of graphical elements. ### Claim 17 (depends on 6) The method of claim 6, the generating the graphical depiction further comprising: generating, with the processor, a plurality of graphical depictions, each respective graphical depiction in the plurality of graphical depictions corresponding to respective visual feature of the sets of visual features, each respective graphical depiction in the plurality of graphical depictions having a respective plurality of graphical elements, each graphical element in the respective plurality graphical elements encoding outputs in the plurality of outputs corresponding to images in the plurality of images that are sorted into a corresponding bin of the plurality of bins, the corresponding bins of the respective plurality graphical elements each defining a range of values for the respective visual feature; determining, with the processor, a hierarchy of the respective visual features corresponding to plurality of graphical depictions; and displaying, on the display screen, the graphical user interface with the plurality of graphical depictions arranged according to determined hierarchy. ### Claim 18 (depends on 17) The method of claim 17 further comprising: generating, with the processor, a further graphical depiction including a further plurality of graphical elements, each graphical element in the further plurality graphical elements corresponding a respective bin of the plurality of bins; receiving, via a user interface, a user input selecting a subset of the further plurality of graphical elements; and determining, with the processor, the hierarchy based on the selected subset of the further plurality of graphical elements. ### Claim 19 A system for visualizing an operation of a neural network model, the system comprising: a display screen; a memory configured to store (i) a plurality of images, (ii) a plurality of outputs, each output in the plurality of outputs having been outputted by the neural network model responsive to a corresponding image in the plurality of images, and (iii) a plurality of sets of visual features, each set of visual features having been extracted from a corresponding image in the plurality of images; and a processor operably connected to the display screen and the memory, the processor being configured to: read, from the memory, (i) the plurality of images, (ii) the plurality of outputs, and (iii) the plurality of sets of visual features; and generate, and operate the display screen to display, a graphical user interface including a graphical depiction of the plurality of outputs, the graphical depiction including a plurality of graphical elements encoding the plurality of outputs which are visually arranged depending on the plurality of sets of visual features. ### Claim 20 A non-transitory computer readable medium for visualizing an operation of a neural network model, the computer-readable medium storing program instructions that, when executed by a processor, cause the processor to: receive a plurality of images; receive a plurality of outputs, each output in the plurality of outputs having been outputted by the neural network model responsive to a corresponding image in the plurality of images; receive a plurality of sets of visual features, each set of visual features having been extracted from a corresponding image in the plurality of images; generate, and operate a display screen to display, a graphical user interface including a graphical depiction of the plurality of outputs, the graphical depiction including a plurality of graphical elements encoding the plurality of outputs which are visually arranged depending on the plurality of sets of visual features.