# US 2021/0279529 A1 · Method and device for ascertaining an explanation map > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0279529 A1 - Document ID: US20210279529A1 - Status: Published application - Kind: A1 - Application number: 17/261,810 - Filed: 2019-07-03 - Published: 2021-09-09 - Earliest claimed: 2018-08-03 - Assignees: Robert Bosch GmbH - Applicants: Robert Bosch GmbH - Inventors: Joerg Wagner; Tobias Gindele; Jan Mathias Koehler; Jakob Thaddaeus Wiedemer; Leon Hetzel - Web page: https://patentgrep.com/patent/US20210279529A1 ## CPC - G06N 3/084: Backpropagation, e.g. using gradient descent - G06F 18/241: relating to the classification model, e.g. parametric or non-parametric approaches - G06F 18/41: {Interactive pattern learning with a human teacher} - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/048: Activation functions - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06N 5/045: Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence - G06V 2201/03: Recognition of patterns in medical or anatomical images - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method for ascertaining an explanation map of an image. All those pixels of the image are highlighted which are significant for a classification of the image ascertained with the aid of a deep neural network. The explanation map is being selected in such a way that it selects a smallest possible subset of the pixels of the image as relevant. The explanation map leads to the same classification result as the image when the explanation map is supplied to the deep neural network for classification. The explanation map is selected in such a way that an activation caused by the explanation map does not essentially exceed an activation caused by the image in feature maps of the deep neural network. ## Claims ### Claim at position 1 1-13. (canceled) ### Claim 14 A method for ascertaining an explanation map of an image, the method comprising: highlighting all those pixels of the image which are significant for a classification of the image ascertained using a deep neural network, the explanation map being selected in such a way that the explanation map selects a smallest possible subset of the pixels of the image as relevant, and the explanation map leads to the same classification result as the image as when the explanation map is supplied to the deep neural network for classification, the explanation map being selected in such a way that an activation caused by the explanation map does not exceed an activation caused by the image in feature maps of the deep neural network. ### Claim 15 (depends on 14) The method as recited in claim 14, wherein the explanation map is selected in such a way that a loss function is optimized, the loss function including a first component, which characterizes how many pixels of the image are selected as relevant, and/or a second component which characterizes how much the classification result which occurs for the explanation map corresponds to the classification result which occurs for the image. ### Claim 16 (depends on 15) The method as recited in claim 15, wherein the second component is constituted by a Kullback-Leibler divergence. ### Claim 17 (depends on 15) The method as recited in claim 15, wherein the second component occurs pixel by pixel, due to a function which characterizes how much a particular pixel contributes to the classification result which occurs for the explanation map. ### Claim 18 (depends on 15) The method as recited in claim 15, wherein a secondary condition that the activation by the explanation map does not exceed the activation by the image for at least one layer in a deep neural network is implemented by a third component in the loss function, which penalizes an exceeding of the activation. ### Claim 19 (depends on 14) The method as recited in claim 14, wherein a secondary condition that the activation by the explanation map does not exceed the activation by the image is achieved in that the activation in the feature map is set to a predefinable value during the ascertainment of the explanation map, to a maximum permissible value, according to the secondary condition. ### Claim 20 (depends on 14) The method as recited in claim 14, wherein a secondary condition that the activation by the explanation map does not exceed the activation by the image is achieved in that a gradient-based optimization method is used, and a gradient of a backpropagation of a feature map of a layer of the deep neural network is set equal to a predefined gradient zero, when the activation by the explanation map exceeds the activation by the image in the layer. ### Claim 21 A method for displaying an image, the method comprising: ascertaining an explanation map of an image, by: highlighting all those pixels of the image which are significant for a classification of the image ascertained using a deep neural network, the explanation map being selected in such a way that the explanation map selects a smallest possible subset of the pixels of the image as relevant, and the explanation map leads to the same classification result as the image as when the explanation map is supplied to the deep neural network for classification, the explanation map being selected in such a way that an activation caused by the explanation map does not exceed an activation caused by the image in feature maps of the deep neural network; and displaying the explanation map. ### Claim 22 (depends on 21) The method as recited in claim 21, wherein the image is also displayed, next to the explanation map. ### Claim 23 A non-transitory machine-readable memory medium on which is stored a computer program for ascertaining an explanation map of an image, the computer program, when executed by a computer, causing the computer to perform the following: highlighting all those pixels of the image which are significant for a classification of the image ascertained using a deep neural network, the explanation map being selected in such a way that the explanation map selects a smallest possible subset of the pixels of the image as relevant, and the explanation map leads to the same classification result as the image as when the explanation map is supplied to the deep neural network for classification, the explanation map being selected in such a way that an activation caused by the explanation map does not exceed an activation caused by the image in feature maps of the deep neural network. ### Claim 24 A machine classification system configured to ascertain an explanation map of an image, the system configured to: highlight all those pixels of the image which are significant for a classification of the image ascertained using a deep neural network, the explanation map being selected in such a way that the explanation map selects a smallest possible subset of the pixels of the image as relevant, and the explanation map leads to the same classification result as the image as when the explanation map is supplied to the deep neural network for classification, the explanation map being selected in such a way that an activation caused by the explanation map does not exceed an activation caused by the image in feature maps of the deep neural network. ### Claim 25 A display system, configured to: ascertain an explanation map of an image, by: highlighting all those pixels of the image which are significant for a classification of the image ascertained using a deep neural network, the explanation map being selected in such a way that the explanation map selects a smallest possible subset of the pixels of the image as relevant, and the explanation map leads to the same classification result as the image as when the explanation map is supplied to the deep neural network for classification, the explanation map being selected in such a way that an activation caused by the explanation map does not exceed an activation caused by the image in feature maps of the deep neural network; and display the explanation map.