# US 11,308,368 B2 · Image processing using generative graphical models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,368 B2 - Document ID: US11308368B2 - Status: Granted - Kind: B2 - Application number: 17/056,400 - Filed: 2019-05-17 - Published: 2021-07-22 - Granted: 2022-04-19 - Earliest claimed: 2018-05-18 - Assignees: Five AI Limited - Applicants: Five AI Limited - Inventors: Sebastian Kaltwang; John Redford - Web page: https://patentgrep.com/patent/US11308368B2 ## CPC - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06F 18/2323: based on graph theory, e.g. minimum spanning trees [MST] or graph cuts - G06F 18/2415: based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate - G06F 18/29: {Graphical models, e.g. Bayesian networks} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06V 10/426: Graphical representations - G06V 10/7635: {based on graphs, e.g. graph cuts or spectral clustering} - G06V 10/84: using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract An image processing technique is presented using a hierarchical image model. The technique may be used as a precursor to subsequent image processing, to fix detected images in a post processing stage or as a segmentation or classification stage. The techniques may also be applied to super resolution. In a first layer of the hierarchical image model, each observed pixel of the image has a representation allocated to one or more input node. A set of the input nodes are assigned to a hidden node of a second layer, and a duplicate set of input nodes of the first layer is assigned to a duplicate of the hidden node in the second layer. In this way, a dense latent tree is formed in which a subtree is duplicated. Variables are assigned to the input nodes, the hidden node and the duplicate nodes and recurringly modified to process the image. Belief propagation messages may be utilised to recursively modify the variables. An image processing system using the method is described. A planning system for an autonomous vehicle comprising the image processing system is described. ## Claims ### Claim 1 A computer implemented method of processing an image comprising a plurality of pixels, using a hierarchical image model stored in computer memory wherein: in a first layer of the hierarchical image model each observed pixel of the image is allocated to one or more input nodes of the first layer, a set of the one or more input nodes is assigned to a hidden node of a second layer representing a part or subpart of the image in the second layer, to create a subtree for the hidden node, in which the hidden node is a parent node and the one or more input nodes are child nodes, a duplicate set of input nodes of the first layer is assigned to a duplicate of the hidden node in the second layer representing the same part or sub part of the image, thereby forming a dense latent tree in which the subtree is duplicated such that no child node has multiple parents in the dense latent tree yet all parent-child relationships between the first and second layer are represented, the method comprising, by at least one processor coupled to access the memory, assigning variables to the one or more input nodes, the hidden node and the duplicate node and recurringly modifying the variables to process the image. ### Claim 2 (depends on 1) The computer implemented method according to claim 1 wherein each observed pixel of the image is allocated to one or more input nodes of the first layer by storing a respective observation variable in a computer memory in association with an identifier of the one or more input nodes. ### Claim 3 (depends on 1) The computer implemented method according to claim 1 in which the image is incomplete and each non-observed pixel of the image is allocated to one or more input nodes of the first layer by storing a respective latent variable in a computer memory in association with an identifier of the one or more input nodes of the first layer to which each non-observed pixel corresponds. ### Claim 4 (depends on 1) The computer implemented method according to claim 1 comprising processing the image by sampling the modified variables to sample a probable image part defining a particular shape at a particular location in the image, the particular shape being generated at the second or a higher layer in the hierarchical image model, wherein all nodes in each duplicate set are constrained to take the same value during sampling. ### Claim 5 (depends on 1) The computer implemented method according to claim 1 comprising completing, in a precursor system, an incomplete image prior to subsequent processing, wherein the completing the incomplete image comprises recurringly modifying variables of input nodes to which no observation variable was assigned. ### Claim 6 (depends on 5) The computer implemented method according to claim 5 comprising the step of subsequently processing the complete image in a convolutional neural network or other artificial intelligent classification system. ### Claim 7 (depends on 1) The computer implemented method according to claim 1 comprising processing images output from a segmentation computer system to complete non-observed pixels of the images output from the segmentation computer system and/or take a different sample of probable images. ### Claim 8 (depends on 1) The computer implemented method according to claim 1 comprising classifying parts or subparts of a captured image representing a scene in which some parts of the scene are obscured in the captured image. ### Claim 9 (depends on 1) The computer implemented method according to claim 1 wherein the method of processing the image comprises generating a version of the image which has been received at a first resolution in a higher resolution, by assigning latent variables to some of the one or more input nodes of the hierarchical image model in the higher resolution. ### Claim 10 (depends on 1) The computer implemented method according to claim 1 in which the hierarchical image model has at least one or more further layers, wherein nodes in the second layer represent sub-parts of the image and nodes in the one or more further layers represents a part of the image comprising multiple sub-parts of the second layer. ### Claim 11 (depends on 1) The computer implemented method according to claim 1 comprising storing in a computer memory, a respective random variable, assigned to each hidden node of the second layer in a first step, and modifying the random variable in subsequent steps to converge on a set of probable variables for the hidden nodes of the second layer. ### Claim 12 A computer memory of an image processing system storing a generative graphical model for processing spatial relationships as a plurality of nodes arranged in layers, the generative graphical model comprising a plurality of kernels, each kernel having a group of input nodes for receiving observations in a lowest layer of the model, and a root node representing a latent variable in an upper layer of the generative graphical model, wherein the generative graphical model comprises additional nodes assigned as duplicate nodes to form a duplicate set of each of at least some of the input nodes, wherein the duplicate nodes are connected in a duplicate subtree resembling the subtree of the input node of which it is a duplicate whereby each input node is exclusive for a specific kernel and no node has multiple parents. ### Claim 13 (depends on 12) The computer memory of an image processing system storing the generative graphical model according to claim 12 wherein the computer memory further stores computer executable instructions that, when executed, cause a computer of the image processing system to process partly complete images to generate probable image data of parts or subparts of the partly complete image by recurringly modifying the latent variable of the root node. ### Claim 14 (depends on 13) The computer memory of the image processing system storing the generative graphical model according to claim 13 in which some input nodes in the lowest layer have observation values assigned to them, and other nodes in the lowest layer have latent variables assigned to them. ### Claim 15 (depends on 12) The computer memory of the image processing system storing the generative graphical model according to claim 12 in which all input nodes in the lowest layer have observations assigned to them. ### Claim 16 (depends on 12) A method of generating belief propagation messages in the generative graphical model stored in the computer memory of claim 12 in which each kernel has k child nodes in a child layer and a parent node in a parent layer, the method comprising for each kernel, determining for each state f of a child node at a first location a state product of (i) a kernel weight for each state f and (ii) a belief propagation message for a child node at a second location displaced by k nodes from the first location; summing the state products over all states in the child layer to generate a state sum of for each child node; generating for each parent node a kernel product of the state sums, the kernel product being a unique belief propagation message to propagate to the parent node. ### Claim 17 (depends on 16) The method according to claim 16, wherein the kernel weight for each state f is the conditional probability that x is in that state f when y is in state g, where x is an observed or latent variable of a child node at a first location and y is the latent variable of the parent node. ### Claim 18 (depends on 16) The method according to claim 16 comprising sampling a probable image part defining a particular shape at a particular location in the image, the particular shape being generated at a parent layer by propagation of the belief propagation messages between layers. ### Claim 19 (depends on 12) A method of training a neural network using the generative graphical model stored in the computer memory as claimed in claim 12, in which each kernel has k child nodes in a child layer and a parent node in a parent layer, the method comprising for each kernel: generating a set of unique belief propagation messages for propagation from each child node in a child layer to its parent node in a parent layer, the set of unique belief propagation messages comprising a unique belief propagation message for a root node of the generative graphical model; calculating a marginal likelihood of the generative graphical model by summing over all possible states in the root node of the generative graphical model and calculating a product of a prior weight of the root node and the belief propagation message for the root node; and optimising the marginal likelihood using a neural network optimisation algorithm. ### Claim 20 An image processing computer system comprising one or more computers programmed to process an image comprising a plurality of pixels, the one or more computers comprising at least one processor arranged to execute one or more computer programs and computer memory accessible by the one or more processors for storing variables allocated to one or more nodes of a hierarchical image model representing the image, the computer memory storing the hierarchical image model wherein: in a first layer of the hierarchical image model each observed pixel of the image is allocated to one or more input nodes of the first layer, a set of the one or more input nodes is assigned to a hidden node of a second layer representing a part or subpart of the image in the second layer, to create a subtree for the hidden node, in which the hidden node is a parent node and the one or more input nodes are child nodes; and a duplicate set of input nodes of the first layer is assigned to a duplicate of the hidden node in the second layer representing the same part or sub part of the image, thereby forming a dense latent tree in which the subtree is duplicated such that no child node has multiple parents in the dense latent tree yet all parent-child relationships between the first and second layer are represented; and the at least one processor coupled to access the memory to assign variables to the one or more input nodes, the hidden node and the duplicate node and recurringly modifying the variables to process the image.