# US 11,907,846 B2 · Learning and propagating visual attributes > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,846 B2 - Document ID: US11907846B2 - Status: Granted - Kind: B2 - Application number: 17/017,597 - Filed: 2020-09-10 - Published: 2022-03-10 - Granted: 2024-02-20 - Earliest claimed: 2020-09-10 - Assignees: NVIDIA Corporation - Applicants: Nvidia Corporation - Inventors: Sifei Liu; Shalini De Mello; Varun Jampani; Jan Kautz; Xueting Li - Web page: https://patentgrep.com/patent/US11907846B2 ## CPC - G06N 3/084: Backpropagation, e.g. using gradient descent - G06V 10/761: {Proximity, similarity or dissimilarity measures} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/2163: {Partitioning the feature space} - G06F 18/22: Matching criteria, e.g. proximity measures - G06F 18/29: {Graphical models, e.g. Bayesian networks} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06V 10/765: {using rules for classification or partitioning the feature space} - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/82: using neural networks - G06N 3/048: Activation functions - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract One embodiment of the present invention sets forth a technique for performing spatial propagation. The technique includes generating a first directed acyclic graph (DAG) by connecting spatially adjacent points included in a set of unstructured points via directed edges along a first direction. The technique also includes applying a first set of neural network layers to one or more images associated with the set of unstructured points to generate (i) a set of features for the set of unstructured points and (ii) a set of pairwise affinities between the spatially adjacent points connected by the directed edges. The technique further includes generating a set of labels for the set of unstructured points by propagating the set of features across the first DAG based on the set of pairwise affinities. ## Claims ### Claim 1 A method for performing spatial propagation, comprising: generating a first structure by connecting two or more points included in a set of spatially distributed points; applying a first set of neural network layers to one or more images associated with the set of spatially distributed points to generate (i) a set of features for the set of spatially distributed points and (ii) a set of affinities between the two or more points; and generating a set of labels for the set of spatially distributed points by propagating the set of features across the first structure based on the set of affinities. ### Claim 2 (depends on 1) The method of claim 1, further comprising: generating one or more structures by connecting two or more additional points included in the set of spatially distributed points via one or more edges along one or more directions; and generating the set of labels by propagating the set of features across the one or more structures based on the set of affinities for the two or more additional points. ### Claim 3 (depends on 2) The method of claim 2, wherein generating the set of labels comprises: aggregating a first result of propagating the set of features across the first structure and one or more results of propagating the set of features across the one or more structures into a set of feature vectors for the set of spatially distributed points; and applying a second set of neural network layers to the set of feature vectors to generate the set of labels. ### Claim 4 (depends on 2) The method of claim 2, wherein the one or more directions comprise a second direction that is orthogonal to the first direction and a third direction that is opposite the first direction. ### Claim 5 (depends on 1) The method of claim 1, wherein applying the first set of neural network layers to the one or more images comprises applying a sequence of residual blocks to an image to produce a first unary feature for a first pixel in the image. ### Claim 6 (depends on 5) The method of claim 5, wherein applying the first set of neural network layers to the one or more images further comprises applying a set of affinity blocks to a first set of outputs of the set of residual blocks to produce a first pairwise feature for the first pixel. ### Claim 7 (depends on 6) The method of claim 6, wherein applying the first set of neural network layers to the one or more images further comprises applying a similarity kernel to the first pairwise feature and a second pairwise feature for a second pixel in the image to produce a pairwise affinity between the first pixel and the second pixel. ### Claim 8 (depends on 6) The method of claim 6, wherein applying the set of affinity blocks to the first set of outputs of the set of residual blocks comprises producing the first pairwise feature as a weighted sum of a second set of outputs of the set of affinity blocks. ### Claim 9 (depends on 1) The method of claim 1, wherein applying the first set of neural network layers to the one or more images comprises: aggregating a subset of the set of features for a set of pixels in the one or more images that correspond to a first spatially distributed point into one or more pointwise features for the first spatially distributed point; and calculating a pairwise affinity between the first spatially distributed point and a second spatially distributed point based on a similarity between a first pairwise feature in the one or more pointwise features and a second pairwise feature for the second unstructured point. ### Claim 10 (depends on 1) The method of claim 1, wherein propagating the set of features across the first structure comprises updating a feature for a first point in the first structure based on a sum of a subset of the set of features for a subset of the set of spatially distributed points with outgoing edges to the first point in the first structure weighted by a subset of the set of pairwise affinities between the subset of the set of spatially distributed points and the first point. ### Claim 11 (depends on 1) The method of claim 1, wherein propagating the set of features across the first structure comprises: dividing the set of spatially distributed points into a plurality of groups of points based on a topology of the first structure, wherein each group included in the plurality of groups of points comprises (i) a subset of the spatially distributed points that are not linked to one another in the first structure and (ii) incoming edges from one or more groups included in the plurality of groups that precede the group along the first direction in the first structure; and propagating in parallel a subset of the set of features from a first group in the plurality of groups to a second group in the plurality of groups that is connected to the first group by a subset of the one or more edges. ### Claim 12 (depends on 1) The method of claim 1, wherein connecting the two or more points comprises connecting two or more spatially adjacent points included in the set of spatially distributed points via directed edges along a first direction. ### Claim 13 (depends on 1) The method of claim 1, wherein the first structure comprises a directed acyclic graph (DAG). ### Claim 14 (depends on 1) The method of claim 1, wherein the set of spatially distributed points comprises at least one of a point cloud or one or more superpixels. ### Claim 15 (depends on 1) The method of claim 1, wherein the set of labels comprises at least one of a class label or a color. ### Claim 16 A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform the steps of: generating a first directed acyclic graph (DAG) by connecting spatially adjacent points included in a set of unstructured points via directed edges along a first direction; applying a first set of neural network layers to one or more images associated with the set of unstructured points to generate (i) a set of features for the set of unstructured points and (ii) a set of pairwise affinities between the spatially adjacent points connected by the directed edges; and generating a set of labels for the set of unstructured points by propagating the set of features across the first DAG based on the set of pairwise affinities. ### Claim 17 (depends on 16) The non-transitory computer readable medium of claim 16, wherein the steps further comprise: generating one or more DAGs by connecting the spatially adjacent points included in the set of unstructured points via additional directed edges along one or more directions; and generating the set of labels by propagating the set of features across the one or more DAGs based on the set of pairwise affinities for the additional directed edges. ### Claim 18 (depends on 17) The non-transitory computer readable medium of claim 17, wherein generating the set of labels comprises: aggregating a first result of propagating the set of features across the first DAG and one or more results of propagating the set of features across the one or more DAGs into a set of feature vectors for the set of unstructured points; and applying a second set of neural network layers to the set of feature vectors to generate the set of labels. ### Claim 19 (depends on 16) The non-transitory computer readable medium of claim 16, wherein applying the first set of neural network layers to the one or more images comprises: applying a sequence of residual blocks to an image to produce a first unary feature for a first pixel in the image; applying a set of affinity blocks to a first set of outputs of the set of residual blocks to produce a first pairwise feature for the first pixel; and applying a similarity kernel to the first pairwise feature and a second pairwise feature for a second pixel in the image to produce a pairwise affinity between the first pixel and the second pixel. ### Claim 20 (depends on 19) The non-transitory computer readable medium of claim 19, wherein the similarity kernel comprises at least one of an inner product and an embedded Gaussian. ### Claim 21 (depends on 16) The non-transitory computer readable medium of claim 16, wherein applying the first set of neural network layers to the one or more images comprises: generating the set of features in parallel on a plurality of processors; and aggregating the set of features into the set of pairwise affinities between the spatially adjacent points connected by the directed edges. ### Claim 22 A system, comprising: a memory that stores instructions, and a processor that is coupled to the memory and, when executing the instructions, is configured to: generate a first directed acyclic graph (DAG) by connecting spatially adjacent points included in a set of spatially distributed points via directed edges along a first direction; apply a first set of neural network layers to one or more images associated with the set of spatially distributed points to generate (i) a set of features for the set of spatially distributed points and (ii) a set of pairwise affinities between the spatially adjacent points connected by the directed edges; and generate a set of labels for the set of spatially distributed points by propagating the set of features across the first DAG based on the set of pairwise affinities.