# US 10,872,275 B2 · Semantic segmentation based on a hierarchy of neural networks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,872,275 B2 - Document ID: US10872275B2 - Status: Granted - Kind: B2 - Application number: 16/361,433 - Filed: 2019-03-22 - Published: 2020-09-24 - Granted: 2020-12-22 - Earliest claimed: 2019-03-22 - Assignees: Nokia Technologies Oy - Applicants: Nokia Technologies Oy - Inventors: Tinghuai Wang - Web page: https://patentgrep.com/patent/US10872275B2 ## CPC - G06T 7/12: Edge-based segmentation - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/048: Activation functions - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06V 10/82: using neural networks - G06V 30/19173: {Classification techniques} - G06V 30/274: {Syntactic or semantic context, e.g. balancing} - G06T 2207/20084: Artificial neural networks [ANN] - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Aspects described herein relate to various methods, systems and apparatuses that may improve the accuracy of object classifications and object boundary definitions for a semantic segmentation technique. For example, the semantic segmentation technique may be based on a hierarchy of two or more layers. The two or more layers may include neural networks that analyze image data at different resolution scales. Each layer of the hierarchy may determine object boundary features and object class features. Each layer of the hierarchy may share its object boundary features and/or its object class features with one or more other layers in the hierarchy. In turn, each of other layers of the hierarchy may determine its object boundary features and/or its object class features based on the shared features. ## Claims ### Claim 1 A method comprising: determining, for a second layer of a hierarchy of neural networks, one or more initial object class features and one or more initial object boundary features; determining, for the second layer, based on the one or more initial object class features and one or more third layer object class features received from a third layer of the hierarchy, one or more second layer object class features; determining, for the second layer, based on the one or more initial object boundary features and one or more first layer object boundary features received from a first layer of the hierarchy, one or more second layer object boundary features; determining, for the second layer, based on the one or more second layer object class features and the one or more second layer object boundary features, one or more fused object class and object boundary features; determining, for the second layer, based on the one or more initial object class features and the one or more fused object class and object boundary features, segmentation data; and determining, based on the segmentation data and additional segmentation data associated with the second layer or the third layer, hierarchy-based segmentation data. ### Claim 2 (depends on 1) The method of claim 1, wherein the neural networks include convolutional neural networks (CNNs). ### Claim 3 (depends on 1) The method of claim 1, wherein determining the one or more second layer object class features is performed based on one or more object class feature weights, and wherein the method further comprises: determining, based on the one or more third layer object class features and the one or more initial object class features, the one or more object class feature weights. ### Claim 4 (depends on 1) The method of claim 1, wherein determining the one or more second layer object boundary features is performed based on one or more object boundary feature weights, and wherein the method further comprises: determining, based on the one or more second layer object boundary features and the one or more initial object boundary features, the one or more object boundary feature weights. ### Claim 5 (depends on 1) The method of claim 1, wherein determining the one or more fused object class and object boundary features is performed based on one or more fusion weights, and wherein the method further comprises: determining, based on the one or more second layer object class features and the one or more second layer object boundary features, the one or more fusion weights. ### Claim 6 (depends on 1) The method of claim 1, further comprising: sending, to the third layer, the one or more second layer object class features; and sending, to the first layer, the one or more second layer object boundary features. ### Claim 7 (depends on 1) The method of claim 1, further comprising: storing and/or sending the hierarchy-based segmentation data to enable access by one or more of a video conferencing service, a smart home service, an Internet-of-Things (IoT) service, or an autonomous driving service. ### Claim 8 An apparatus comprising: one or more processors; and memory storing executable instructions that, when executed by the one or more processors, cause the apparatus to at least: determine, for a second layer of a hierarchy of neural networks, one or more initial object class features and one or more initial object boundary features; determine, for the second layer, based on the one or more initial object class features and one or more third layer object class features received from a third layer of the hierarchy, one or more second layer object class features; determine, for the second layer, based on the one or more initial object boundary features and one or more first layer object boundary features received from a first layer of the hierarchy, one or more second layer object boundary features; determine, for the second layer, based on the one or more second layer object class features and the one or more second layer object boundary features, one or more fused object class and object boundary features; determine, for the second layer, based on the one or more initial object class features and the one or more fused object class and object boundary features, segmentation data; and determine, based on the segmentation data and additional segmentation data associated with the second layer or the third layer, hierarchy-based segmentation data. ### Claim 9 (depends on 8) The apparatus of claim 8, wherein the neural networks include convolutional neural networks (CNNs). ### Claim 10 (depends on 8) The apparatus of claim 8, wherein the executable instructions, when executed by the one or more processors, cause the apparatus to: determine, based on the one or more third layer object class features and the one or more initial object class features, one or more object class feature weights; and determine the one or more second layer object class features based on the one or more object class feature weights. ### Claim 11 (depends on 8) The apparatus of claim 8, wherein the executable instructions, when executed by the one or more processors, cause the apparatus to: determine, based on the one or more second layer object boundary features and the one or more initial object boundary features, one or more object boundary feature weights; and determine the one or more second layer object boundary features based on the one or more object boundary feature weights. ### Claim 12 (depends on 8) The apparatus of claim 8, wherein the executable instructions, when executed by the one or more processors, cause the apparatus to: determine, based on the one or more second layer object class features and the one or more second layer object boundary features, one or more fusion weights; and determine the one or more fused object class and one or more object boundary features based on the one or more fusion weights. ### Claim 13 (depends on 8) The apparatus of claim 8, wherein the executable instructions, when executed by the one or more processors, cause the apparatus to: send, to the third layer, the one or more second layer object class features; and send, to the first layer, the one or more second layer object boundary features. ### Claim 14 (depends on 8) The apparatus of claim 8, wherein the executable instructions, when executed by the one or more processors, cause the apparatus to: store and/or send the hierarchy-based segmentation data to enable access by one or more of a video conferencing service, a smart home service, an Internet-of-Things (IoT) service, or an autonomous driving service. ### Claim 15 One or more non-transitory computer-readable media storing executable instructions that, when executed, cause an apparatus to at least: determine, for a second layer of a hierarchy of neural networks, one or more initial object class features and one or more initial object boundary features; determine, for the second layer, based on the one or more initial object class features and one or more third layer object class features received from a third layer of the hierarchy, one or more second layer object class features; determine, for the second layer, based on the one or more initial object boundary features and one or more first layer object boundary features received from a first layer of the hierarchy, one or more second layer object boundary features; determine, for the second layer, based on the one or more second layer object class features and the one or more second layer object boundary features, one or more fused object class and object boundary features; determine, for the second layer, based on the one or more initial object class features and the one or more fused object class and object boundary features, segmentation data; and determine, based on the segmentation data and additional segmentation data associated with the second layer or the third layer, hierarchy-based segmentation data. ### Claim 16 (depends on 15) The one or more non-transitory computer-readable media of claim 15, wherein the neural networks include convolutional neural networks (CNNs). ### Claim 17 (depends on 15) The one or more non-transitory computer-readable media of claim 15, wherein the executable instructions, when executed, cause the apparatus to: determine, based on the one or more third layer object class features and the one or more initial object class features, one or more object class feature weights; and determine the one or more second layer object class features based on the one or more object class feature weights. ### Claim 18 (depends on 15) The one or more non-transitory computer-readable media of claim 15, wherein the executable instructions, when executed, cause the apparatus to: determine, based on the one or more second layer object boundary features and the one or more initial object boundary features, one or more object boundary feature weights; and determine the one or more second layer object boundary features based on the one or more object boundary feature weights. ### Claim 19 (depends on 15) The one or more non-transitory computer-readable media of claim 15, wherein the executable instructions, when executed, cause the apparatus to: determine, based on the one or more second layer object class features and the one or more second layer object boundary features, one or more fusion weights; and determine the one or more fused object class and one or more object boundary features based on the one or more fusion weights. ### Claim 20 (depends on 15) The one or more non-transitory computer-readable media of claim 15, wherein the executable instructions, when executed, cause the apparatus to: send, to the third layer, the one or more second layer object class features; and send, to the first layer, the one or more second layer object boundary features.