# US 12,008,476 B2 · Image segmentation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,476 B2
- Document ID: US12008476B2
- Status: Granted
- Kind: B2
- Application number: 18/084,784
- Filed: 2022-12-20
- Published: 2023-04-20
- Granted: 2024-06-11
- Earliest claimed: 2018-07-06
- Assignees: Five AI Limited
- Applicants: Five AI Limited
- Inventors: John Redford; Sina Samangooei
- Web page: https://patentgrep.com/patent/US12008476B2

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F 18/2115: by evaluating different subsets according to an optimisation criterion, e.g. class separability, forward selection or backward elimination
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/24323: {Tree-organised classifiers}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/09: Supervised learning
- G06V 10/40: Extraction of image or video features
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/771: Feature selection, e.g. selecting representative features from a multi-dimensional feature space
- G06V 10/82: using neural networks
- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

In one aspect, hierarchical image segmentation is applied to an image formed of a plurality of pixels, by classifying the pixels according to a hierarchical classification scheme, in which at least some of those pixels are classified by a parent level classifier in relation to a set of parent classes, each of which is associated with a subset of child classes, and each of those pixels is also classified by at least one child level classifier in relation to one of the subsets of child classes, wherein each of the parent classes corresponds to a category of visible structure, and each of the subset of child classes associated with it corresponds to a different type of visible structure within that category.

## Claims

### Claim 1

A computer system comprising:
one or more hardware processors configured to implement:
an input layer configured to receive an image formed of a plurality of pixels, and
a plurality of classifiers configured to classify the plurality of pixels according to a hierarchical classification scheme, in which at least some of those pixels are classified by a parent level classifier of the plurality of classifiers in relation to a set p of parent classes wherein each parent class Ai of the set of parent classes is associated with a subset q of child classes, wherein each of those pixels is also classified by at least one child level classifier of the plurality of classifiers in relation to one of the subsets of child classes, wherein each of the parent classes Ai corresponds to a category of visible structure, and each child class Bj of the subset q of child classes associated with it corresponds to a different type of visible structure within that category;
wherein the child level classifier is configured to determine an absolute child classification probability Pr(Px,y∈Bj) of that pixel belonging to a child class Bj of that subset of child classes, and the parent level classifier is configured to determine a parent classification probability Pr(Px,y∈Ai) of that pixel belonging to the parent class Ai with which that subset of child classes is associated;
wherein the child level classifier is configured to determine for that pixel a conditional child classification probability Pr(Px,y∈Bj|Px,y∈Ai) of the pixel Px,y belonging to that child class Bj, which is determined independently of the parent level classifier and the parent classification probability Pr(Px,y∈Ai), on an assumption that the pixel Px,y belongs to the parent class Ai; and
wherein the child level classifier is configured to determine the absolute child classification probability Pr(Px,y∈Bj) by multiplying the parent classification probability Pr(Px,y∈Ai), with the conditional classification probability Pr(Px,y∈Bj|Px,y∈Ai) according to:
Pr(Px,y∈Bj)=Pr(Px,y∈Bj|Px,y∈Ai)×Pr(Px,y∈Ai).

### Claim 2 (depends on 1)

The computer system of claim 1, wherein the one or more hardware processors are configured to perform a decision-making process based on the classifications of the plurality of pixels.

### Claim 3 (depends on 2)

The computer system of claim 2, wherein the one or more hardware processors are configured, in performing the decision-making process, to detect a failure of the child level classifier to classify at least one of the pixels, and make a decision based on the classification of that pixel by the parent level classifier.

### Claim 4 (depends on 2)

The computer system of claim 2, wherein the decision-making process is an autonomous vehicle control process.

### Claim 5 (depends on 1)

The computer system of claim 1, wherein the one or more hardware processors are configured to implement a feature extractor, the feature extractor configured to generate a plurality of pixel feature vectors ƒ, each representing one of the pixels Px,y, based on that pixel and a set of the pixels related thereto, wherein the pixels are classified based on their pixel feature vectors.

### Claim 6 (depends on 5)

The computer system of claim 5, wherein the set of related pixels is formed of the pixels in a portion of the image surrounding that pixel.

### Claim 7 (depends on 5)

The computer system of claim 5, wherein the feature extractor is formed of one or more layers of a neural network.

### Claim 8 (depends on 7)

The computer system of claim 7, wherein the feature extractor is formed of multiple layers of a convolutional neural network.

### Claim 9 (depends on 5)

The computer system of claim 5, wherein the one or more hardware processors are configured to process each feature vector according to feature vector modification parameters in order to generate:
a modified feature vector ƒp for the set of parent classes, which is used to classify the pixel it represents in relation to the set p of parent classes; and
a modified feature vector for each subset q of child classes, which is used to classify that pixel in relation to that subset of child classes.

### Claim 10 (depends on 9)

The computer system of claim 9, wherein the one or more hardware processors are configured to combine the modified feature vector ƒq for each subset q of child classes with the modified feature vector ƒp for the set of parent classes to determine a combined feature vector ƒq′ for that subset q of child classes, which is used to classify that pixel in relation to that subset.

### Claim 11 (depends on 10)

The computer system of claim 10, wherein the one or more hardware processors are configured to combine the modified feature vectors by concatenation.

### Claim 12 (depends on 1)

The computer system of claim 1, wherein the parent level classifier and the child level classifier are probabilistic classifiers, which determine a probability distribution over the set p of parent classes and the subset q of child classes respectively.

### Claim 13 (depends on 1)

The computer system of claim 1, wherein the one or more hardware processors are configured to classify each of those pixels by at least one of: a first child level classifier, in relation to a first subset q1 of child classes associated with a first parent class A1 of the parent classes, and a second child level classifier, in relation to a second subset q2 of child classes associated with a second parent class A2 of the parent classes.

### Claim 14 (depends on 1)

The computer system of claim 1, wherein the one or more hardware processors are configured to classify each of at least some of the pixels by the parent level classifier, the child level classifier, and at least one further child level classifier in relation to a further set r of child classes, the further child level classifier being a child of the child level classifier in that each class in the further set r of child classes is a child of a child class in the subset q of child classes within the hierarchical classification scheme.

### Claim 15

A computer-implemented method of training image pixel classifiers, the method comprising:
receiving an image formed of a plurality of pixels;
receiving labelling data for labelling at least some of the pixels according to a hierarchical classification scheme, in which each of those pixels is labelled as belonging to one of a set p of parent classes Ai, each of which is associated with a subset q of child classes Bj, and each of those pixels is also labelled as belonging to one of the subset of child classes associated with the parent class to which it belongs, wherein each of the parent classes Ai corresponds to a category of visible structure, and each of the subset of child classes Bj associated with it corresponds to a different type of visible structure within that category; and
using data of the pixels and the labelling data to train a parent level classifier to classify image pixels in relation to the set p of parent classes, and at least one child level classifier to classify image pixels in relation to the subset q of child classes associated with one of the parent classes;
wherein the child level classifier is trained to determine an absolute child classification probability Pr(Px,y∈Bj) of that pixel belonging to a child class Bj of that subset of child classes, and the parent level classifier is trained to determine a parent classification probability Pr(Px,y∈Ai) of that pixel belonging to the parent class Ai with which that subset of child classes is associated; and
wherein the child level classifier is trained to determine for that pixel a conditional child classification probability Pr(Px,y∈Bj|Px,y∈Ai) of the pixel Px,y belonging to that child class Bj, which is determined independently of the parent level classifier and the parent classification probability Pr(Px,y∈Ai), on an assumption that the pixel Px,y belongs to the parent class Ai;
wherein the child level classifier is trained to determine the child classification probability Pr(Px,y∈Bj) by multiplying the parent classification probability Pr(Px,y∈Ai), with the conditional classification probability Pr(Px,y∈Bj|Px,y∈Ai) according to:
Pr(Px,y∈Bj)=Pr(Px,y∈Bj|Px,y∈Ai)×Pr(Px,y∈Ai).

### Claim 16 (depends on 15)

A method according to claim 15, wherein the classifiers are trained by using the classifiers to classify each of the pixels in relation to the set of parent classes and each subset of child classes, determining respective error signals for the set of parent classes and each subset of child classes respectively, and adapting parameters of the classifiers to minimize the error signals;
wherein the error signal for the set of parent classes provides a measure of difference between the label data for the set of parent classes and the results of the pixel classifications performed in relation to the set of parent classes; and
wherein the error signal for each subset of child classes provides a measure of difference between the label data for that subset of child classes and the results of the pixel classifications performed in relation to that subset of child classes, wherein any pixels which are not labelled in relation that subset of child class does not contribute to the error signal for that subset of child classes.

### Claim 17 (depends on 16)

A method according to claim 16, wherein the data of the pixels comprise a plurality of pixel feature vectors, each representing one of the pixels, as generated by a feature extractor based on that pixel and a set of the pixels related thereto.

### Claim 18 (depends on 17)

A method according to claim 17, wherein parameters of the feature extractor are adapted in training along with the parameters of the classifiers in order to minimize the error signals.

### Claim 19 (depends on 16)

A method according to claim 16 or any claim dependent thereon, wherein the error signals are combined in a cost function which is minimized in training.

### Claim 20

A computer program comprising computer readable instructions stored on a non-transitory storage medium and configured, when executed on one or more processors, to cause the one or more processors to implement:
an input layer configured to receive an image formed of a plurality of pixels, and
a plurality of classifiers configured to classify the pixels according to a hierarchical classification scheme, in which at least some of those pixels are classified by a parent level classifier of the plurality of classifiers in relation to a set of parent classes, each of which is associated with a subset of child classes, and each of those pixels is also classified by at least one child level classifier of the plurality of classifiers in relation to one of the subsets of child classes, wherein each of the parent classes corresponds to a category of visible structure, and each of the subset of child classes associated with it corresponds to a different type of visible structure within that category;
wherein the child level classifier is configured to determine an absolute child classification probability Pr(Px,y∈Bj) of that pixel belonging to a child class Bj of that subset of child classes, and the parent level classifier is configured to determine a parent classification probability Pr(Px,y∈Ai) of that pixel belonging to the parent class Ai with which that subset of child classes is associated;
wherein the child level classifier is configured to determine for that pixel a conditional child classification probability Pr(Px,y∈Bj|Px,y∈Ai) of the pixel Px,y belonging to that child class Bj, which is determined independently of the parent level classifier and the parent classification probability Pr(Px,y∈Ai), on the assumption that the pixel Px,y belongs to the parent class Ai;
wherein the child level classifier is configured to determine the child classification probability Pr(Px,y∈Bj) by multiplying the parent classification probability Pr(Px,y∈Ai), with the conditional classification probability Pr(Px,y∈Bj|Px,y∈Ai) according to:
Pr(Px,y∈Bj)=Pr(Px,y∈Bj|Px,y∈Ai)×Pr(Px,y∈Ai).