# US 10,769,744 B2 · Computer vision system and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,769,744 B2
- Document ID: US10769744B2
- Status: Granted
- Kind: B2
- Application number: 16/176,801
- Filed: 2018-10-31
- Published: 2020-04-30
- Granted: 2020-09-08
- Earliest claimed: 2018-10-31
- Assignees: Kabushiki Kaisha Toshiba
- Applicants: Kabushiki Kaisha Toshiba
- Inventors: Rudra Prasad Poudel Karmatha; Ujwal Bonde; Stephan Liwicki; Christopher Zach
- Web page: https://patentgrep.com/patent/US10769744B2

## CPC

- G06T 7/11: Region-based segmentation
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06T 1/0014: {Image feed-back for automatic industrial control, e.g. robot with camera (robots B25J19/023)}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 20/64: Three-dimensional [3D] objects
- G06T 2207/10024: Color image
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An image processing method for segmenting an image, the method comprising: receiving first image; producing a second image from said first image, wherein said second image is a lower resolution representation of said first image; processing said first image with a first processing stage to produce a first feature map; processing said second image with a second processing stage to produce a second feature map; and combining the first feature map with the second feature map to produce a semantic segmented image; wherein the first processing stage comprises a first neural network comprising at least one separable convolution module configured to perform separable convolution and said second processing stage comprises a second neural network comprising at least one separable convolution module configured to perform separable convolution; the number of layers in the first neural network being smaller than the number of layers in the second neural network.

## Claims

### Claim 1

An image processing method for segmenting an image, the method comprising:
receiving first image;
producing a second image from said first image, wherein said second image is a lower resolution representation of said first image;
processing said first image with a first processing stage to produce a first feature map;
processing said second image with a second processing stage to produce a second feature map; and
combining the first feature map with the second feature map to produce a semantic segmented image;
wherein the first processing stage comprises a first neural network comprising at least one separable convolution module configured to perform separable convolution and said second processing stage comprises a second neural network comprising at least one separable convolution module configured to perform separable convolution; a number of layers in the first neural network being smaller than a number of layers in the second neural network.

### Claim 2 (depends on 1)

An image processing method according to claim 1, wherein the first feature map and the second feature map are combined at just one stage.

### Claim 3 (depends on 2)

An image processing method according to claim 2, wherein the first feature map and the second feature map are combined by adding.

### Claim 4 (depends on 2)

An image processing method according to claim 2, wherein the first feature map and the second feature map are combined at a fusion stage, wherein said fusion stage comprises upsampling the second feature map and adding the first feature map to the second feature map wherein adding comprises adding corresponding values of the upsampled second feature map and the first feature map.

### Claim 5 (depends on 4)

An image processing method according to claim 4, wherein upsampled second feature map is subjected to depthwise convolution prior to adding to the first feature map, wherein the depthwise convolution is performed with a dilation factor that is greater than 1.

### Claim 6 (depends on 5)

An image processing method according to claim 5, wherein upsampled second feature map has been subjected to depthwise convolution with a dilation factor that is greater than 1 and the first feature map are subjected to two dimensional convolution prior to adding.

### Claim 7 (depends on 1)

An image processing method according to claim 1, wherein separable convolution modules in the second processing stage are depthwise convolution modules.

### Claim 8 (depends on 1)

An image processing method according to claim 1, wherein separable convolution modules in the second processing stage are bottleneck architecture modules.

### Claim 9 (depends on 1)

An image processing method according to claim 1, wherein separable convolution modules in the second processing stage are bottleneck residual architecture modules.

### Claim 10 (depends on 1)

An image processing method according to claim 1, wherein the second image is subjected to standard convolution prior to be processed by said separable convolution modules.

### Claim 11 (depends on 9)

An image processing method, according to claim 9, wherein there are a plurality of residual bottleneck architecture modules.

### Claim 12 (depends on 11)

An image processing method according to claim 11, wherein a final standard convolution module is provided after the residual bottleneck architecture modules.

### Claim 13 (depends on 12)

An image processing method according to claim 12, wherein a pyramid pooling module is provided between the final standard convolution module and the residual bottleneck architecture modules.

### Claim 14 (depends on 1)

An image processing method according to claim 1, wherein the first processing stage comprises a plurality of depthwise separable convolution modules.

### Claim 15

A method of training a model, said model for segmenting an image, the model comprising:
a first neural network having a plurality of layers comprising at least one separable convolution module configured to perform separable convolutions;
a second neural network with a plurality of layers, comprising at least one separable convolution module configured to perform separable convolutions; a number of layers in the first neural network being smaller than a number of layers in the second neural network,
the first neural network being configured to process an image at a first resolution and the second neural network being configured to process the same image at a lower resolution,
a feature map of the first and second neural networks being combined by addition at a single stage;
the training method comprising:
providing training data, the training data comprising images and semantic segmented information concerning said images;
training said model using said images as input and the semantic segmented information as output, wherein two stages are trained together.

### Claim 16 (depends on 15)

A method according to claim 15, further comprising increasing a number of filters during training, and reducing their number to discard filters of lower importance.

### Claim 17 (depends on 15)

A method according to claim 15, further comprising adapting the model during training to add a second output on said second processing stage, the method further comprising training using the images as input and determining loss by comparing with both the semantic segmented information at both the output and at the second output and updating weights during training by using the determined losses from both outputs.

### Claim 18

An image processing system for segmenting an image, the system comprising:
an interface and a processor,
said interface having an image input and being adapted to receive a first image, said processor being adapted to;
produce a second image from said first image, wherein said second image is a lower resolution representation of said first image;
process said first image with a first processing stage to produce a first feature map;
process said second image with a second processing stage to produce a second feature map; and
combine the first feature map with the second feature map to produce a semantic segmented image;
wherein the first processing stage comprises a first neural network comprising at least one separable convolution module configured to perform separable convolution and said second processing stage comprises a second neural network comprising at least one separable convolution module configured to perform separable convolution; a number of layers in the first neural network being smaller than a number of layers in the second neural network.

### Claim 19 (depends on 18)

A detection system for a vehicle, said detection system, comprising the image processing system of claim 18, being adapted to receive an image and determine objects from said image by segmenting said image.

### Claim 20 (depends on 1)

A non-transitory carrier medium carrying computer readable instructions adapted to cause a computer to perform the method of claim 1.