# US 2022/0036562 A1 · Vision-based working area boundary detection system and method, and machine equipment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036562 A1
- Document ID: US20220036562A1
- Status: Published application
- Kind: A1
- Application number: 17/309,406
- Filed: 2019-01-18
- Published: 2022-02-03
- Earliest claimed: 2018-11-27
- Assignees: Bongos Robotics Shanghai Co., LTD
- Applicants: Bongos Robotics Shanghai Co., LTD
- Inventors: Yifei WU; Wei ZHANG; Xinliang BAO
- Web page: https://patentgrep.com/patent/US20220036562A1

## CPC

- G06T 7/11: Region-based segmentation
- G06T 7/12: Edge-based segmentation
- G05D 1/02: Control of position or course in two dimensions
- G06T 7/90: Determination of colour characteristics
- G06V 10/82: using neural networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The invention discloses a vision-based working area boundary detection system, method and machine equipment. When implementing the solution, we firstly train the constructed neural network model based on the training data set, extract and learn the features of the corresponding working area. Next, the neural network after training and learning is employed to perform real-time image semantic segmentation on the collected video images based on the extracted features, thereby perceiving the environment and identifying the boundaries of the working area. The solution provided by the present invention is based on neural network technology, and can efficiently identify the boundary of the working area through the extraction and learning of the features of the working area in the early stage, and has strong robustness to environmental perturbation such as lighting.

## Claims

### Claim 1

A vision-based working area boundaries detection system including a processor and a computer-readable medium storing a computer program; wherein when the computer program is executed by the processor:
a constructed neural network model performs autonomous training and learning based on a training data set, extracts and learns features of a corresponding working area;
the neural network after training performs real-time image semantic segmentation on collected video images based on the extracted features extracted by the training and learning, thereby perceiving the environment and identifying boundaries of the working area.

### Claim 2 (depends on 1)

The vision-based detection system of a working area boundary according to claim 1, wherein the neural network model in the vision-based detection system includes multiple convolution layers, multiple pooling layers, and an output layer; the convolution layers and the pooling layers are stacked to extract image features; all layers update their parameters during a training phase, and outputs the image real time segmentation result during an inference phase.

### Claim 3 (depends on 2)

The vision-based detection system of a working area boundary according to claim 2, wherein the pooling layers performs statistics and data dimensionality reduction on the output features of the convolutional layers, along the direction of the image rows and columns; the statistics are carried out, and the maximum value of N pixels is extracted as the statistical characteristics of the area, and the data volume is reduced to one-Nth of the original.

### Claim 4 (depends on 2)

The vision-based detection system of a working area boundary according to claim 2, wherein the neural network model also includes a dilated convolution layer module, which is formed by a number of dilated convolution layers in parallel and is arranged after the pooling layer; the dilated convolution layer which insert holes into the original convolution layer, expanding a receptive field of feature extraction and retaining a global information of the image.

### Claim 5 (depends on 4)

The vision-based detection system of a working area boundary according to claim 4, wherein the neural network model includes up-sampling layers, which are arranged in front of the output layer, aiming at raising a size of the reduced image and restoring the detailed content of the image.

### Claim 6

A vision-based working area boundary detection method wherein:
a constructed neural network model performs autonomous training and learning based on a training data set, and extracts features of a corresponding working area;
after a training and learning phase, the system performs real-time image semantic segmentation on collected video images based on the extracted features, thereby perceiving the environment and identifying boundaries of the working area.

### Claim 7 (depends on 6)

The method for detecting a vision-based work area boundary according to claim 6, wherein the training data is formed by acquiring pictures of real outdoor working scenes, pre-processing the pictures and labeling pictures manually according to the category of the target object.

### Claim 8 (depends on 6)

The method for detecting a vision-based work area boundary according to claim 6, wherein the neural network model is trained based on training data set, including
initialization, determine a number of layers and initial parameters of each layer;
feed the images in the training set into an initialized neural network model for parameter calculation;
compare output of neural network with the image label, calculate the error and update the values of the neural network parameters;
repeat the above steps until the training error is minimum, then complete training phase.

### Claim 9 (depends on 6)

The method for detecting a vision-based work area boundary according to claim 6, wherein the method includes:
the trained deep neural network model performs feature extraction on the video images collected in real time;
the deep neural network model performs data statistics and dimensionality reduction on the extracted feature data;
the deep neural network model outputs image segmentation images through model inference.

### Claim 10 (depends on 9)

The method for detecting a vision-based work area boundary according to claim 9, when the deep neural network model performs model inference, the probability of each pixel in the real-time input image belonging to each category in the training set is calculated and each pixel is marked as the category with the highest probability; the segmented image is obtained after all pixels are labeled.

### Claim 11 (depends on 9)

The method for detecting a vision-based work area boundary according to claim 9, when the deep neural network model performs model inference to obtain segmented images, the same color denotes the same classification; in the segmented image, the boundary line between the target classification color and other color blocks is exactly the working area that needs to be detected.

### Claim 12

A vision-based working area boundary detection machine is equipped with the above vision-based detection system for the boundary of the working area.
