# US 2019/0377949 A1 · Image Processing Method, Electronic Device and Computer Readable Storage Medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0377949 A1
- Document ID: US20190377949A1
- Status: Published application
- Kind: A1
- Application number: 16/249,515
- Filed: 2019-01-16
- Published: 2019-12-12
- Earliest claimed: 2018-06-08
- Assignees: Guangdong Oppo Mobile Telecommunications Corp., Ltd.
- Applicants: Guangdong OPPO Mobile Telecommunications Corp., Ltd.
- Inventors: Yan Chen
- Web page: https://patentgrep.com/patent/US20190377949A1

## CPC

- G06V 10/764: using classification, e.g. of video objects
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06V 10/82: using neural networks
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 20/35: {Categorising the entire scene, e.g. birthday party or wedding scene}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An image processing method, an electronic device and a computer readable storage medium are described. The image processing method may include: obtaining an image to be detected, and inputting the image to be detected to a neural network; detecting the image to be detected via the neural network to obtain confidence levels of image categories for a background image in the image and confidence levels of object categories for a foreground target in the image; determining an image category of the background image according to the confidence levels of the image categories for the background image and a confidence level threshold; and determining an object category of the foreground target according to the confidence levels of the object categories for the foreground target and the confidence level threshold.

## Claims

### Claim 1

A method for image processing, comprising:
obtaining an image to be detected, and inputting the image to be detected to a neural network;
detecting the image to be detected via the neural network to obtain confidence levels of image categories for a background image in the image and confidence levels of object categories for a foreground target in the image;
determining an image category of the background image according to the confidence levels of the image categories for the background image and a confidence level threshold; and
determining an object category of the foreground target according to the confidence levels of the object categories for the foreground target and the confidence level threshold.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the neural network comprises at least one input layer, at least one base layer, at least one classification layer, and at least one target detection layer; wherein, the input layer is cascaded to the base layer, and the base layer is cascaded to the classification layer and the target detection layer;
wherein, inputting the image to be detected to the neural network comprises: inputting the image to be detected to the input layer of the neural network; and
wherein, detecting the image to be detected via the neural network to obtain confidence levels of image categories for the background image in the image and confidence levels of object categories for the foreground target in the image comprises:
extracting one or more features of the image to be detected via the base layer of the neural network, and inputting the one or more extracted image features to the classification layer and the target detection layer;
performing category detection on the background image in the image at the classification layer to obtain the confidence levels of the image categories for the background image; and
performing target detection on the foreground target in the image at the target detection layer to obtain the confidence levels of the object categories for the foreground target.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the neural network further comprises a first output layer cascaded to the classification layer and a second output layer cascaded to the target detection layer; and
the method further comprises:
outputting by the first output layer the confidence levels of the image categories for the background image; and
outputting by the second output layer the confidence levels of the object categories for the foreground target.

### Claim 4 (depends on 3)

The method according to claim 3, further comprising: inputting a training image pre-labeled with an image category and an object category to the neural network to train the neural network.

### Claim 5 (depends on 4)

The method according to claim 4, wherein inputting the training image pre-labeled with the image category and the object category to the neural network to train the neural network comprises:
inputting the training image to the input layer of the neural network;
extracting one or more features of the training image via the base layer and inputting the one or more extracted image features to the classification layer and the target detection layer;
obtaining a first loss function at the classification layer, wherein the first loss function indicates a difference between a first predicted confidence level and a first true confidence level, the first predicted confidence level is a confidence level, which is predicted by the neural network, of an image category of a background image in the training image, and the first true confidence level is a true confidence level of the image category pre-labeled in the training image;
obtaining a second loss function at the target detection layer, wherein, the second loss function indicates a difference between a second predicted confidence level and a second true confidence level, the second predicted confidence level is a confidence level, which is predicted by the neural network, of an object category of a foreground target in the training image, and the second true confidence level is a true confidence level of the object category pre-labeled in the training image;
calculating a weighted summation of the first loss function and the second loss function to obtain a target loss function; and
adjusting one or more parameters of the neural network according to the target loss function to minimize the first loss function and the second loss function.

### Claim 6 (depends on 2)

The method according to claim 2, further comprising: performing the target detection on the foreground target in the image at the target detection layer to obtain a position of the foreground target.

### Claim 7 (depends on 3)

The method according to claim 3, further comprising: inputting a training image pre-labeled with an image category, an object category and a true boundary box corresponding to the object category to the neural network to train the neural network.

### Claim 8 (depends on 7)

The method according to claim 7, wherein inputting a training image pre-labeled with an image category, an object category and a true boundary box corresponding to the object category to the neural network to train the neural network comprises:
inputting the training image to the input layer of the neural network;
extracting one or more features of the training image via the base layer, and inputting the one or more extracted image features to the classification layer and the target detection layer;
obtaining a first loss function at the classification layer, wherein the first loss function indicates a difference between a first predicted confidence level and a first true confidence level;
the first predicted confidence level is a confidence level, which is predicted by the neural network, of an image category of a background image in the training image, and the first true confidence level is a true confidence level of the image category pre-labeled in the training image;
obtaining a second loss function and a position loss function at the target detection layer, wherein the second loss function indicates a difference between a second predicted confidence level and a second true confidence level, the second predicted confidence level is a confidence level, which is predicted by the neural network, of an object category of a foreground target in the training image, and the second true confidence level is a true confidence level of the object category pre-labeled in the training image; the position loss function is constructed according to offset parameters of each pre-selected default boundary box relative to a true boundary box corresponding to the object category;
calculating a weighted summation of the first loss function, the second loss function and the position loss function to obtain a target loss function; and
adjusting one or more parameters of the neural network according to the target loss function to minimize the first loss function, the second loss function, and the position loss function.

### Claim 9 (depends on 8)

The method according to claim 8, wherein the offset parameters of each pre-selected default boundary box relative to the true boundary box corresponding to the object category comprises: coordinates of a center point of the default boundary box, a width offset of the default boundary box relative to the true boundary box, and a height offset of the default boundary box relative to the true boundary box.

### Claim 10 (depends on 8)

The method according to claim 8, further comprising: outputting by the second output layer the offset parameters of each pre-selected default boundary box relative to the true boundary box corresponding to the object category.

### Claim 11 (depends on 3)

The method according to claim 3, wherein the base layer comprises a Visual Geometry Group (VGG) network, the classification layer comprises a Mobilenet layer, the target detection layer comprises a Single Shot Multibox Detector (SSD) network, and the first output layer and the second output layer respectively comprise a softmax classifier.

### Claim 12 (depends on 1)

The method according to claim 1, further comprising:
constructing a neural network comprising a first branch and a second branch;
configuring the first branch to output confidence levels, which are detected by the neural network, of the image categories for the background image; and
configuring the second branch to output confidence levels, which are detected by the neural network, of the object categories for the foreground target, or configuring the second branch to output confidence levels, which are detected by the neural network, of the object categories for the foreground target and offset parameters, which are detected by the neural network, of each pre-selected default boundary box relative to a true boundary box corresponding to an object category.

### Claim 13

An electronic device, the electronic device comprising a processor and a memory storing a computer program that is executable by the processor to perform actions of:
obtaining an image to be detected, and inputting the image to be detected to a neural network;
detecting the image to be detected via the neural network to obtain confidence levels of image categories for a background image in the image and confidence levels of object categories for a foreground target in the image;
determining an image category of the background image according to the confidence levels of the image categories for the background image and a confidence level threshold; and
determining an object category of the foreground target according to the confidence levels of the object categories for the foreground target and the confidence level threshold.

### Claim 14 (depends on 13)

The electronic device according to claim 13, wherein the neural network comprises at least one input layer, at least one base layer, at least one classification layer, and at least one target detection layer; wherein, the input layer is cascaded to the base layer, and the base layer is cascaded to the classification layer and the target detection layer; and
when the computer program is executed by the processor, the processor further performs actions of:
inputting the image to be detected to the input layer of the neural network;
extracting one or more features of the image to be detected via the base layer of the neural network, and inputting the one or more extracted image features to the classification layer and the target detection layer;
performing category detection on the background image in the image at the classification layer to obtain the confidence levels of the image categories for the background image; and
performing target detection on the foreground target in the image at the target detection layer to obtain the confidence levels of the object categories for the foreground target.

### Claim 15 (depends on 14)

The electronic device according to claim 14, wherein the neural network further comprises a first output layer connected to the classification layer and a second output layer connected to the target detection layer; and
when the computer program is executed by the processor, the processor further performs actions of:
outputting at the first output layer the confidence levels of the image categories for the background image; and
outputting at the second output layer the confidence levels of the object categories for the foreground target.

### Claim 16 (depends on 15)

The electronic device according to claim 15, wherein when the computer program is executed by the processor, the processor further performs actions of: inputting a training image pre-labeled with an image category and an object category to the neural network to train the neural network.

### Claim 17 (depends on 16)

The electronic device according to claim 16, wherein inputting the training image pre-labeled with the image category and the object category to the neural network to train the neural network comprises:
inputting the training image to the input layer of the neural network;
extracting one or more features of the training image via the base layer and inputting the one or more extracted image features to the classification layer and the target detection layer;
obtaining a first loss function at the classification layer, wherein the first loss function indicates a difference between a first predicted confidence level and a first true confidence level, the first predicted confidence level is a confidence level, which is predicted by the neural network, of an image category of a background image in the training image, and the first true confidence level is a true confidence level of the image category pre-labeled in the training image;
obtaining a second loss function at the target detection layer, wherein, the second loss function indicates a difference between a second predicted confidence level and a second true confidence level, the second predicted confidence level is a confidence level, which is predicted by the neural network, of an object category of a foreground target in the training image, and the second true confidence level is a true confidence level of the object category pre-labeled in the training image;
calculating a weighted summation of the first loss function and the second loss function to obtain a target loss function; and
adjusting one or more parameters of the neural network according to the target loss function to minimize the first loss function and the second loss function.

### Claim 18 (depends on 15)

The electronic device according to claim 15, wherein, when the computer program is executed by the processor, the processor further performs actions of: inputting a training image pre-labeled with an image category, an object category and a true boundary box corresponding to the object category to the neural network to train the neural network.

### Claim 19 (depends on 18)

The electronic device according to claim 18, wherein inputting a training image pre-labeled with an image category, an object category and a true boundary box corresponding to the object category to the neural network to train the neural network comprises:
inputting the training image to the input layer of the neural network;
extracting one or more features of the training image via the base layer, and inputting the one or more extracted image features to the classification layer and the target detection layer;
obtaining a first loss function at the classification layer, wherein the first loss function indicates a difference between a first predicted confidence level and a first true confidence level;
the first predicted confidence level is a confidence level, which is predicted by the neural network, of an image category of a background image in the training image, and the first true confidence level is a true confidence level of the image category pre-labeled in the training image;
obtaining a second loss function and a position loss function at the target detection layer, wherein the second loss function indicates a difference between a second predicted confidence level and a second true confidence level, the second predicted confidence level is a confidence level, which is predicted by the neural network, of an object category of a foreground target in the training image, and the second true confidence level is a true confidence level of the object category pre-labeled in the training image; the position loss function is constructed according to offset parameters of each pre-selected default boundary box relative to a true boundary box corresponding to the object category;
calculating a weighted summation of the first loss function, the second loss function and the position loss function to obtain a target loss function; and
adjusting one or more parameters of the neural network according to the target loss function to minimize the first loss function, the second loss function, and the position loss function.

### Claim 20

A non-transitory computer readable storage medium having stored thereon a computer program that is executable by a processor to perform actions of:
obtaining an image to be detected, and inputting the image to be detected to a neural network;
detecting the image to be detected via the neural network to obtain confidence levels of image categories for a background image in the image and confidence levels of object categories for a foreground target in the image;
determining an image category of the background image according to the confidence levels of the image categories for the background image and a confidence level threshold; and
determining an object category of the foreground target according to the confidence levels of the object categories for the foreground target and the confidence level threshold.
