# US 11,308,351 B2 · Method and apparatus for recognizing sequence in image, electronic device, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,351 B2
- Document ID: US11308351B2
- Status: Granted
- Kind: B2
- Application number: 16/848,680
- Filed: 2020-04-14
- Published: 2021-03-11
- Granted: 2022-04-19
- Earliest claimed: 2019-09-05
- Assignees: Sensetime International Pte. Ltd.
- Applicants: Sensetime International Pte. Ltd.
- Inventors: Wenyang Hu; Xiaocong Cai; Jun Hou; Shuai Yi
- Web page: https://patentgrep.com/patent/US11308351B2

## CPC

- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06F 18/243: relating to the number of classes
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06T 7/11: Region-based segmentation
- G06V 10/422: for representing the structure of the pattern or shape of an object therefor
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present disclosure relates to a method and apparatus for recognizing a sequence in an image, an electronic device, and a storage medium. The method includes: performing feature extraction on a to-be-processed image to obtain a first feature map of the to-be-processed image, where the to-be-processed image includes a sequence formed by stacking at least one object along a stacking direction; determining a region feature of each segmented region in the to-be-processed image based on the first feature map, where all segmented regions are obtained by dividing the to-be-processed image into k regions along the stacking direction, k is a set number of objects stacked along the stacking direction, and k is an integer greater than 1; and determining a category of each object in the sequence based on the region feature of each segmented region. Embodiments of the present disclosure may implement recognition of stacked objects in a sequence.

## Claims

### Claim 1

A method for recognizing a sequence in an image, comprising:
performing feature extraction on a to-be-processed image to obtain a first feature map of the to-be-processed image, wherein the to-be-processed image comprises a sequence formed by stacking a plurality of objects along a stacking direction;
determining a region feature of each segmented region of a plurality of segmented regions in the to-be-processed image based on the first feature map, wherein all segmented regions of the plurality of segmented regions are obtained by dividing the to-be-processed image into k regions along the stacking direction, k is a set number of objects stacked along the stacking direction, and k is an integer greater than 1; and
determining a category of each object in the sequence based on the region feature of each segmented region, wherein the determining a category of each object in the sequence based on the region feature of each segmented region further comprises:
determining, based on the region feature of a first segmented region, a probability that an object in the first segmented region belongs to each of a plurality of object categories, wherein the first segmented region is any one of all the segmented regions, and
determining an object category with a maximum probability among the plurality of object categories as an object category to which the object in the first segmented region belongs.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the determining, based on the region feature of a first segmented region, a probability that an object in the first segmented region belongs to each of a plurality of object categories is implemented by a classification network, the classification network comprises a first convolution layer and a softmax layer, the number of convolution kernels in the first convolution layer is equal to the number of set object categories, a size of the convolution kernels is 1×N, N is a length of the region feature of each segmented region, and N is an integer greater than 1; and
the determining, based on the region feature of a first segmented region, a probability that an object in the first segmented region belongs to each of a plurality of object categories comprises:
performing convolution processing on the region feature of the first segmented region by using each convolution kernel in the first convolution layer respectively, to determine a category feature belonging to each of the plurality of object categories among features of the first segmented region; and
processing the category feature by using the softmax layer in the classification network, to determine a probability that the features of the first segmented region belong to each of the plurality of object categories.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the step of performing feature extraction on a to-be-processed image to obtain a first feature map of the to-be-processed image is implemented by a feature extraction network; the feature extraction network comprises a second convolution layer and a pooling layer; and
the performing feature extraction on a to-be-processed image to obtain a first feature map of the to-be-processed image comprises:
performing feature extraction on the to-be-processed image by using the second convolution layer, to obtain a second feature map; and
performing pooling processing on the second feature map by using the pooling layer, to obtain the first feature map, wherein
a feature dimension of the first feature map is less than a feature dimension of the second feature map.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the to-be-processed image comprises an image of a surface of an object constituting the sequence along the stacking direction, and the stacking direction is a thickness direction of the object in the sequence.

### Claim 5 (depends on 1)

The method according to claim 1, wherein at least one object among the plurality of objects in the sequence is a sheet-like object with a certain thickness.

### Claim 6 (depends on 5)

The method according to claim 5, wherein a surface of the at least one object in the sequence along the stacking direction has a set identifier, and the identifier comprises at least one of a color, a texture, or a pattern.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the to-be-processed image is obtained by cropping the sequence from a collected image, and one end of the sequence is aligned with one edge of the to-be-processed image.

### Claim 8 (depends on 7)

The method according to claim 7, wherein the plurality of objects in the sequence have a same thickness, and a length of the to-be-processed image along the stacking direction is k times the thickness of the objects in the sequence, or
the plurality of objects in the sequence have different thicknesses, and the length of the to-be-processed image along the stacking direction is k times a maximum thickness of an object of a possible category in the sequence.

### Claim 9 (depends on 1)

The method according to claim 1, further comprising:
after the category of each object in the sequence is determined, determining a total value represented by the sequence based on a correspondence between the category and a value represented by the category.

### Claim 10

An apparatus for recognizing a sequence in an image, comprising:
a processor; and
a memory configured to store processor executable instructions, wherein
the processor is configured to invoke the instructions stored in the memory, to:
perform feature extraction on a to-be-processed image to obtain a first feature map of the to-be-processed image, wherein the to-be-processed image comprises a sequence formed by stacking a plurality of objects along a stacking direction;
determine a region feature of each segmented region of a plurality of segmented regions in the to-be-processed image based on the first feature map, wherein all segmented regions of the plurality of segmented regions are obtained by dividing the to-be-processed image into k regions along the stacking direction, k is a set number of objects stacked along the stacking direction, and k is an integer greater than 1; and
determine a category of each object in the sequence based on the region feature of each segmented region, wherein the determining a category of each object in the sequence based on the region feature of each segmented region further comprises:
determining, based on the region feature of a first segmented region, a probability that an object in the first segmented region belongs to each of a plurality of object categories, wherein the first segmented region is any one of all the segmented regions, and
determining an object category with a maximum probability among the plurality of object categories as an object category to which the object in the first segmented region belongs.

### Claim 11 (depends on 10)

The apparatus according to claim 10, wherein step of determining, based on the region feature of the first segmented region, the probability that an object in the first segmented region belongs to each of a plurality of object categories is implemented by a classification network, the classification network comprises a first convolution layer and a softmax layer, the number of convolution kernels in the first convolution layer is equal to the number of set object categories, a size of the convolution kernels is 1×N, N is a length of the region feature of each segmented region, and N is an integer greater than 1; and
the determining, based on the region feature of the first segmented region, the probability that an object in the first segmented region belongs to each of a plurality of object categories comprises:
performing convolution processing on the region feature of the first segmented region by using each convolution kernel in the first convolution layer respectively, to determine a category feature belonging to each set object category among features of the first segmented region; and
processing the category feature by using the softmax layer in the classification network, to determine a probability that the features of the first segmented region belong to each set object category.

### Claim 12 (depends on 10)

The apparatus according to claim 10, wherein the step of performing feature extraction on the to-be-processed image to obtain the first feature map of the to-be-processed image is implemented by a feature extraction network; the feature extraction network comprises a second convolution layer and a pooling layer; and
performing feature extraction on the to-be-processed image to obtain the first feature map of the to-be-processed image comprises:
performing feature extraction on the to-be-processed image by using the second convolution layer, to obtain a second feature map; and
performing pooling processing on the second feature map by using the pooling layer, to obtain the first feature map, wherein
a feature dimension of the first feature map is less than a feature dimension of the second feature map.

### Claim 13 (depends on 10)

The apparatus according to claim 10, wherein the to-be-processed image comprises an image of a surface of an object constituting the sequence along the stacking direction, and the stacking direction is a thickness direction of the object in the sequence.

### Claim 14 (depends on 10)

The apparatus according to claim 10, wherein a surface of at least one object among the plurality of objects in the sequence along the stacking direction has a set identifier, and the identifier comprises at least one of a color, a texture, or a pattern.

### Claim 15 (depends on 10)

The apparatus according to claim 10, wherein the to-be-processed image is obtained by cropping the sequence from a collected image, and one end of the sequence is aligned with one edge of the to-be-processed image.

### Claim 16 (depends on 15)

The apparatus according to claim 15, wherein the plurality of objects in the sequence have a same thickness, and a length of the to-be-processed image along the stacking direction is k times the thickness of the objects in the sequence, or
the plurality of objects in the sequence have different thicknesses, and the length of the to-be-processed image along the stacking direction is k times a maximum thickness of an object of a possible category in the sequence.

### Claim 17 (depends on 10)

The apparatus according to claim 10, wherein the processor is further configured to:
after the category of each object in the sequence is determined, determine a total value represented by the sequence based on a correspondence between the category and a value represented by the category.

### Claim 18

A non-transitory computer readable storage medium having computer program instructions stored thereon, wherein the computer program instructions upon execution by a processor cause the processor to perform operations, the operations comprising:
performing feature extraction on a to-be-processed image to obtain a first feature map of the to-be-processed image, wherein the to-be-processed image comprises a sequence formed by stacking a plurality of objects along a stacking direction;
determining a region feature of each segmented region of a plurality of segmented regions in the to-be-processed image based on the first feature map, wherein all segmented regions of the plurality of segmented regions are obtained by dividing the to-be-processed image into k regions along the stacking direction, k is a set number of objects stacked along the stacking direction, and k is an integer greater than 1; and
determining a category of each object in the sequence based on the region feature of each segmented region, wherein the determining a category of each object in the sequence based on the region feature of each segmented region further comprises:
determining, based on the region feature of a first segmented region, a probability that an object in the first segmented region belongs to each of a plurality of object categories, wherein the first segmented region is any one of all the segmented regions, and
determining an object category with a maximum probability among the plurality of object categories as an object category to which the object in the first segmented region belongs.