# US 10,872,227 B2 · Automatic object recognition method and system thereof, shopping device and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,872,227 B2 - Document ID: US10872227B2 - Status: Granted - Kind: B2 - Application number: 16/114,907 - Filed: 2018-08-28 - Published: 2019-10-03 - Granted: 2020-12-22 - Earliest claimed: 2018-03-29 - Assignees: Boe Technology Group Co., Ltd. - Applicants: Boe Technology Group Co., Ltd. - Inventors: Tong Liu; Xiangyang Liu; Xiaojun Tang; Zhiguo Zhang - Web page: https://patentgrep.com/patent/US10872227B2 ## CPC - G06V 20/64: Three-dimensional [3D] objects - G06T 7/38: Registration of image sequences - G06T 7/50: Depth or shape recovery - G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching} - G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines - G06F 18/24143: {Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]} - G06F 18/24147: {Distances to closest patterns, e.g. nearest neighbour classification} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract An automatic object recognition method and an automatic object recognition system, a shopping device and a non-transitory computer readable storage medium are disclosed. The automatic object recognition method includes: acquiring a first image and a second image, both the first image and the second image including a same object, and the second image being a depth image; extracting a feature point of the object based on the first image; obtaining a joint feature based on the feature point from the first image and depth information of the second image; and recognizing the object based on the joint feature. ## Claims ### Claim 1 An automatic object recognition method, comprising: acquiring a first image and a second image, both the first image and the second image comprising a same object, and the second image being a depth image; extracting a feature point of the object based on the first image; obtaining a joint feature based on the feature point from the first image and depth information of the second image; and recognizing the object based on the joint feature, wherein obtaining the joint feature based on the feature point from the first image and depth information of the second image includes computing a three-dimensional feature corresponding to the feature point based on the feature point and the depth information of the second image, and merging the three-dimensional feature and the feature point to obtain the joint feature; wherein computing the three-dimensional feature corresponding to the feature point based on the feature point and the depth information of the second image includes establishing a three-dimensional coordinate system of the feature point based on the feature point and the depth information of the second image, and computing the three-dimensional feature corresponding to the feature point according to the three-dimensional coordinate system of the feature point; and wherein establishing the three-dimensional coordinate system of the feature point based on the feature point and the depth information of the second image includes performing a registration operation on the first image and the second image to obtain registration information between the first image and the second image, the registration information including the depth information of the second image, determining a feature surface of the object according to the feature point, computing three-dimensional information of feature description points of the feature surface based on the registration information, and establishing the three-dimensional coordinate system of the feature point based on the three-dimensional information of the feature description points and the feature point. ### Claim 2 (depends on 1) The automatic object recognition method according to claim 1, wherein extracting the feature point of the object based on the first image comprises: acquiring a target region of the first image, the target region comprising the object; and extracting the feature point of the object within the target region. ### Claim 3 (depends on 2) The automatic object recognition method according to claim 2, wherein acquiring the target region of the first image comprises: performing image segmentation processing on the first image to obtain the target region of the first image; or performing image segmentation processing on the second image to obtain a depth target region of the second image, performing a registration operation on the first image and the second image to obtain registration information between the first image and the second image, and obtaining the target region of the first image based on the registration information and the depth target region. ### Claim 4 (depends on 1) The automatic object recognition method according to claim 1, wherein establishing the three-dimensional coordinate system of the feature point based on the three-dimensional information of the feature description points and the feature point comprises: computing a normal vector at the feature point according to the three-dimensional information of the feature description points, a direction of the normal vector being taken as a first-axis direction of the three-dimensional coordinate system; and obtaining, according to the normal vector, a tangent plane at the feature point and principal directions of two principal curvatures of the feature point on the tangent plane, the principal directions of the two principal curvatures being perpendicular to each other, and the principal directions of the two principal curvatures being respectively taken as a second-axis direction and a third-axis direction of the three-dimensional coordinate system. ### Claim 5 (depends on 1) The automatic object recognition method according to claim 1, wherein computing the three-dimensional feature corresponding to the feature point according to the three-dimensional coordinate system of the feature point comprises: computing coordinates of the feature description points of the feature surface in the three-dimensional coordinate system; and computing the three-dimensional feature corresponding to the feature point according to a coordinate of the feature point in the three-dimensional coordinate system and the coordinates of the feature description points in the three-dimensional coordinate system. ### Claim 6 (depends on 1) The automatic object recognition method according to claim 1, wherein merging the three-dimensional feature and the feature point to obtain the joint feature comprises: merging a dimension of the three-dimensional feature and a dimension of the feature point to obtain the joint feature. ### Claim 7 (depends on 1) The automatic object recognition method according to claim 1, wherein recognizing the object based on the joint feature comprises: using a classifier to compute, based on the joint feature, a plurality of scores when the object belongs to different categories; and determining a recognition result of the object according to the plurality of scores. ### Claim 8 (depends on 7) The automatic object recognition method according to claim 7, further comprising: outputting the recognition result of the object. ### Claim 9 (depends on 1) The automatic object recognition method according to claim 1, wherein the first image is a color image. ### Claim 10 An automatic object recognition system, comprising: an image acquisition module, configured to acquire a first image and a second image, both the first image and the second image comprising a same object, and the second image being a depth image; a feature extraction module, configured to extract a feature point of the object based on the first image; a feature joint module, configured to obtain a joint feature based on the feature point from the first image and depth information of the second image; and a recognition module, configured to recognize the object based on the joint feature; wherein the feature joint module is further configured to compute a three-dimensional feature corresponding to the feature point based on the feature point and the depth information of the second image, and merge the three-dimensional feature and the feature point to obtain the joint feature; the feature joint module is further configured to establish a three-dimensional coordinate system of the feature point based on the feature point and the depth information of the second image, and compute the three-dimensional feature corresponding to the feature point according to the three-dimensional coordinate system of the feature point; and the feature joint module is further configured to perform a registration operation on the first image and the second image to obtain registration information between the first image and the second image, the registration information including the depth information of the second image, determine a feature surface of the object according to the feature point, compute three-dimensional information of feature description points of the feature surface based on the registration information, and establish the three-dimensional coordinate system of the feature point based on the three-dimensional information of the feature description points and the feature point. ### Claim 11 (depends on 10) The automatic object recognition system according to claim 10, wherein an operation of extracting the feature point of the object based on the first image comprises: acquiring a target region of the first image, the target region comprising the object; and extracting the feature point of the object within the target region. ### Claim 12 (depends on 10) The automatic object recognition system according to claim 10, wherein an operation of obtaining the joint feature based on the feature point from the first image and depth information of the second image comprises: computing a three-dimensional feature corresponding to the feature point based on the feature point and the depth information of the second image; and merging the three-dimensional feature and the feature point to obtain the joint feature. ### Claim 13 An automatic object recognition system, comprising: an image acquisition device, configured to acquire a first image and a second image, both the first image and the second image comprising a same object, and the second image being a depth image; a non-transitory memory, configured to store computer readable instructions; and a processor, configured to execute the computer readable instructions, wherein when the computer readable instructions are executed by the processor, the processor performs: acquiring a first image and a second image, both the first image and the second image comprising a same object, and the second image being a depth image; extracting a feature point of the object based on the first image; obtaining a joint feature based on the feature point from the first image and depth information of the second image; and recognizing the object based on the joint feature, wherein obtaining the joint feature based on the feature point from the first image and depth information of the second image includes computing a three-dimensional feature corresponding to the feature point based on the feature point and the depth information of the second image, and merging the three-dimensional feature and the feature point to obtain the joint feature; wherein computing the three-dimensional feature corresponding to the feature point based on the feature point and the depth information of the second image includes establishing a three-dimensional coordinate system of the feature point based on the feature point and the depth information of the second image, and computing the three-dimensional feature corresponding to the feature point according to the three-dimensional coordinate system of the feature point; wherein establishing the three-dimensional coordinate system of the feature point based on the feature point and the depth information of the second image includes performing a registration operation on the first image and the second image to obtain registration information between the first image and the second image, the registration information including the depth information of the second image, determining a feature surface of the object according to the feature point, computing three-dimensional information of feature description points of the feature surface based on the registration information, and establishing the three-dimensional coordinate system of the feature point based on the three-dimensional information of the feature description points and the feature point. ### Claim 14 (depends on 13) The automatic object recognition system according to claim 13, wherein the image acquisition device comprises a depth camera. ### Claim 15 (depends on 13) The automatic object recognition system according to claim 13, further comprising an output device, wherein the output device is configured to output a recognition result of the object. ### Claim 16 (depends on 13) A shopping device, comprising a loading region and the automatic object recognition system according to claim 13, wherein the automatic object recognition system is provided on the loading region. ### Claim 17 (depends on 1) A non-transitory computer readable storage medium, for storing computer readable instructions, wherein when the computer readable instructions are executed by a computer, the computer performs the automatic object recognition method according to claim 1.