# US 11,113,680 B2 · Self-service checkout counter checkout > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,113,680 B2 - Document ID: US11113680B2 - Status: Granted - Kind: B2 - Application number: 16/810,670 - Filed: 2020-03-05 - Published: 2020-10-22 - Granted: 2021-09-07 - Earliest claimed: 2019-04-16 - Assignees: Advanced New Technologies Co., Ltd. - Applicants: Advanced New Technologies Co., Ltd. - Inventors: Yang Song - Web page: https://patentgrep.com/patent/US11113680B2 ## CPC - G06Q 20/201: {Price look-up processing, e.g. updating} - G06T 7/12: Edge-based segmentation - G06K 7/1404: {Methods for optical code recognition} - G06K 7/1443: {locating of the code in an image} - G06K 7/1491: {the method including a reconstruction step, e.g. stitching two pieces of bar code together to derive the full bar code} - G06Q 20/18: involving self-service terminals [SST], vending machines, kiosks or multimedia terminals - G06Q 20/208: {Input by product or record sensing, e.g. weighing or scanner processing} - G06T 7/11: Region-based segmentation - G07G 1/0054: {with control of supplementary check-parameters, e.g. weight or number of articles} - G07G 1/0063: {with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration} - G06T 2207/20081: Training; Learning - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS - G07G: REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS ## Abstract Computer-implemented methods, non-transitory, computer-readable media, and computer-implemented systems for self-service checkout counter checkout. One computer-implemented method includes: obtaining, by using a camera, an image of at least one product placed on a checkout counter; performing image segmentation on the image to obtain at least one image region; identifying a product code included in a code region in an image region of the at least one image region; determining, based on the product code, a product category of a product associated with the product code; and determining a price of the product based on the product category. ## Claims ### Claim 1 A computer-implemented method for self-service checkout, comprising: obtaining, by using a camera, an image, wherein the image is of at least one product placed on a checkout counter; performing image segmentation on the image to obtain at least one image region; identifying a product code included in a code region in an image region of the at least one image region; determining that a product category of a product associated with the product code fails to be identified; in response to determining that the product category fails to be identified, determining the product category by using an object detection model obtained by pre-training using labeled training sample images, each labeled training sample image comprising a rectangle that frames the product code to label the code region, wherein the object detection model generates a candidate image region in the image, performs target identification within the candidate image region, performs bounding box regression in the candidate image region, and corrects the code region by using a perspective transformation relative to position markers to obtain a corrected code region, and determines the product category based on the corrected code region; and determining a price of the product based on the product category. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein obtaining the image comprises: controlling the camera to face the at least one product to obtain the image. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein obtaining the image comprises receiving the image from a self-service checkout counter. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein the image is taken from one of a top direction, a front direction, a left direction, a right direction, a rear direction, and an oblique direction of the at least one product, and an angle between the oblique direction and a direction vertical to a countertop of the self-service checkout counter is between 30 degrees to 60 degrees. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the image segmentation is performed by using an image segmentation model, and the image segmentation model is obtained by pre-training segmentation sample images comprising product images and annotations of silhouettes of products. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the product code is a barcode, and identifying the product code comprises: detecting a barcode region in the image region by using an object detection model, the object detection model is obtained by pre-training training sample images comprising product images and annotations for identifying barcode regions in the product images. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein identifying the product code comprises: adjusting the barcode region based on the perspective transformation to obtain an adjusted barcode region; and identifying the barcode in the adjusted barcode region. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the product code is a two-dimensional code, and identifying the product code comprises: detecting position markers of the two-dimensional code in the image region; and determining that a two-dimensional code region is detected in response to detecting at least two positioning markers. ### Claim 9 (depends on 8) The computer-implemented method of claim 8, wherein identifying the product code comprises: adjusting the two-dimensional code region based on the perspective transformation to obtain an adjusted two-dimensional code; determining a corner-module relationship in the adjusted two-dimensional code based on the at least two positioning markers; and identifying the two-dimensional code based on the corner-module relationship. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein identifying the product category of the product associated with the product further comprises: determining the product category based on an object detection model, wherein the object detection model is obtained by pre-training training sample images comprising product images and annotations for identifying products and product categories. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein the image is a first image, the camera is a first camera, the at least one image region is a first at least one image region, the product is a first product, and the method further comprising: obtaining, by using a second camera, a second image, wherein the second image is of the at least one product; performing image segmentation on the second image to obtain a second at least one image region; determining a product category of a second product associated with the second at least one image region based on a product code or visual identification; and determining, based on a relative position between the first camera and the second camera, that the first at least one image region and the second at least one image region are associated with an identical product. ### Claim 12 A computer-implemented system for self-service checkout, comprising: one or more computers; and one or more computer memory devices interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers, perform one or more operations, comprising: obtaining, by using a camera, an image, wherein the image is of at least one product placed on a checkout counter; performing image segmentation on the image to obtain at least one image region; identifying a product code included in a code region in an image region of the at least one image region; determining that a product category of a product associated with the product code fails to be identified; in response to determining that the product category fails to be identified, determining the product category by using an object detection model obtained by pre-training using labeled training sample images, each labeled training sample image comprising a rectangle that frames the product code to label the code region, wherein the object detection model generates a candidate image region in the image, performs target identification within the candidate image region, performs bounding box regression in the candidate image region, and corrects the code region by using a perspective transformation relative to position markers to obtain a corrected code region, and determines the product category based on the corrected code region; and determining a price of the product based on the product category. ### Claim 13 (depends on 12) The computer-implemented system of claim 12, wherein obtaining the image comprises: controlling the camera to face the at least one product to obtain the image. ### Claim 14 (depends on 12) The computer-implemented system of claim 12, wherein obtaining the image comprises receiving the image from a self-service checkout counter. ### Claim 15 (depends on 14) The computer-implemented system of claim 14, wherein the image is taken from one of a top direction, a front direction, a left direction, a right direction, a rear direction, and an oblique direction of the at least one product, and an angle between the oblique direction and a direction vertical to a countertop of the self-service checkout counter is between 30 degrees to 60 degrees. ### Claim 16 (depends on 12) The computer-implemented system of claim 12, wherein the image segmentation is performed by using an image segmentation model, the image segmentation model is obtained by pre-training segmentation sample images comprising product images and annotations of silhouettes of products. ### Claim 17 (depends on 12) The computer-implemented system of claim 12, wherein the product code is a barcode, and identifying the product code comprises: detecting a barcode region in the image region by using an object detection model, the object detection model is obtained by pre-training training sample images comprising product images and annotations for identifying barcode regions in the product images. ### Claim 18 (depends on 17) The computer-implemented system of claim 17, wherein identifying the product code comprises: adjusting the barcode region based on the perspective transformation to obtain an adjusted barcode region; and identifying the barcode in the adjusted barcode region. ### Claim 19 (depends on 12) The computer-implemented system of claim 12, wherein the product code is a two-dimensional code, and identifying the product code comprises: detecting position markers of the two-dimensional code in the image region; and determining that a two-dimensional code region is detected in response to detecting at least two positioning markers. ### Claim 20 (depends on 19) The computer-implemented system of claim 19, wherein identifying the product code comprises: Adjusting the two-dimensional code region based on the perspective transformation to obtain an adjusted two-dimensional code; determining a corner-module relationship in the adjusted two-dimensional code based on the at least two positioning markers; and identifying the two-dimensional code based on the corner-module relationship. ### Claim 21 A non-transitory, computer-readable medium storing one or more instructions executable by a computer-implemented system to perform one or more operations for self-service checkout comprising: obtaining, by using a camera, an image, wherein the image is of at least one product placed on a checkout counter; performing image segmentation on the image to obtain at least one image region; identifying a product code included in a code region in an image region of the at least one image region; determining that a product category of a product associated with the product code fails to be identified; in response to determining that the product category fails to be identified, determining the product category by using an object detection model obtained by pre-training using labeled training sample images, each labeled training sample image comprising a rectangle that frames the product code to label the code region, wherein the object detection model generates a candidate image region in the image, performs target identification within the candidate image region, performs bounding box regression in the candidate image region, and corrects the code region by using a perspective transformation relative to position markers to obtain a corrected code region, and determines the product category based on the corrected code region; and determining a price of the product based on the product category. ### Claim 22 (depends on 21) The non-transitory, computer-readable medium of claim 21, wherein obtaining the image comprises: controlling the camera to face the at least one product placed on the checkout counter to obtain the image. ### Claim 23 (depends on 21) The non-transitory, computer-readable medium of claim 21, wherein obtaining the image comprises receiving the image from a self-service checkout counter. ### Claim 24 (depends on 23) The non-transitory, computer-readable medium of claim 23, wherein the image is taken from one of a top direction, a front direction, a left direction, a right direction, a rear direction, and an oblique direction of the at least one product placed on the self-service checkout counter, and an angle between the oblique direction and a direction vertical to a countertop of the self-service checkout counter is between 30 degrees to 60 degrees. ### Claim 25 (depends on 21) The non-transitory, computer-readable medium of claim 21, wherein the image segmentation is performed by using an image segmentation model, the image segmentation model is obtained by pre-training segmentation sample images comprising product images and annotations of silhouettes of products. ### Claim 26 (depends on 21) The non-transitory, computer-readable medium of claim 21, wherein the product code is a barcode, and identifying the product code comprises: detecting a barcode region in the image region by using an object detection model, the object detection model is obtained by pre-training training sample images comprising product images and annotations for identifying barcode regions in the product images. ### Claim 27 (depends on 26) The non-transitory, computer-readable medium of claim 26, wherein identifying the product code comprises: adjusting the barcode region based on the perspective transformation to obtain an adjusted barcode region; and identifying the barcode in the adjusted barcode region. ### Claim 28 (depends on 21) The non-transitory, computer-readable medium of claim 21, wherein the product code is a two-dimensional code, and identifying the product code comprises: detecting position markers of the two-dimensional code in the image region; and determining that a two-dimensional code region is detected in response to detecting at least two positioning markers. ### Claim 29 (depends on 28) The non-transitory, computer-readable medium of claim 28, wherein identifying the product code comprises: adjusting the two-dimensional code region based on the perspective transformation to obtain an adjusted two-dimensional code; determining a corner-module relationship in the adjusted two-dimensional code based on the at least two positioning markers; and identifying the two-dimensional code based on the corner-module relationship. ### Claim 30 (depends on 21) The non-transitory, computer-readable medium of claim 21, wherein identifying the product category of the product associated with the product further comprises: determining the product category based on an object detection model, wherein the object detection model is obtained by pre-training training sample images comprising product images and annotations for identifying products and product categories.