# US 10,503,961 B2 · Object recognition for bottom of basket detection using neural network > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,503,961 B2 - Document ID: US10503961B2 - Status: Granted - Kind: B2 - Application number: 15/979,157 - Filed: 2018-05-14 - Published: 2018-09-13 - Granted: 2019-12-10 - Earliest claimed: 2016-08-08 - Assignees: Indaflow LLC - Applicants: Indaflow LLC - Inventors: Jacob D. Richards - Web page: https://patentgrep.com/patent/US10503961B2 ## CPC - G06V 10/82: using neural networks - G06F 18/2414: {Smoothing the distance, e.g. radial basis function networks [RBFN]} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 5/046: Forward inferencing; Production systems - G06T 7/90: Determination of colour characteristics - G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes} - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/764: using classification, e.g. of video objects - G06V 20/64: Three-dimensional [3D] objects - G07G 1/0009: {Details of the software in the checkout register, electronic cash register [ECR] or point of sale terminal [POS]} - G07G 1/0036: {Checkout procedures} - G07G 1/01: Details for indicating - G06T 2207/20084: Artificial neural networks [ANN] - G06V 20/68: Food, e.g. fruit or vegetables - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G07G: REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS ## Abstract A checkout lane management system is described that uses object recognition to detect whether one or more items is under a shopping cart during a customer checkout process. The checkout lane management system may comprise one or more of cameras for collecting a stream of images focused on a checkout lane. The checkout lane management system may also comprise one or more of a light device, electronic approval button, and a speaker device as part of a checkout lane device for displaying a BOB indicator or sounding the BOB indicator. ## Claims ### Claim 1 A method of object recognition for a cart in a checkout lane at a retail store location, the method comprising: collecting a stream of images from a checkout lane at a sampling rate, where each image includes color data and coordinate data; applying each image from the stream of images to a convolutional neural network to determine a set of class scores for each image; selecting a class score from the set of class scores for an image where the class score is the highest value in the set of class scores, meets a threshold, or is the highest value in the set of class scores and meets the threshold; cross-referencing the selected class score to a BOB classification value from a table of BOB classification labels; setting a BOB status for the checkout lane using the BOB classification value; translating the BOB status to a BOB indicator; and communicating the BOB indicator to a checkout lane device, wherein communicating the BOB indicator to a checkout lane device comprises at least one of displaying the BOB indicator on a light device of the checkout lane device and sounding the BOB indicator on a speaker device of the checkout lane device. ### Claim 2 (depends on 1) The method of claim 1, wherein the convolutional neural network comprises a convolutional layer, a pooling layer, and a fully-connected layer. ### Claim 3 (depends on 1) The method of claim 1, wherein setting the BOB status using the BOB classification value comprises setting the BOB status to active, indicating a BOB item is located on a lower tray of the cart, when the BOB classification value represents a likelihood the cart is present in the checkout lane with at least one item on the lower tray of the cart. ### Claim 4 (depends on 1) The method of claim 1, wherein setting the BOB status using the BOB classification value comprises setting the BOB status to inactive, indicating a BOB item is not located on a lower tray of the cart, when the BOB classification value represents a likelihood the cart is not present in the checkout lane. ### Claim 5 (depends on 1) The method of claim 1, wherein setting the BOB status for the checkout lane using the BOB classification value further comprises: setting the BOB status to inactive, indicating a BOB item is not located on a lower tray of the cart, when the BOB classification value for the image represents a likelihood the cart is not present in the checkout lane; and a plurality of consecutive images collected subsequent to the image result in class scores that cross-reference to the same BOB classification value that represents a likelihood the cart is not present in the checkout lane. ### Claim 6 (depends on 1) The method of claim 1, wherein setting the BOB status for the checkout lane using the BOB classification value further comprises: setting the BOB status to active, indicating a BOB item is located on a lower tray of the cart, when the BOB classification value for the image represents a likelihood the cart is present in the checkout lane with at least one item on the lower tray of the cart; and a plurality of consecutive images collected subsequent to the image result in class scores that cross-reference to the same BOB classification value that represents a likelihood the cart is present in the checkout lane with at least one item on the lower tray of the cart. ### Claim 7 An object recognition system for a cart in a checkout lane at a retail store location, the system comprising: at least one camera that collects a stream of images from a checkout lane, where each image includes color data and coordinate data; a computing device that receives the stream of images from the camera according to a sampling rate, the computing device including an application component that applies each image from the stream of images to a convolutional neural network to determine a set of class scores for each image, an analyzing component that selects a class score from the set of class scores for an image where the class score is the highest value in the set of class scores, meets a threshold, or is the highest value in the set of class scores and meets the threshold; cross-references the selected class score to a BOB classification value from a table of BOB classification labels; and sets a BOB status for the checkout lane using the BOB classification value, a translation component that translates the BOB status to a BOB indicator, and a communication component that communicates the BOB indicator to a checkout lane device, wherein communication of the BOB indicator to a checkout lane device comprises production of the BOB indicator as at least one of light from a light device of the checkout lane device and sound from a speaker device of the checkout lane device. ### Claim 8 (depends on 7) The system of claim 7, wherein the convolutional neural network comprises a convolutional layer, a pooling layer, and a fully-connected layer. ### Claim 9 (depends on 7) The system of claim 7, wherein the analyzing component uses the BOB classification value to set the BOB status to active, indicating a BOB item is located on a lower tray of the cart, when the BOB classification value represents a likelihood the cart is present in the checkout lane with at least one item on the lower tray of the cart. ### Claim 10 (depends on 7) The system of claim 7, wherein the analyzing component uses the BOB classification value to set the BOB status to inactive, indicating a BOB item is not located on a lower tray of the cart, when the BOB classification value represents a likelihood the cart is not present in the checkout lane. ### Claim 11 (depends on 7) The system of claim 7, wherein the analyzing component sets the BOB status for the checkout lane using the BOB classification value to inactive, indicating a BOB item is not located on a lower tray of the cart, when: the BOB classification value for the image represents a likelihood the cart is no present in the checkout lane; and a plurality of consecutive images collected subsequent to the image result in class scores that cross-reference to the same BOB classification value that represents a likelihood the cart is not present in the checkout lane. ### Claim 12 (depends on 7) The system of claim 7, wherein the analyzing component sets the BOB status for the checkout lane using the BOB classification value to active, indicating a BOB item is located on a lower tray of the cart, when: the BOB classification value for the image represents a likelihood the cart is present in the checkout lane with at least one item on the lower tray of the cart; and a plurality of consecutive images collected subsequent to the image result in class scores that cross-reference to the same BOB classification value that represents a likelihood the cart is present in the checkout lane with at least one item on the lower tray of the cart. ### Claim 13 A computer program product for object recognition for a cart in a checkout lane at a retail store location, the computer program product stored on a non-transitory computer-readable medium and including instructions adapted to cause a computer to execute steps comprising: collecting a stream of images from a checkout lane at a sampling rate, where each image includes color data and coordinate data; applying each image from the stream of images to a convolutional neural network to determine a set of class scores for each image; selecting a class score from the set of class scores for an image where the class score is the highest value in the set of class scores, meets a threshold, or is the highest value in the set of class scores and meets the threshold; cross-referencing the selected class score to a BOB classification value from a table of BOB classification labels; setting a BOB status for the checkout lane using the BOB classification value; translating the BOB status to a BOB indicator; and communicating the BOB indicator to a checkout lane device, wherein communicating the BOB indicator to a checkout lane device comprises at least one of displaying the BOB indicator on a light device of the checkout lane device and sounding the BOB indicator on a speaker device of the checkout lane device. ### Claim 14 (depends on 13) The computer program product of claim 13, wherein the convolutional neural network comprises a convolutional layer, a pooling layer, and a fully-connected layer. ### Claim 15 (depends on 13) The computer program product of claim 13, wherein setting the BOB status using the BOB classification value comprises setting the BOB status to active, indicating a BOB item is located on a lower tray of the cart, when the BOB classification value represents a likelihood the cart is present in the checkout lane with at least one item on the lower tray of the cart. ### Claim 16 (depends on 13) The computer program product of claim 13, wherein setting the BOB status using the BOB classification value comprises setting the BOB status to inactive, indicating a BOB item is not located on a lower tray of the cart, when the BOB classification value represents a likelihood the cart is not present in the checkout lane. ### Claim 17 (depends on 13) The computer program product of claim 13, wherein setting the BOB status for the checkout lane using the BOB classification value further comprises: setting the BOB status to inactive, indicating a BOB item is not located on a lower tray of the cart, when the BOB classification value for the image represents a likelihood the cart is not present in the checkout lane; and a plurality of consecutive images collected subsequent to the image result in class scores that cross-reference to the same BOB classification value that represents a likelihood the cart is not present in the checkout lane. ### Claim 18 (depends on 13) The computer program product of claim 13, wherein setting the BOB status for the checkout lane using the BOB classification value further comprises setting the BOB status to active, indicating a BOB item is located on a lower tray of the cart, when the BOB classification value for the image represents a likelihood the cart is present in the checkout lane with at least one item on the lower tray of the cart; and a plurality of consecutive images collected subsequent to the image result in class scores that cross-reference to the same BOB classification value that represents a likelihood the cart is present in the checkout lane with at least one item on the lower tray of the cart.