# US 10,482,345 B2 · Systems and methods for automated object recognition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,345 B2 - Document ID: US10482345B2 - Status: Granted - Kind: B2 - Application number: 15/630,913 - Filed: 2017-06-22 - Published: 2017-12-28 - Granted: 2019-11-19 - Earliest claimed: 2016-06-23 - Assignees: Capital One Services, LLC - Applicants: Capital One Services, LLC - Inventors: Hannes Mikael Jouhikainen; Drew Jacobs - Web page: https://patentgrep.com/patent/US10482345B2 ## CPC - G06N 20/00: Machine learning - G06F 16/7837: {using objects detected or recognised in the video content} - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06Q 30/0241: Advertisements - G06Q 30/0641: {utilising user interfaces specially adapted for shopping} - G06T 7/11: Region-based segmentation - G06V 10/443: {by matching or filtering} - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/70: using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88) - G06V 10/82: using neural networks - G06V 20/20: in augmented reality scenes - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/47815: {Electronic shopping (payment schemes, payment architectures or payment protocols for electronic shopping systems G06Q20/12)} - H04N 21/812: {involving advertisement data (advertising per se G06Q30/02)} - G06T 2207/10016: Video; Image sequence - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method for recognizing an object in a video stream may include receiving a video stream from a video source, the video stream comprising a plurality of video frames. The method may also include selecting at least one video frame from the video frames according to a frame selection rate. The method may also include partitioning the selected video frame into a first plurality of image blocks. The method may also include recognizing, out of the first plurality of image blocks, a second plurality of image blocks which comprise an image of an object, the recognition being based on an image recognition parameter determined by a machine-learning algorithm. The method may also include determining that at least one of the second plurality of image blocks corresponds to the object based on a likelihood metric, the likelihood metric being determined by the processor based on at least the frame selection rate. The method may further include displaying, on a display, information identifying the object. A system and non-transitory computer-readable medium may also be provided. ## Claims ### Claim 1 A system for recognizing an object in a video stream, comprising: a memory storing instructions; and a processor configured to execute the stored instructions to: receive the video stream from a video source, the video stream comprising a first set of video frames; select at least one video frame from the first set of the video frames according to a frame selection rate, wherein the frame selection rate determines a number of the selected video frames; partition the one or more selected video frames into one or more sets of image blocks, each set of image blocks corresponding to a respective video frame; identify, within one or more sets of image blocks, a region which comprise an image of an object, the identification being based on a machine-learning algorithm for determining regions characterized by an image recognition parameter; calculate a likelihood metric that the region corresponds to the object; adjust the frame selection rate when the likelihood metric is less than a predetermined threshold; and display, on a display, information identifying the object. ### Claim 2 (depends on 1) The system of claim 1, wherein the processor is further configured to execute the stored instructions to apply the machine-learning algorithm to a second set of video frames. ### Claim 3 (depends on 1) The system of claim 1, wherein the processor is further configured to execute the stored instructions to adjust the image recognition parameter in response to an input received from a user device. ### Claim 4 (depends on 1) The system of claim 1, wherein determining that the region corresponds to the object comprises: comparing the likelihood metric to a predetermined threshold; and when it is determined that the likelihood metric exceeds or equals a predetermined threshold, determining that the region corresponds to the object. ### Claim 5 (depends on 1) The system of claim 1, wherein the processor is further configured to execute the stored instructions to adjust the frame selection rate in response to an input from a user device. ### Claim 6 (depends on 1) The system of claim 1, wherein the processor is further configured to execute the stored instructions to determine the frame selection rate based on at least one of an image quality of the video stream, a location of the object in the one or more selected video frames, or a viewable angle of the object in the one or more selected video frames. ### Claim 7 (depends on 1) The system of claim 1, wherein the processor is further configured to execute the stored instructions to determine the likelihood metric based additionally on the information identifying the object. ### Claim 8 (depends on 7) The system of claim 7, wherein the information identifying the object comprises information specifying at least one of a price of the object, availability of the object, or a location of the object. ### Claim 9 (depends on 1) The system of claim 1, wherein the processor is further configured to execute the stored instructions to receive the information identifying the object from at least one data server. ### Claim 10 (depends on 1) The system of claim 1, wherein: the processor is further configured to execute the stored instructions to display the video stream; and displaying the information identifying the object comprises displaying a digital shopping cart over the displayed video stream. ### Claim 11 A computer-implemented method for recognizing an object in a video stream, comprising: receiving the video stream from a video source, the video stream comprising a first set of video frames; selecting at least one video frame from the first set of the video frames according to a frame selection rate, wherein the frame selection rate determines a number of the selected video frames; partitioning the one or more selected video frames into one or more sets of image blocks, each set of image blocks corresponding to a respective video frame; identifying within one or more sets, of image blocks, a region which comprise an image of an object, the identification being based on a machine-learning algorithm for determining regions characterized by an image recognition parameter; calculating a likelihood metric that the region corresponds to the object; adjusting the frame selection rate when the likelihood metric is less than a predetermined threshold; and displaying, on a display, information identifying the object. ### Claim 12 (depends on 11) The method of claim 11, further comprising: applying the machine-learning algorithm to a second set of video frames. ### Claim 13 (depends on 11) The method of claim 11, wherein determining that the region corresponds to the object comprises: comparing the likelihood metric to a predetermined threshold; and when it is determined that the likelihood metric exceeds or equals a predetermined threshold, determining that the region corresponds to the object. ### Claim 14 (depends on 11) The method of claim 11, further comprising: determining the frame selection rate based on at least one of an image quality of the video stream, a location of the object in the one or more selected video frames, or a viewable angle of the object in the one or more selected video frames. ### Claim 15 (depends on 11) The method of claim 11, further comprising: determining the likelihood metric based additionally on the information identifying the object. ### Claim 16 A non-transitory computer-readable medium storing instructions which, when executed, cause at least one processor to perform operations for recognizing an object in a video stream, the operations comprising: receiving the video stream from a video source, the video stream comprising a first set of video frames; selecting at least one video frame from the first set of the video frames according to a frame selection rate, wherein the frame selection rate determines a number of the selected video frames; partitioning the one or more selected video frames into one or more sets of image blocks, each set of image blocks corresponding to a respective video frame; identifying within one or more sets, of image blocks, a region which comprise an image of an object, the identification being based on a machine-learning algorithm for determining regions characterized by an image recognition parameter; calculating a likelihood metric that the region corresponds to the object; adjusting the frame selection rate when the likelihood metric is less than a predetermined threshold; and displaying, on a display, information identifying the object. ### Claim 17 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein the operations further comprise: applying the machine-learning algorithm to a second set of video frames. ### Claim 18 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein determining that the region corresponds to the object comprises: comparing the likelihood metric to a predetermined threshold; and when it is determined that the likelihood metric exceeds or equals a predetermined threshold, determining that the region corresponds to the object. ### Claim 19 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein the operations further comprise: determining the frame selection rate based on at least one of an image quality of the video stream, a location of the object in the one or more selected video frames, or a viewable angle of the object in the one or more selected video frames. ### Claim 20 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein the operations further comprise: determining the likelihood metric based additionally on the information identifying the object.