# US 2021/0240967 A1 · Selection of object recognition models for computer vision > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0240967 A1 - Document ID: US20210240967A1 - Status: Published application - Kind: A1 - Application number: 16/884,233 - Filed: 2020-05-27 - Published: 2021-08-05 - Earliest claimed: 2020-01-31 - Assignees: salesforce.com, inc. - Applicants: salesforce.com, inc. - Inventors: Rajdeep DUA; Mani Kandar Kandar MADDURI; Nutana Sukumar MURAMREDDY; Piyush SINGH - Web page: https://patentgrep.com/patent/US20210240967A1 ## CPC - G06V 10/764: using classification, e.g. of video objects - G06F 18/2431: Multiple classes - G06F 18/25: Fusion techniques - G06T 7/0014: {using an image reference approach} - G06V 10/446: {using Haar-like filters, e.g. using integral image techniques} - G06V 20/64: Three-dimensional [3D] objects - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Disclosed herein are system, method, and computer program product embodiments for compliance auditing using cloud based computer vision. In one aspect, a system is configured to receive, from a mobile device, a compliance audit request to at least recognize one or more products within the audit image. The system is further configured to select a first object recognition model having a first associated object recognition model identifier from a model selection list based at least on a required object recognition list, wherein the first object recognition model is configured to recognize a first set of object names within the required object recognition list. The system is further configured to request the computer vision system to perform object recognition using the first object recognition model to recognize the first set of object names within the audit image, and transmit audit result information to the mobile device. ## Claims ### Claim 1 At least one server device, comprising: a memory configured to store operations; and one or more processors configured to perform the operations, the operations comprising: receiving, from a mobile device, a compliance audit request to at least recognize one or more products within an audit image, wherein the compliance audit request includes the audit image captured by the mobile device and is associated with an audit identifier identifying a compliance audit, selecting a first object recognition model having a first associated object recognition model identifier from a model selection list based at least on a required object recognition list, wherein the first object recognition model is configured to recognize a first set of object names within the required object recognition list, requesting a computer vision system to perform object recognition using the first object recognition model to recognize the first set of object names within the audit image, and transmitting audit result information to the mobile device, wherein the audit result information includes a set of recognized objects determined based at least on a first set of objects names that is recognized by the computer vision system within the audit image. ### Claim 2 (depends on 1) The at least one server device of claim 1, wherein the operations further comprising: determining the required object recognition list based at least on one or more reference product names of compliance audit information having the associated audit identifier, wherein the required object recognition list includes at least one object name identifying at least one product to be recognized within the audit image, and generating at least a portion of the audit result information before transmitting the audit result information to the mobile device, wherein the mobile device is configured to visually present one or more annotations identifying one or more locations of the set of recognized objects within the audit image. ### Claim 3 (depends on 2) The at least one server device of claim 2, wherein the operation for selecting the first object recognition model further comprises: filtering, from the model selection list, two or more model selection elements that do not include any recognized objects that are in the required object recognition list, and filtering, from the two or more model selection elements in the model selection list, one or more recognized objects that are not in the required object recognition list. ### Claim 4 (depends on 3) The at least one server device of claim 3, wherein the operation for selecting the first object recognition model further comprises: sorting the two or more model selection elements of the model selection list based at least on a number recognized objects in each model selection element, and removing, from the model selection list, a model selection element that includes a highest number of recognized objects among all model selection elements in the model selection list. ### Claim 5 (depends on 4) The at least one server device of claim 4, wherein the operations further comprises: filtering, from the two or more model selection elements in model selection list and from the required object recognition list, one or more recognized objects that are in the removed model selection element, and filtering, from the model selection list, at least one model selection element that does not include any recognized objects. ### Claim 6 (depends on 5) The at least one server device of claim 5, wherein the operations further comprises: determining that neither the model selection list nor the required object recognition list is empty, and selecting a second object recognition model having a second object recognition model identifier from the model selection list based at least on the required object recognition list. ### Claim 7 (depends on 6) The at least one server device of claim 6, wherein the second object recognition model is configured to recognize a second set of object names in the required object recognition list, and the second set of object names includes at least one object name that is not within the first set of object names in the required object recognition list. ### Claim 8 (depends on 3) The at least one server device of claim 3, wherein each recognized object of the set of recognized objects is represented as recognized object information, the recognized object information includes at least one recognized object tag that identifies at least one rectangular region of a recognized object within the audit image, and the recognized object tag includes at least a minimum X coordinate and a minimum Y coordinate and a maximum X coordinate and maximum Y coordinate defining at least two corners of a rectangular region of where the recognized object is located within the audit image. ### Claim 9 (depends on 8) The at least one server device of claim 8, wherein the recognized object information includes a recognized object name identifying a name of the recognized object, a recognized object unique identifier that uniquely identifies the recognized object, a recognized object placement description that identifies a placement location of the recognized object and a number of facings at that placement location, a recognized object facing count that indicates a number of facings for a recognized within the audit image, a recognized object share of shelf that identifies a percentage of the recognized object facing count as compared to all available facings within a reference image of the compliance audit information. ### Claim 10 (depends on 2) The at least one server device of claim 2, wherein the audit image is a digital image captured by an image capture component of the mobile device, and the audit image includes one or more physical products for sale as arranged at a physical location from a viewpoint captured by the image capture component. ### Claim 11 (depends on 2) The at least one server device of claim 2, wherein the operations further comprising: validating the audit image for compliance with an image format and image size associated with the computer vision system configured to recognize one or more objects within the audit image, before requesting the computer vision system to perform the object recognition using the first object recognition model. ### Claim 12 (depends on 11) The at least one server device of claim 11, wherein the operation for requesting the computer vision system to perform object recognition using the first object recognition model further comprises: transmitting, to the computer vision system, an object recognition request to recognize one or more objects within the audit image by using object recognition model identified by the object recognition model identifier, wherein the object recognition request includes the audit image information representative of the audit image and a first object recognition model identifier that identifies the first object recognition model and, and receiving, from the computer vision system, an object recognition response, wherein the object recognition response includes the first object recognition model identifier and at least a portion of the recognized object information for a recognized object within the audit image. ### Claim 13 A computer implemented method, comprising: receiving, from a mobile device, a compliance audit request to at least recognize one or more products within an audit image, wherein the compliance audit request includes the audit image captured by the mobile device and is associated with an audit identifier identifying a compliance audit, selecting a first object recognition model having a first associated object recognition model identifier from a model selection list based at least on a required object recognition list, wherein the first object recognition model is configured to recognize a first set of object names within the required object recognition list, requesting a computer vision system to perform object recognition using the first object recognition model to recognize the first set of object names within the audit image, and transmitting audit result information to the mobile device, wherein the audit result information includes a set of recognized objects determined based at least on a first set of objects names that is recognized by the computer vision system within the audit image. ### Claim 14 (depends on 13) The computer implemented method of claim 13, further comprising: determining the required object recognition list based at least on one or more reference product names of compliance audit information having the associated audit identifier, wherein the required object recognition list includes at least one object name identifying at least one product to be recognized within the audit image; and generating at least a portion of the audit result information before transmitting the audit result information to the mobile device, wherein the mobile device is configured to visually present one or more annotations identifying one or more locations of the set of recognized objects within the audit image. ### Claim 15 (depends on 14) The computer implemented method of claim 14, wherein the selecting of the first object recognition model further comprises: filtering, from the model selection list, two or more model selection elements that do not include any recognized objects that are in the required object recognition list; and filtering, from the two or more model selection elements in the model selection list, one or more recognized objects that are not in the required object recognition list. ### Claim 16 (depends on 15) The computer implemented method of claim 15, wherein the selecting of the first object recognition model further comprises: sorting the two or more model selection elements of the model selection list based at least on a number recognized objects in each model selection element; and removing, from the model selection list, a model selection element that includes a highest number of recognized objects among all model selection elements in the model selection list. ### Claim 17 (depends on 16) The computer implemented method of claim 16, further comprising: filtering, from the two or more model selection elements in model selection list and from the required object recognition list, one or more recognized objects that are in the removed model selection element; and filtering, from the model selection list, at least one model selection element that does not include any recognized objects. ### Claim 18 (depends on 17) The computer implemented method of claim 17, further comprising: determining that neither the model selection list nor the required object recognition list is empty; and selecting a second object recognition model having a second object recognition model identifier from the model selection list based at least on the required object recognition list. ### Claim 19 A computer readable storage device having instructions stored thereon, execution of which, by one or more processing devices, causes the one or more processing devices to perform operations comprising: receiving, from a mobile device, a compliance audit request to at least recognize one or more products within an audit image, wherein the compliance audit request includes the audit image captured by the mobile device and is associated with an audit identifier identifying a compliance audit; determining a required object recognition list based at least on one or more reference product names of compliance audit information having the associated audit identifier, wherein the required object recognition list includes at least one object name identifying at least one product to be recognized within the audit image; selecting a first object recognition model having a first associated object recognition model identifier from a model selection list based at least on the required object recognition list, wherein the first object recognition model is configured to recognize a first set of object names within the required object recognition list; requesting a computer vision system to perform object recognition using the first object recognition model to recognize the first set of object names within the audit image; generating at least a portion of audit result information, wherein the audit result information includes a set of recognized objects determined based at least on a first set of objects names that is recognized by the computer vision system within the audit image; and transmitting the audit result information to the mobile device, wherein the mobile device is configured to visually present one or more annotations identifying one or more locations of the set of recognized objects within the audit image. ### Claim 20 (depends on 19) The computer readable storage device of claim 19, wherein the operation to select a first object recognition model further comprises: filtering, from the model selection list, two or more model selection elements that do not include recognized objects that are in the required object recognition list; filtering, from the two or more model selection elements in the model selection list, one or more recognized objects that are not in the required object recognition list; sorting the two or more model selection elements of the model selection list based at least on a number recognized objects in each model selection element; and removing, from the model selection list, a model selection element that includes a highest number of recognized Objects among all model selection elements in the model selection list.