# US 12,223,458 B2 · Analyzing images to assess task performance > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,458 B2 - Document ID: US12223458B2 - Status: Granted - Kind: B2 - Application number: 18/316,606 - Filed: 2023-05-12 - Published: 2023-09-07 - Granted: 2025-02-11 - Earliest claimed: 2020-06-01 - Assignees: Trax Technology Solutions Pte Ltd. - Applicants: Trax Technology Solutions Pte Ltd. - Inventors: Youval Bronicki; Sheila Meftah - Web page: https://patentgrep.com/patent/US12223458B2 ## CPC - G06Q 10/06395: {Quality analysis or management} - G01C 21/206: {specially adapted for indoor navigation} - G01G 19/4144: {for controlling weight of goods in commercial establishments, e.g. supermarket, P.O.S. systems} - G01G 19/52: Weighing apparatus combined with other objects, e.g. furniture (with walking sticks A45B3/08) - G05D 1/0094: {involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target} - G05D 1/0282: {generated in a local control room} - G05D 1/249: from positioning sensors located off-board the vehicle, e.g. from cameras - G05D 1/648: Performing a task within a working area or space, e.g. cleaning - G06F 18/2431: Multiple classes - G06F 18/253: {of extracted features} - G06F 18/285: {Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system} - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 5/04: Inference or reasoning models - G06N 20/00: Machine learning - G06Q 10/06311: {Scheduling, planning or task assignment for a person or group} - G06Q 10/06315: {Needs-based resource requirements planning or analysis} - G06Q 10/06375: {Prediction of business process outcome or impact based on a proposed change} - G06Q 10/0639: Performance analysis of employees; Performance analysis of enterprise or organisation operations - G06Q 10/06398: {Performance of employee with respect to a job function} - G06Q 10/08724: {by shelf level inventory management, e.g. planograms} - G06Q 10/0877: {by inventory control or reporting using inventory tracking or counting} - G06Q 30/018: Certifying business or products - G06Q 30/0201: Market modelling; Market analysis; Collecting market data - G06Q 30/0208: Trade or exchange of goods or services in exchange for incentives or rewards - G06Q 30/0281: {Customer communication at a business location, e.g. providing product or service information, consulting} - G06Q 30/0601: Electronic shopping [e-shopping] - G06Q 30/0609: {Qualifying participants for shopping transactions (payment transaction verification G06Q20/401)} - G06Q 30/0623: {by investigating goods or services} - G06Q 30/0639: {Locating goods or services, e.g. based on physical position of the goods or services within a shopping facility} - G06Q 30/0643: {graphically representing goods, e.g. 3D product representation} - G06Q 90/20: {Destination assistance within a business structure or complex} - G06T 7/0002: {Inspection of images, e.g. flaw detection} - G06T 7/50: Depth or shape recovery - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/73: using feature-based methods - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)} - G06V 10/82: using neural networks - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 20/20: in augmented reality scenes - G06V 20/36: {Indoor scenes} - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - H04L 67/303: Terminal profiles - H04N 7/18: Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast - H04N 7/181: {for receiving images from a plurality of remote sources} - H04W 4/024: Guidance services - H04W 4/029: Location-based management or tracking services - H04W 4/35: for the management of goods or merchandise - A47L 11/4011: {Regulation of the cleaning machine by electric means; Control systems and remote control systems therefor} - A47L 2201/04: Automatic control of the travelling movement; Automatic obstacle detection - G01G 19/14: for weighing suspended loads (G01G3/00 takes precedence; incorporation of weighing devices in cranes B66C1/40, B66C13/16) - G01G 23/18: Indicating devices, e.g. for remote indication; Recording devices; Scales, e.g. graduated - G01J 1/4204: {with determination of ambient light (solar light G01J2001/4266)} - G01L 1/00: Measuring force or stress, in general (measuring force due to impact G01L5/00) - G06N 3/045: Combinations of networks - G06N 3/08: Learning methods - G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders - G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)} - G06T 2207/10016: Video; Image sequence - G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence - G06T 2207/10024: Color image - G06T 2207/30242: Counting objects in image - G06V 2201/09: Recognition of logos - H04N 7/188: {Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position (signal generation from motion picture films H04N5/253)} - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G01G: WEIGHING (sorting by weighing B07C5/16) - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) - A47L: DOMESTIC WASHING OR CLEANING (brushes A46B; cleaning quantities of bottles or of other hollow articles of one kind B08B9/00; laundry D06F); SUCTION CLEANERS IN GENERAL (cleaning in general B08) - G01J: MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY (light sources F21, H01J, H01K, H05B; investigating properties of materials by optical means G01N) - G01L: MEASURING FORCE, STRESS, TORQUE, WORK, MECHANICAL POWER, MECHANICAL EFFICIENCY, OR FLUID PRESSURE (weighing G01G) ## Abstract A non-transitory computer-readable medium includes instructions that when executed by a processor cause the processor to perform a method for assessing quality of tasks performed by persons in retail stores. The method may include receiving an indication that a person completed a task corresponding to at least one shelf in a retail store. The method may also include receiving at least one image of the at least one shelf, the at least one image being captured using an image sensor after the completion of the task. The method may further include analyzing the at least one image to determine at least one property associated with performing the task. The method may further also using the at least one property to determine a reward for performing the task. ## Claims ### Claim 1 A non-transitory computer-readable medium including instructions that when executed by a processor cause the processor to perform a method for assessing quality of tasks performed by persons in retail stores, the method comprising: receiving an indication that a person completed a task corresponding to at least one shelf in a retail store; receiving at least one image of the at least one shelf, the at least one image being captured using an image sensor after the completion of the task; analyzing the at least one image to determine at least one property associated with performing the task, wherein analyzing the at least one image to determine the at least one property comprises: identifying at least one visual indicator in the at least one image, the visual indicator comprising at least one of: a color, a brightness, a character, or a shape; comparing the at least one visual indicator to a reference visual indicator; and determining a degree of similarity between the identified at least one visual indicator and the reference visual indicator; and using the at least one property to determine a reward for performing the task. ### Claim 2 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the indication is based on an input from the person. ### Claim 3 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the indication is based on an analysis of the at least one image. ### Claim 4 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the at least one property includes a quality indication of at least one aspect of performing the task, and the determination of the reward is based on the quality indication. ### Claim 5 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the at least one image is at least one image captured by at least one of the person or an image sensor mounted to a shelf. ### Claim 6 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the method further comprises: analyzing the at least one image to determine a property of the person; and basing the determination of the reward on the property of the person. ### Claim 7 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the task includes at least one of a restocking of a product associated with the at least one shelf, a correction of product facings at the at least one shelf, removing a product from the at least one shelf, changing a price of at least one product, or changing an orientation of at least one product on the at least one shelf. ### Claim 8 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the method further comprises: analyzing the at least one image to determine at least one additional available task corresponding to the at least one shelf in the retail store; and providing an indication of the additional available task to the person. ### Claim 9 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the method further comprises: analyzing the at least one image to determine that the person performed a positive action corresponding to the at least one shelf in the retail store, the positive action not being included in the task; and basing the determination of the reward on the positive action. ### Claim 10 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the method further comprises: determining an impact of the performed task; and basing the determination of the reward on the impact. ### Claim 11 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the task includes at least one of positioning or removing a promotional sign at the at least one shelf. ### Claim 12 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the method further comprises: analyzing the at least one image to determine at least one aspect lacking in the performance of the task; providing an indication of the at least one aspect to the person; receiving at least one additional image of the at least one shelf, the at least one image being captured using the image sensor after the indication of the at least one aspect is provided; and analyzing the at least one additional image to determine the reward for performing the task. ### Claim 13 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the reward includes a monetary reward, and the determination of the reward includes a determination of an amount associated with the monetary reward. ### Claim 14 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the method further comprises: receiving input from at least one pressure sensor positioned on the at least one shelf; and basing the determination of the at least one property on an analysis of the received input. ### Claim 15 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the method further comprises: receiving input from at least one touch sensor positioned on the at least one shelf; and basing the determination of the at least one property on an analysis of the received input. ### Claim 16 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the method further comprises: receiving input from at least one weight sensor positioned on the at least one shelf; and basing the determination of the at least one property on an analysis of the received input. ### Claim 17 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the method further comprises: receiving input from at least one light sensor positioned on the at least one shelf; and basing the determination of the at least one property on an analysis the received input. ### Claim 18 A system for assessing quality of tasks performed by persons in retail stores, the system comprising: at least one processor; and a non-transitory computer-readable medium containing instructions that, when executed by the at least one processor, cause the at least one processor to: receive an indication that a person completed a task corresponding to at least one shelf in a retail store; receive at least one image of the at least one shelf captured using an image sensor after the completion of the task; analyze the at least one image to determine at least one property associated with performing the task, wherein analyzing the at least one image to determine the at least one property comprises: identifying at least one visual indicator in the at least one image, the visual indicator comprising at least one of: a color, a brightness, a character, or a shape; comparing the at least one visual indicator to a reference visual indicator; and determining a degree of similarity between the identified at least one visual indicator and the reference visual indicator; and use the at least one property to determine a reward for performing the task. ### Claim 19 A method for assessing quality of tasks performed by persons in retail stores, comprising: receiving an indication that a person completed a task corresponding to at least one shelf in a retail store; receiving at least one image of the at least one shelf captured using an image sensor after the completion of the task; analyzing the at least one image to determine at least one property associated with performing the task, wherein analyzing the at least one image to determine the at least one property comprises: identifying at least one visual indicator in the at least one image, the visual indicator comprising at least one of: a color, a brightness, a character, or a shape; comparing the at least one visual indicator to a reference visual indicator; and determining a degree of similarity between the identified at least one visual indicator and the reference visual indicator; and using the at least one property to determine a reward for performing the task.