# US 12,223,697 B1 · System and method for unsupervised concept extraction from reinforcement learning agents > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,697 B1 - Document ID: US12223697B1 - Status: Granted - Kind: B1 - Application number: 17/901,703 - Filed: 2022-09-01 - Granted: 2025-02-11 - Earliest claimed: 2019-09-26 - Assignees: HRL Laboratories, LLC - Applicants: HRL Laboratories, LLC - Inventors: Praveen K. Pilly; Nicholas A. Ketz - Web page: https://patentgrep.com/patent/US12223697B1 ## CPC - G06F 30/20: Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30) - G06V 10/776: Validation; Performance evaluation - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/092: Reinforcement learning - G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]} - G06V 10/761: {Proximity, similarity or dissimilarity measures} - G06V 10/762: using clustering, e.g. of similar faces in social networks - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/778: Active pattern-learning, e.g. online learning of image or video features - G06V 10/82: using neural networks - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Described is a method for improved performance of agent-based machine learning. The method includes training a reinforcement learning (RL) agent on an image processing task. A dataset of states and corresponding actions is then extracted from the RL agent. A measure of attention is applied to an input space of the RL agent. During action selection by the RL agent, image patches of the input space are extracted based on the applied measure of attention. Portions of a set of inputs are clustered based on similarity to the image patches, generating a set of clusters having cluster centers. Non-semantic concept labels are provided as distances to the cluster centers for each state in the dataset. ## Claims ### Claim 1 A system for performance of agent-based machine learning with unsupervised concept extraction from reinforcement learning agents, the system comprising: one or more processors and a non-transitory computer-readable medium having executable instructions encoded thereon such that when executed, the one or more processors perform operations of: training a reinforcement learning (RL) agent on an image processing task; extracting a dataset of a plurality of states and corresponding actions from the RL agent, the plurality of states corresponding to a plurality of image frames; perturbing each image frame by blurring each pixel in the image frame and focusing the blur around each pixel using a Gaussian filter centered on the pixel; processing each image frame with the retrained RL agent to obtain an output; comparing the output to the image frame prior to perturbation to determine a saliency score for each pixel in each image frame; extracting, during action selection by the RL agent, a plurality of image patches of the input space based on the saliency score; clustering, using an unsupervised clustering algorithm, portions of a first set of inputs based on similarity to the plurality of image patches, thereby generating a first set of clusters having cluster centers; and providing non-semantic concept labels as distances to the cluster centers for each state in the dataset. ### Claim 2 (depends on 1) The system as set forth in claim 1, wherein the one or more processors further perform an operation of using the non-semantic concept labels, retraining the RL agent on the image processing task. ### Claim 3 (depends on 1) The system as set forth in claim 1, wherein the one or more processors further perform an operation of using the non-semantic concept labels, training another system that relies on the same input space. ### Claim 4 (depends on 1) The system as set forth in claim 1, wherein the one or more processors perform operations of: determining a saliency map for the image frame based on the saliency scores of the pixels in the image frame; using the saliency map, extracting contiguous salient portions of the image frame; and filtering the image frame using the extracted contiguous salient portions to highlight portions of the image frame for the RL agent to use for action selection. ### Claim 5 A method for performance of agent-based machine learning with unsupervised concept extraction from reinforcement learning agents, the method comprising acts of: training a reinforcement learning (RL) agent on an image processing task; extracting a dataset of a plurality of states and corresponding actions from the RL agent, the plurality of states corresponding to a plurality of image frames; perturbing each image frame by blurring each pixel in the image frame and focusing the blur around each pixel using a Gaussian filter centered on the pixel; processing each image frame with the retrained RL agent to obtain an output; comparing the output to the image frame prior to perturbation to determine a saliency score for each pixel in each image frame; extracting, during action selection by the RL agent, a plurality of image patches of the input space based on the saliency score; clustering, using an unsupervised clustering algorithm, portions of a first set of inputs based on similarity to the plurality of image patches, thereby generating a first set of clusters having cluster centers; and providing non-semantic concept labels as distances to the cluster centers for each state in the dataset. ### Claim 6 (depends on 5) The method as set forth in claim 5, further comprising an act of using the non-semantic concept labels, retraining the RL agent on the image processing. ### Claim 7 (depends on 5) The method as set forth in claim 5, further comprising an act of using the non-semantic concept labels, training another system that relies on the same input space. ### Claim 8 (depends on 5) The method as set forth in claim 5, further comprising acts of: determining a saliency map for the image frame based on the saliency scores of the pixels in the image frame; using the saliency map, extracting contiguous salient portions of the image frame; and filtering the image frame using the extracted contiguous salient portions to highlight portions of the image frame for the RL agent to use for action selection. ### Claim 9 A computer program product for performance of agent-based machine learning with unsupervised concept extraction from reinforcement learning agents, the computer program product comprising: a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions by one or more processors, the one or more processors perform operations of: training a reinforcement learning (RL) agent on an image processing task; extracting a dataset of a plurality of states and corresponding actions from the RL agent, the plurality of states corresponding to a plurality of image frames; perturbing each image frame by blurring each pixel in the image frame and focusing the blur around each pixel using a Gaussian filter centered on the pixel; processing each image frame with the retrained RL agent to obtain an output; comparing the output to the image frame prior to perturbation to determine a saliency score for each pixel in each image frame; extracting, during action selection by the RL agent, a plurality of image patches of the input space based on the saliency score; clustering, using an unsupervised clustering algorithm, portions of a first set of inputs based on similarity to the plurality of image patches, thereby generating a first set of clusters having cluster centers; and providing non-semantic concept labels as distances to the cluster centers for each state in the dataset. ### Claim 10 (depends on 9) The computer program product as set forth in claim 9, further comprising instructions for causing the one or more processors to perform an operation of using the non-semantic concept labels, retraining the RL agent on the image processing task. ### Claim 11 (depends on 9) The computer program product as set forth in claim 9, further comprising instructions for causing the one or more processors to perform an operation of using the non-semantic concept labels, training another system that relies on the same input space. ### Claim 12 (depends on 9) The computer program product as set forth in claim 9, further comprising instructions for causing the one or more processors to perform operations of: determining a saliency map for the image frame based on the saliency scores of the pixels in the image frame; using the saliency map, extracting contiguous salient portions of the image frame; and filtering the image frame using the extracted contiguous salient portions to highlight portions of the image frame for the RL agent to use for action selection.