# US 2024/0005146 A1 · Extraction of high-value sequential patterns using reinforcement learning techniques

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005146 A1
- Document ID: US20240005146A1
- Status: Published application
- Kind: A1
- Application number: 17/855,085
- Filed: 2022-06-30
- Published: 2024-01-04
- Earliest claimed: 2022-06-30
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.; Delhi Technological University
- Inventors: Tanay Anand; Piyush Gupta; Pinkesh Badjatiya; Nikaash Puri; Jayakumar Subramanian; Balaji Krishnamurthy; Chirag Singla; Rachit Bansal; Anil Singh Parihar
- Web page: https://patentgrep.com/patent/US20240005146A1

## CPC

- G06N 3/08: Learning methods
- G06N 3/092: Reinforcement learning
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/048: Activation functions
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

In some embodiments, techniques for extracting high-value sequential patterns are provided. For example, a process may involve training a machine learning model to learn a state-action map that contains high-utility sequential patterns; extracting at least one high-utility sequential pattern from the trained machine learning model; and causing a user interface of a computing environment to be modified based on information from the at least one high-utility sequential pattern.

## Claims

### Claim 1

A method comprising:
training, by a reinforcement learning server, a machine learning model to learn a state-action map that contains high-value sequential patterns, comprising:
generating, by an agent module and based on a plurality of next-actions predicted by the machine learning model, a plurality of generated sequential patterns, wherein each of the plurality of generated sequential patterns describes a corresponding sequence of system interaction events; and
for each generated sequential pattern of the plurality of generated sequential patterns:
obtaining, by an environment module and from among a plurality of recorded sequences of system interaction events, at least one recorded sequence that matches the generated sequential pattern;
determining, by the environment module and based on the at least one recorded sequence, a reward for the generated sequential pattern; and
updating the machine learning model, by the agent module and based on the reward;
extracting, by a pattern-extracting server, at least one high-value sequential pattern from the trained machine learning model; and
causing a user interface of a computing environment to be modified based on information from the at least one high-value sequential pattern.

### Claim 2 (depends on 1)

The method of claim 1, wherein the system interaction events among the plurality of recorded sequences of system interaction events include click events or entity-initiated communications.

### Claim 3 (depends on 1)

The method of claim 1, wherein, for each generated sequential pattern of the plurality of generated sequential patterns, determining the reward is based on a utility measure that is not anti-monotonic.

### Claim 4 (depends on 1)

The method of claim 1, wherein, for each generated sequential pattern of the plurality of generated sequential patterns, determining the reward is based on a utility measure that indicates an average value across a plurality of products.

### Claim 5 (depends on 1)

The method of claim 1, wherein, for at least one generated sequential pattern of the plurality of generated sequential patterns, generating the generated sequential pattern comprises selecting a next-action of the generated sequential pattern at random.

### Claim 6 (depends on 1)

The method of claim 1, wherein each generated sequential pattern of the plurality of generated sequential patterns ends with a stop action.

### Claim 7 (depends on 1)

The method of claim 1, wherein the machine learning model comprises a deep neural network.

### Claim 8 (depends on 1)

The method of claim 1, wherein extracting the at least one high-value sequential pattern from the trained machine learning model comprises searching the trained machine learning model using a depth-first-search algorithm.

### Claim 9 (depends on 1)

The method of claim 1, wherein, for each of the plurality of generated sequential patterns, each of the corresponding sequence of system interaction events comprises a plurality of attributes.

### Claim 10 (depends on 9)

The method of claim 9, wherein the machine learning model includes a long short-term memory network and a Q-network.

### Claim 11

A non-transitory computer-readable medium having program code that is stored thereon, the program code executable by one or more processing devices for performing operations comprising:
training a machine learning model to learn a state-action map that contains high-utility sequential patterns, comprising:
generating, based on a plurality of next-actions predicted by the machine learning model being trained, a plurality of generated sequential patterns, wherein each of the plurality of generated sequential patterns describes a corresponding sequence of system interaction events and ends with a stop action; and
for each generated sequential pattern of the plurality of generated sequential patterns:
obtaining, from among a plurality of recorded sequences of system interaction events, at least one recorded sequence that matches the generated sequential pattern;
determining, based on the at least one recorded sequence and a utility measure that is not anti-monotonic, a reward for the generated sequential pattern; and
updating the machine learning model based on the reward;
extracting at least one high-utility sequential pattern from the trained machine learning model; and
causing a user interface of a computing environment to be modified based on information from the at least one high-utility sequential pattern.

### Claim 12 (depends on 11)

The non-transitory computer-readable medium of claim 11, wherein the system interaction events among the plurality of recorded sequences of system interaction events include click events or entity-initiated communications.

### Claim 13 (depends on 11)

The non-transitory computer-readable medium of claim 11, wherein the utility measure indicates an average value across a plurality of products.

### Claim 14 (depends on 11)

The non-transitory computer-readable medium of claim 11, wherein, for at least one generated sequential pattern of the plurality of generated sequential patterns, generating the generated sequential pattern comprises selecting a next-action of the generated sequential pattern at random.

### Claim 15 (depends on 11)

The non-transitory computer-readable medium of claim 11, wherein extracting the at least one high-utility sequential pattern from the trained machine learning model comprises searching the trained machine learning model using a depth-first-search algorithm.

### Claim 16

A system comprising:
a reinforcement learning module configured to train a machine learning model to learn a state-action map that contains high-utility sequential patterns, comprising:
an agent module configured to generate, based on a plurality of next-actions predicted by the machine learning model being trained, a plurality of generated sequential patterns, wherein each of the plurality of generated sequential patterns describes a corresponding sequence of system interaction events and ends with a stop action; and
an environment module configured to, for each generated sequential pattern of the plurality of generated sequential patterns:
obtain, from among a plurality of recorded sequences of system interaction events stored in a data store, at least one recorded sequence that matches the generated sequential pattern; and
determine, based on the at least one recorded sequence and a utility measure that is not anti-monotonic, a reward for the generated sequential pattern,
wherein the agent module is further configured to, for each generated sequential pattern of the plurality of generated sequential patterns, update the machine learning model based on the corresponding reward;
a pattern-extracting module configured to extract a plurality of high-utility sequential patterns from the trained machine learning model; and
an interface-modification server configured to cause a user interface of a computing environment to be modified based on information from the plurality of high-utility sequential patterns.

### Claim 17 (depends on 16)

The system of claim 16, wherein the system interaction events among the plurality of recorded sequences of system interaction events include click events or entity-initiated communications.

### Claim 18 (depends on 16)

The system of claim 16, wherein the utility measure indicates an average value across a plurality of products.

### Claim 19 (depends on 16)

The system of claim 16, wherein the agent module is configured to generate, for at least one generated sequential pattern of the plurality of generated sequential patterns, the generated sequential pattern by selecting at least one next-action of the generated sequential pattern at random.

### Claim 20 (depends on 16)

The system of claim 16, wherein the pattern-extracting module is configured to extract the at least one high-utility sequential pattern from the trained machine learning model by using a depth-first-search algorithm to search the trained machine learning model.
