# US 2020/0394534 A1 · Multi task oriented recommendation system for benchmark improvement

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0394534 A1
- Document ID: US20200394534A1
- Status: Published application
- Kind: A1
- Application number: 16/442,004
- Filed: 2019-06-14
- Published: 2020-12-17
- Earliest claimed: 2019-06-14
- Assignees: Sap SE
- Applicants: Sap SE
- Inventors: Kavitha Krishnan; Naga Sai Narasimha Guru Charan Koduri
- Web page: https://patentgrep.com/patent/US20200394534A1

## CPC

- G06N 5/04: Inference or reasoning models
- G06Q 10/06393: {Score-carding, benchmarking or key performance indicator [KPI] analysis}
- G06F 11/3428: {Benchmarking}
- G06F 11/3495: {for systems}
- G06F 17/15: Correlation function computation {including computation of convolution operations (arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00)}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06Q 30/0201: Market modelling; Market analysis; Collecting market data
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods and systems are used for improving benchmark key performance indicators (KPIs). As an example, a set of KPIs associated with a particular client is identified, each KPI of the identified set of KPIs associated with a plurality of KPI attributes. A set of particular KPI attributes associated with the identified set of KPIs associated with the particular client is identified. A recommendation assessment of the identified set of particular KPI attributes is performed using a trained recommendation reference model (RRM) to identify at least one operational recommendation for the particular client. A ranked set of the at least one operational recommendation is generated based on a ranking associated with each KPI of the at least one KPI of the identified set of KPIs. The ranked set of the at least one operational recommendation is provided to the particular client.

## Claims

### Claim 1

A computer-implemented method, comprising:
identifying, by a recommendation system, a set of key performance indicators (KPIs) associated with a particular client of a set of clients as an identified set of KPIs, each KPI of the identified set of KPIs associated with a plurality of KPI attributes used to determine a KPI score associated with each KPI;
identifying, by the recommendation system, a set of particular KPI attributes associated with the identified set of KPIs associated with the particular client as an identified set of particular KPI attributes;
performing, by the recommendation system, a recommendation assessment of the identified set of particular KPI attributes using a trained recommendation reference model (RRM) to identify at least one operational recommendation for the particular client as the identified at least one operational recommendation, the identified at least one operational recommendation associated with at least one KPI of the identified set of KPIs;
generating, by the recommendation system, a ranked set of the identified at least one operational recommendation based on a ranking associated with each KPI of the at least one KPI of the identified set of KPIs; and
providing, by the recommendation system, the ranked set of the identified at least one operational recommendation to the particular client.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the at least one KPI of the identified set of KPIs associated with the identified at least one operational recommendation performed poorly compared against at least one of a best in class KPI calculated across the set of clients, an average KPI calculated across the set of clients, an industry best in class KPI calculated across the set of clients in the same industry as the particular client, an industry average KPI calculated across the set of clients in the same industry as the particular client, a region best in class KPI calculated across the set of clients in the same region as the particular client, or a region average KPI calculated across the set of clients in the same region as the particular client.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
training a RRM to generate the trained RRM using a machine learning algorithm, a set of reference KPI attributes, a set of reference KPIs associated with at least one client, and a set of reference operational recommendations, wherein the set of reference KPIs includes at least the identified set of KPIs, and wherein the set of reference operational recommendations includes at least the at least one operational recommendation.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein training the RRM further comprises using a loss function.

### Claim 5 (depends on 3)

The computer-implemented method of claim 3, wherein the loss function may comprise at least one of a binary logistic loss function that predicts the performance of each reference KPI of the set of KPIs or a multi-level logistic loss function that determines at least one reference operational recommendation of the set of reference operational recommendations to recommend for each reference KPI of the set of reference KPIs.

### Claim 6 (depends on 3)

The computer-implemented method of claim 3, wherein the machine learning algorithm comprises at least one of a supervised learning algorithm using a deep neural network (DNN).

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein the DNN comprises at least one of a convolution neural network or a long short-term memory (LSTM).

### Claim 8

A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform operations comprising:
identifying, by a recommendation system, a set of key performance indicators (KPIs) associated with a particular client of a set of clients as an identified set of KPIs, each KPI of the identified set of KPIs associated with a plurality of KPI attributes used to determine a KPI score associated with each KPI;
identifying, by the recommendation system, a set of particular KPI attributes associated with the identified set of KPIs associated with the particular client as an identified set of particular KPI attributes;
performing, by the recommendation system, a recommendation assessment of the identified set of particular KPI attributes using a trained recommendation reference model (RRM) to identify at least one operational recommendation for the particular client as the identified at least one operational recommendation, the identified at least one operational recommendation associated with at least one KPI of the identified set of KPIs;
generating, by the recommendation system, a ranked set of the identified at least one operational recommendation based on a ranking associated with each KPI of the at least one KPI of the identified set of KPIs; and
providing, by the recommendation system, the ranked set of the identified at least one operational recommendation to the particular client.

### Claim 9 (depends on 8)

The non-transitory, computer-readable medium of claim 8, wherein the at least one KPI of the identified set of KPIs associated with the identified at least one operational recommendation performed poorly compared against at least one of a best in class KPI calculated across the set of clients, an average KPI calculated across the set of clients, an industry best in class KPI calculated across the set of clients in the same industry as the particular client, an industry average KPI calculated across the set of clients in the same industry as the particular client, a region best in class KPI calculated across the set of clients in the same region as the particular client, or a region average KPI calculated across the set of clients in the same region as the particular client.

### Claim 10 (depends on 8)

The non-transitory, computer-readable medium of claim 8, further comprising:
training a RRM to generate the trained RRM using a machine learning algorithm, a set of reference KPI attributes, a set of reference KPIs associated with at least one client, and a set of reference operational recommendations, wherein the set of reference KPIs includes at least the identified set of KPIs, and wherein the set of reference operational recommendations includes at least the at least one operational recommendation.

### Claim 11 (depends on 10)

The non-transitory, computer-readable medium of claim 10, wherein training the RRM further comprises using a loss function.

### Claim 12 (depends on 10)

The non-transitory, computer-readable medium of claim 10, wherein the loss function may comprise at least one of a binary logistic loss function that predicts the performance of each reference KPI of the set of KPIs or a multi-level logistic loss function that determines at least one reference operational recommendation of the set of reference operational recommendations to recommend for each reference KPI of the set of reference KPIs.

### Claim 13 (depends on 10)

The non-transitory, computer-readable medium of claim 10, wherein the machine learning algorithm comprises at least one of a supervised learning algorithm using a deep neural network (DNN).

### Claim 14 (depends on 13)

The non-transitory, computer-readable medium of claim 13, wherein the DNN comprises at least one of a convolution neural network or a long short-term memory (LSTM).

### Claim 15

A computer-implemented system, comprising:
one or more computers; and
one or more computer memory devices interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers, perform one or more operations comprising:
identifying, by a recommendation system, a set of key performance indicators (KPIs) associated with a particular client of a set of clients as an identified set of KPIs, each KPI of the identified set of KPIs associated with a plurality of KPI attributes used to determine a KPI score associated with each KPI;
identifying, by the recommendation system, a set of particular KPI attributes associated with the identified set of KPIs associated with the particular client as an identified set of particular KPI attributes;
performing, by the recommendation system, a recommendation assessment of the identified set of particular KPI attributes using a trained recommendation reference model (RRM) to identify at least one operational recommendation for the particular client as the identified at least one operational recommendation, the identified at least one operational recommendation associated with at least one KPI of the identified set of KPIs;
generating, by the recommendation system, a ranked set of the identified at least one operational recommendation based on a ranking associated with each KPI of the at least one KPI of the identified set of KPIs; and
providing, by the recommendation system, the ranked set of the identified at least one operational recommendation to the particular client.

### Claim 16 (depends on 15)

The computer-implemented system of claim 15, wherein the at least one KPI of the identified set of KPIs associated with the identified at least one operational recommendation performed poorly compared against at least one of a best in class KPI calculated across the set of clients, an average KPI calculated across the set of clients, an industry best in class KPI calculated across the set of clients in the same industry as the particular client, an industry average KPI calculated across the set of clients in the same industry as the particular client, a region best in class KPI calculated across the set of clients in the same region as the particular client, or a region average KPI calculated across the set of clients in the same region as the particular client.

### Claim 17 (depends on 15)

The computer-implemented system of claim 15, further comprising:
training a RRM to generate the trained RRM using a machine learning algorithm, a set of reference KPI attributes, a set of reference KPIs associated with at least one client, and a set of reference operational recommendations, wherein the set of reference KPIs includes at least the identified set of KPIs, and wherein the set of reference operational recommendations includes at least the at least one operational recommendation.

### Claim 18 (depends on 17)

The computer-implemented system of claim 17, wherein training the RRM further comprises using a loss function.

### Claim 19 (depends on 17)

The computer-implemented system of claim 17, wherein the loss function may comprise at least one of a binary logistic loss function that predicts the performance of each reference KPI of the set of KPIs or a multi-level logistic loss function that determines at least one reference operational recommendation of the set of reference operational recommendations to recommend for each reference KPI of the set of reference KPIs.

### Claim 20 (depends on 17)

The computer-implemented system of claim 17, wherein the machine learning algorithm comprises at least one of a supervised learning algorithm using a deep neural network (DNN).
