Measuring fairness in large-scale recommendation systems with missing labels
OwnerKun Jin, Yulong Dong +1inventors
InventorsKun Jin, Yulong Dong, Xinghai Hu
Life of this patent
Estimating the term…
Abstract
Example computer-implemented methods and systems for fairness metric estimation are disclosed. One example method includes, for each user of multiple users, obtaining first data associated with a first collection of items, the first collection of items being recommended to the user by a recommendation model. Second data associated with a second collection of items recommended to the user is obtained, the second collection of items being randomly selected for recommendation to the user. A fairness metric is calculated as a calculated fairness metric and based on the first data and the second data.
Claims17
A method for evaluating a fairness metric in a large-scale item recommendations system, comprising:
- for a user of a plurality of users:
- obtaining, by a computer system, first data associated with a first collection of items, the first collection of items being recommended to the user by a recommendation model from a candidate pool, wherein the candidate pool comprises items in the large-scale item recommendations system;
- obtaining, by the computer system, second data associated with a second collection of items, the second collection of items being randomly selected from the candidate pool for recommendation to the user and comprising unlabeled user-item pairs;
- determining, by the computer system as a calculated fairness metric, a fairness metric based on the first data and the second data, wherein determining the fairness metric based on the first data and the second data comprises:
- partitioning user-item pairs in the first data and the second data into a plurality of groups;
- determining a ranking-based true positive rate (RTPR) utility for each of the plurality of groups to determine a penalty corresponding to the fairness metric;
- and
- determining the fairness metric based on penalties determined from RTPR utilities that are determined for the plurality of groups;
- and
- determining, by the computer system, one or more recommendation strategies for the large-scale item recommendations system based on the calculated fairness metric.
A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
- for a user of a plurality of users:
- obtaining, by a computer system, first data associated with a first collection of items, the first collection of items being recommended to the user by a recommendation model from a candidate pool, wherein the candidate pool comprises items in a large-scale item recommendations system;
- obtaining, by the computer system, second data associated with a second collection of items, the second collection of items being randomly selected from the candidate pool for recommendation to the user and comprising unlabeled user-item pairs;
- and
- determining, by the computer system as a calculated fairness metric, a fairness metric based on the first data and the second data, wherein determining the fairness metric based on the first data and the second data comprises:
- partitioning user-item pairs in the first data and the second data into a plurality of groups;
- determining a ranking-based true positive rate (RTPR) utility for each of the plurality of groups to determine a penalty corresponding to the fairness metric;
- and
- determining the fairness metric based on penalties determined from RTPR utilities that are determined for the plurality of groups,
- determining, by the computer system, one or more recommendation strategies for the large-scale item recommendations system based on the calculated fairness metric.
A non-transitory computer-readable storage medium encoded with a computer program, the computer program comprising instructions that are operable, when executed by a data processing apparatus, to cause the data processing apparatus to perform operations comprising:
- for a user of a plurality of users:
- obtaining, by a computer system, first data associated with a first collection of items, the first collection of items being recommended to the user by a recommendation model from a candidate pool, wherein the candidate pool comprises items in a large-scale item recommendations system;
- obtaining, by the computer system, second data associated with a second collection of items, the second collection of items being randomly selected from the candidate pool for recommendation to the user and comprising unlabeled user-item pairs;
- and
- determining, by the computer system as a calculated fairness metric, a fairness metric based on the first data and the second data, wherein determining the fairness metric based on the first data and the second data comprises:
- partitioning user-item pairs in the first data and the second data into a plurality of groups;
- determining a ranking-based true positive rate (RTPR) utility for each of the plurality of groups to determine a penalty corresponding to the fairness metric;
- and
- determining the fairness metric based on penalties determined from RTPR utilities that are determined for the plurality of groups,
- determining, by the computer system, one or more recommendation strategies for the large-scale item recommendations system based on the calculated fairness metric.
Dependencies are inferred from the claim text.