Systems and methods for safe policy improvement for task oriented dialogues
Assigneesalesforce.com, inc.
InventorsGovardana Sachithanandam Ramachandran, Kazuma Hashimoto, Caiming Xiong, Richard Socher
Life of this patent
Estimating the term…
Abstract
Embodiments described herein provide safe policy improvement (SPI) in a batch reinforcement learning framework for a task-oriented dialogue. Specifically, a batch reinforcement learning framework for dialogue policy learning is provided, which improves the performance of the dialogue and learns to shape a reward that reasons the invention behind human response rather than just imitating the human demonstration.
Claims20
A method for causal-aware safety policy improvement in task-oriented learning, comprising:
- receiving a training dataset comprising a plurality of dialogues, wherein the plurality of dialogues includes a first dialogue rollout generated according to a latent stochastic behavior policy;
- repeatedly sampling the training dataset for a number of times to generate a number of training subsets and a number of validation subsets;
- training a task-oriented dialogue model based on a cross-entropy loss using training data in a first training subset of the number of training subsets;
- generating, by the task-oriented dialogue model, predicted dialogue rollouts based on dialogue data in a first validation subset of the number of validation subsets;
- adding the predicted dialogue rollouts to a pairwise causal learning subset;
- sampling a pair of dialogue rollouts from the pairwise causal learning subset;
- and
- training the task-oriented dialogue model based on a binary cross-entropy loss between a preferred probability between the pair of dialogue rollouts and a normalized metric score based on the pair of dialogue rollouts.
A system for causal-aware safety policy improvement in task-oriented learning, the system comprising:
- a communication interface receiving a training dataset comprising a plurality of dialogues, wherein the plurality of dialogues includes a first dialogue rollout generated according to a latent stochastic behavior policy;
- a memory storing a plurality of processor-executable instructions;
- and
- a processor reading the plurality of processor-executable instructions from the memory to perform operations comprising:
- repeatedly sampling the training dataset for a number of times to generate a number of training subsets and a number of validation subsets;
- training a task-oriented dialogue model based on an entropy loss using training data in a first training subset of the number of training subsets;
- generating, by the task-oriented dialogue model, predicted dialogue rollouts from dialogue data in a first validation subset of the number of validation subsets;
- adding the predicted dialogue rollouts to a pairwise causal learning subset;
- sampling a pair of dialogue rollouts from the pairwise causal learning subset;
- and
- training the task-oriented dialogue model based on a binary cross-entropy loss between a preferred probability between the pair of dialogue rollouts and a normalized metric score based on the pair of dialogue rollouts.
A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for causal-aware safety policy improvement in task-oriented learning, the instructions being executed by a processor to perform operations comprising:
- receiving a training dataset comprising a plurality of dialogues, wherein the plurality of dialogues includes a first dialogue rollout generated according to a latent stochastic behavior policy;
- repeatedly sampling the training dataset for a number of times to generate a number of training subsets and a number of validation subsets;
- training a task-oriented dialogue model based on an entropy loss using training data in a first training subset of the number of training subsets;
- generating, by the task-oriented dialogue model, predicted dialogue rollouts from dialogue data in a first validation subset of the number of validation subsets;
- adding the predicted dialogue rollouts to a pairwise causal learning subset;
- sampling a pair of dialogue rollouts from the pairwise causal learning subset;
- and
- training the task-oriented dialogue model based on a binary cross-entropy loss between a preferred probability between the pair of dialogue rollouts and a normalized metric score based on the pair of dialogue rollouts.
Dependencies are inferred from the claim text.