# US 11,443,335 B2 · Model-based deep reinforcement learning for dynamic pricing in an online ride-hailing platform > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,335 B2 - Document ID: US11443335B2 - Status: Granted - Kind: B2 - Application number: 16/720,753 - Filed: 2019-12-19 - Published: 2021-06-24 - Granted: 2022-09-13 - Earliest claimed: 2019-12-19 - Assignees: Beijing Didi Infinity Technology and Development Co., Ltd. - Applicants: Beijing DiDi Infinity Technology and Development Co., Ltd. - Inventors: Xiaocheng Tang - Web page: https://patentgrep.com/patent/US11443335B2 ## CPC - G06Q 30/0206: {Price or cost determination based on market factors} - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/08: Learning methods - G06N 3/092: Reinforcement learning - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Dynamic pricing may be applied in an online ride-hailing platform. Information may be obtained. The information may include a set of pricing candidates and an initial status of a ride-hailing platform. The set of pricing candidates may be updated based on the initial status of the ride-hailing platform to minimize a cross-entropy between the set of pricing candidates and a target pricing policy that maximizes a total income of the ride-hailing platform. A price for at least one current trip request on the ride-hailing platform may be generated based on the updated set of pricing candidates. ## Claims ### Claim 1 A computer-implemented method for dynamic pricing, comprising: training a reinforcement learning (RL) model in a simulation of a ride-hailing platform to predict a future state of the simulation based on a previous state of the simulation and a previous pricing action, wherein the training comprises minimizing a divergence between the predicted future state of the simulation and an observed futured state of the simulation, and the simulation is an offline version of the ride-hailing platform; obtaining information comprising a set of pricing candidates and an initial status of the ride-hailing platform, wherein the initial status comprises a number of available drivers and a number of trip requests from passengers of the ride-hailing platform; updating the set of pricing candidates based on the initial status of the ride-hailing platform to minimize a cross-entropy between the set of pricing candidates and a target pricing policy that maximizes a total income of the ride-hailing platform, wherein the updating comprises: inputting the obtained information into the trained RL model; obtaining a cumulative reward from the trained RL model for each pricing candidate in the set of pricing candidates based on the initial status of the ride-hailing platform; obtaining a set of top pricing candidates comprising one or more pricing candidates with the highest cumulative rewards; and updating the set of pricing candidates based on the set of top pricing candidates; and generating a price for at least one current trip request on the ride-hailing platform based on the updated set of pricing candidates; collecting an observed status of the ride-hailing platform after the generated price is deployed online; retraining the RL model to reach a convergence between the prediction of the future status of the ride-hailing platform and the observed status of the ride-hailing platform. ### Claim 2 (depends on 1) The method of claim 1, wherein the initial status of the ride-hailing platform further comprises a status of at least one origin-destination (OD) pair associated with an origin grid region and a destination grid region. ### Claim 3 (depends on 2) The method of claim 2, wherein the status of the at least one OD pair comprises: a running estimate of a number of created orders, a number of accepted orders at a set of price increment, a number of finished orders at the set of price increment, and a number of canceled orders due to time running out. ### Claim 4 (depends on 1) The method of claim 1, wherein the set of pricing candidates comprise a plurality of historical pricing sequences sampled from a set of historical pricing sequences. ### Claim 5 (depends on 1) The method of claim 1, wherein the set of pricing candidates comprise a plurality of randomly generated pricing sequences. ### Claim 6 (depends on 1) The method of claim 1, wherein obtaining the cumulative reward from the trained RL model comprises: obtaining an initial state of the simulation of the ride-hailing platform based on the initial status of the ride-hailing platform; obtaining a set of future states of the simulation of the ride-hailing platform based on the set of pricing candidates, the initial status of the ride-hailing platform, and the initial state of the simulation of the ride-hailing platform; and generating the cumulative reward based on the set of future states of the simulation of the ride-hailing platform. ### Claim 7 (depends on 6) The method of claim 6, wherein generating the cumulative reward based on the set of future states of the simulation of the ride-hailing platform comprises summing a state reward for each state of the set of future states of the simulation of the ride-hailing platform. ### Claim 8 (depends on 7) The method of claim 7, wherein the trained RL model comprises: a transition neural network configured for determining a prediction of each future state of the simulation of the ride-hailing platform based on a previous state of the simulation of the ride-hailing platform and a previous pricing action; an observation neural network configured for determining a set of future observations of the simulation of the ride-hailing platform based on each future state of a simulation of the ride-hailing platform; a state neural network configured for determining each future state of the simulation of the ride-hailing platform based on each future observation of the simulation of the ride-hailing platform; and a reward neural network configured for determining each state reward based on each future state of the simulation of the ride-hailing platform. ### Claim 9 (depends on 8) The method of claim 8, wherein the training further comprises maximizing a predicted accumulative reward. ### Claim 10 (depends on 7) The method of claim 7, wherein the trained RL model comprises: a recurrent neural network configured for determining a set of recurrent states of the simulation of the ride-hailing platform based on a set of pervious recurrent states of the simulation of the ride-hailing platform, a set of previous states of the simulation of the ride-hailing platform, and a set of previous pricing actions of the simulation of the ride-hailing platform; a transition neural network configured for determining a prediction of each future state of the simulation of the ride-hailing platform based on each recurrent state of the simulation of the ride-hailing platform; an observation neural network configured for determining a set of future observations of the simulation of the ride-hailing platform based on each recurrent state of the simulation of the ride-hailing platform and each future state of the simulation of the ride-hailing platform; a state neural network configured for determining each future state of the simulation of the ride-hailing platform based on a set of previous observations the simulation of the ride-hailing platform and the set of previous pricing actions; and a reward neural network configured for determining each state reward based on each recurrent state of the simulation of the ride-hailing platform and each future state of the simulation of the ride-hailing platform. ### Claim 11 (depends on 10) The method of claim 10, wherein the training further comprises: maximizing a data log-likelihood of obtaining each future observation and each state reward. ### Claim 12 (depends on 1) The method of claim 1, wherein generating the price for the at least one current trip request comprises generating a pricing multiplier for multiplying with a base price of the at least one current trip request. ### Claim 13 A system for dynamic pricing, comprising one or more processors and one or more non-transitory computer-readable memories coupled to the one or more processors and configured with instructions executable by the one or more processors to cause the system to perform operations comprising: training a reinforcement learning (RL) model in a simulation of a ride-hailing platform to predict a future state of the simulation based on a previous state of the simulation and a previous pricing action, wherein the training comprises minimizing a divergence between the predicted future state of the simulation and an observed futured state of the simulation; obtaining information comprising a set of pricing candidates and an initial status of the ride-hailing platform, wherein the initial status comprises a number of available drivers and a number of trip requests from passengers of the ride-hailing platform; updating the set of pricing candidates based on the initial status of the ride-hailing platform to minimize a cross-entropy between the set of pricing candidates and a target pricing policy that maximizes a total income of the ride-hailing platform, wherein the updating comprises: inputting the obtained information into the trained RL model; obtaining a cumulative reward from the trained RL model for each pricing candidate in the set of pricing candidates based on the initial status of the ride-hailing platform; obtaining a set of top pricing candidates comprising one or more pricing candidates with the highest cumulative rewards; and updating the set of pricing candidates based on the set of top pricing candidates; and generating a price for at least one current trip request on the ride-hailing platform based on the updated set of pricing candidates; collecting an observed status of the ride-hailing platform after the generated price is deployed; retraining the RL model to reach a convergence between the prediction of the future status of the ride-hailing platform and the observed status of the ride-hailing platform. ### Claim 14 (depends on 13) The system of claim 13, wherein obtaining the cumulative reward from the trained RL model comprises: obtaining an initial state of the simulation of the ride-hailing platform based on the initial status of the ride-hailing platform; obtaining a set of future states of the simulation of the ride-hailing platform based on the set of pricing candidates, the initial status of the ride-hailing platform, and the initial state of the simulation of the ride-hailing platform; and generating the cumulative reward based on the set of future states of the simulation of the ride-hailing platform. ### Claim 15 (depends on 14) The system of claim 14, wherein generating the cumulative reward based on the set of future states of the simulation of the ride-hailing platform comprises summing a state reward for each state of the set of future states of the simulation of the ride-hailing platform. ### Claim 16 A non-transitory computer-readable storage medium configured with instructions executable by one or more processors to cause the one or more processors to perform operations comprising: training a reinforcement learning (RL) model in a simulation of a ride-hailing platform to predict a future state of the simulation based on a previous state of the simulation and a previous pricing action, wherein the training comprises minimizing a divergence between the predicted future state of the simulation and an observed futured state of the simulation; obtaining information comprising a set of pricing candidates and an initial status of the ride-hailing platform, wherein the initial status comprises a number of available drivers and a number of trip requests from passengers of the ride-hailing platform; updating the set of pricing candidates based on the initial status of the ride-hailing platform to minimize a cross-entropy between the set of pricing candidates and a target pricing policy that maximizes a total income of the ride-hailing platform, wherein the updating comprises: inputting the obtained information into the trained RL model; obtaining a cumulative reward from the trained RL model for each pricing candidate in the set of pricing candidates based on the initial status of the ride-hailing platform; obtaining a set of top pricing candidates comprising one or more pricing candidates with the highest cumulative rewards; and updating the set of pricing candidates based on the set of top pricing candidates; and generating a price for at least one current trip request on the ride-hailing platform based on the updated set of pricing candidates; collecting an observed status of the ride-hailing platform after the generated price is deployed; retraining the RL model to reach a convergence between the prediction of the future status of the ride-hailing platform and the observed status of the ride-hailing platform.