Intelligent recommendation method and apparatus, model training method and apparatus, electronic device, and storage medium

A model training method, comprising:

• determining a value target function of a multi-agent strategy model and a strategy target function of the multi-agent strategy model;
• and
• training, according to the value target function and the strategy target function, the multi-agent strategy model;
• wherein the multi-agent strategy model is configured to determine, in conjunction with an object recommendation request, object execution actions of at least two agent objects matching the object recommendation request to intelligently recommend a target agent object according to the object execution actions, the object recommendation request is a charging station recommendation request, and the at least two agent objects are charging stations;
• wherein determining the strategy target function of the multi-agent strategy model comprises:
• determining the strategy target function of the multi-agent strategy model based on the following equations:
• L
• A
• (
• θ
• a
• )
• =
• -
• 𝔼
• t
• ′
• [
• min
• ⁡
• (
• ρ
• t
• ′
• i
• ,
• A
• ^
• t
• ′
• i
• ,
• clip
• (
• ρ
• t
• ′
• i
• ,
• 1
• -
• ϵ
• ,
• 1
• +
• ϵ
• )
• ⁢
• A
• ^
• t
• ′
• i
• )
• ]
• ρ
• t
• ′
• i
• =
• π
• θ
• a
• (
• a
• t
• ′
• i
• |
• o
• t
• ′
• i
• )
• π
• θ
• t
• ′
• a
• (
• a
• t
• ′
• i
• |
• o
• t
• ′
• i
• )
• A
• ^
• t
• ′
• i
• =
• R
• t
• ′
• :
• t
• +
• γ
• (
• T
• t
• +
• T
• t
• ′
• )
• ⁢
• V
• θ
• t
• ′
• c
• (
• o
• t
• i
• )
• -
• V
• θ
• t
• ′
• c
• (
• o
• t
• ′
• i
• )
• R
• t
• :
• t
• +
• j
• =
• ∑
• T
• t
• <
• T
• t
• ′
• c
• ≤
• T
• t
• +
• j
• γ
• (
• T
• t
• ′
• c
• -
• T
• t
• -
• 1
• )
• ⁢
• r
• ⁡
• (
• S
• t
• ′
• ,
• u
• t
• ′
• )
• wherein LA(θa) denotes the strategy target function, θ denotes a network parameter of a strategy target neural network, t and t′ denote numbers of object recommendation requests, t′ denotes an expectation, ρti, denotes a probability ratio, Âti, denotes an advantage function, clip ( ) denotes a clipping function, ϵ denotes a numerical value in an interval [0, 1], i and j denote natural numbers, Rt′:t and Rt:t+j denote accumulated discounted rewards, respectively, γ denotes a discount factor, Tt and Tt′ denote initiation times of the object recommendation requests, Tt′c denotes a completion time of a t′-th object recommendation request, r(st′, ut′) denotes a reward corresponding to the t′-th object recommendation request, st′ denotes a state of each of all the agent objects in a t′-th step, and ut′ denotes a joint action of all agent objects in the t′-th step;
• wherein determining the value target function of the multi-agent strategy model comprises:
• determining the value target function of the multi-agent strategy model based on the following equation:
• L
• C
• (
• θ
• c
• )
• =
• 𝔼
• t
• ′
• [
• (
• R
• t
• ′
• :
• t
• +
• γ
• (
• T
• t
• -
• T
• t
• ′
• )
• ⁢
• V
• θ
• t
• ′
• c
• (
• o
• t
• i
• )
• -
• V
• θ
• c
• (
• o
• t
• ′
• i
• )
• )
• 2
• wherein LC(θc) denotes a mean square error (MSE) loss function of a value target neural network, θc is a model parameter of the value target neural network, and Vθt′c and Vθc denote the value target function.

An electronic device, comprising:

• at least one processor;
• and
• a memory which is in a communication connection with the at least one processor;
• wherein
• the memory is configured to store instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform:
• determining an object recommendation request, wherein the object recommendation request is a charging station recommendation request;
• determining, according to a multi-agent strategy model and the object recommendation request, object execution actions of at least two agent objects matching the object recommendation request, wherein the at least two agent objects are charging stations;
• determining a target object execution action according to the object execution actions;
• and
• recommending the object recommendation request to a target agent object corresponding to the target object execution action;
• wherein the multi-agent strategy model is trained according to a value target function of the multi-agent strategy model and a strategy target function of the multi-agent strategy model;
• wherein the strategy target function of the multi-agent strategy model is determined based on the following equations:
• L
• A
• (
• θ
• a
• )
• =
• -
• 𝔼
• t
• ′
• [
• min
• ⁡
• (
• ρ
• t
• ′
• i
• ,
• A
• ^
• t
• ′
• i
• ,
• clip
• (
• ρ
• t
• ′
• i
• ,
• 1
• -
• ϵ
• ,
• 1
• +
• ϵ
• )
• ⁢
• A
• ^
• t
• ′
• i
• )
• ]
• ρ
• t
• ′
• i
• =
• π
• θ
• a
• (
• a
• t
• ′
• i
• |
• o
• t
• ′
• i
• )
• π
• θ
• t
• ′
• a
• (
• a
• t
• ′
• i
• |
• o
• t
• ′
• i
• )
• A
• ^
• t
• ′
• i
• =
• R
• t
• ′
• :
• t
• +
• γ
• (
• T
• t
• +
• T
• t
• ′
• )
• ⁢
• V
• θ
• t
• ′
• c
• (
• o
• t
• i
• )
• -
• V
• θ
• t
• ′
• c
• (
• o
• t
• ′
• i
• )
• R
• t
• :
• t
• +
• j
• =
• ∑
• T
• t
• <
• T
• t
• ′
• c
• ≤
• T
• t
• +
• j
• γ
• (
• T
• t
• ′
• c
• -
• T
• t
• -
• 1
• )
• ⁢
• r
• ⁡
• (
• s
• t
• ′
• ,
• u
• t
• ′
• )
• wherein LA(θa) denotes the strategy target function, θa denotes a network parameter of a strategy target neural network, t and t′ denote numbers of object recommendation requests, t′ denotes an expectation, ρti, denotes a probability ratio, Âti, denotes an advantage function, clip ( ) denotes a clipping function, ϵ denotes a numerical value in an interval [0, 1], i and j denote natural numbers, Rt′:t and Rt:t+j denote accumulated discounted rewards, respectively, γ denotes a discount factor, Tt and Tt′ denote initiation times of the object recommendation requests, Tt′c denotes a completion time of a t′-th object recommendation request, r(st′, ut′) denotes a reward corresponding to the t′-th object recommendation request, st′ denotes a state of each of all the agent objects in a t′-th step, and ut′ denotes a joint action of all agent objects in the t′-th step;
• wherein the value target function of the multi-agent strategy model is determined based on the following equation:
• L
• C
• (
• θ
• c
• )
• =
• 𝔼
• t
• ′
• [
• (
• R
• t
• ′
• :
• t
• +
• γ
• (
• T
• t
• -
• T
• t
• ′
• )
• ⁢
• V
• θ
• t
• ′
• c
• (
• o
• t
• i
• )
• -
• V
• θ
• c
• (
• o
• t
• ′
• i
• )
• )
• 2
• wherein LC(θc) denotes a mean square error (MSE) loss function of a value target neural network, θc is a model parameter of the value target neural network, and Vθt′c and Vθc denote the value target function;
• wherein determining, according to the multi-agent strategy model and the object recommendation request, the object execution actions of the at least two agent objects matching the object recommendation request comprises:
• acquiring an agent observation determined by each of the at least two agent objects according to the object recommendation request;
• and acquiring an object execution action determined by each of the at least two agent objects according to the multi-agent strategy model and the agent observation;
• wherein acquiring the object execution action determined by each of the at least two agent objects according to the multi-agent strategy model and the agent observation comprises:
• acquiring, based on the following equation, the object execution action determined by each of the at least two agent objects according to the multi-agent strategy model and the agent observation:
• ati=σ0a(oti)
• wherein ati denotes the object execution action, πθa denotes the multi-agent strategy model, θa denotes a model parameter of the multi-agent strategy model, and of denotes the agent observation;
• wherein determining the target object execution action according to the object execution actions comprises:
• selecting, among the object execution actions, an object execution action that has a largest action value as the target object execution action.