Wireless network resource allocation method employing generative adversarial reinforcement learning

A wireless network resource allocating method based on generative adversarial reinforcement learning, wherein a generative adversarial reinforcement learning algorithm comprises two neural networks, which are a generator network G and a discriminator network D, wherein the method comprises:

• (1) initializing the generator network G and the discriminator network D, comprising:
• (1.1) randomly initializing weights of the generator network G and the discriminator network D through Gaussian distribution;
• and setting a network Ĝ which has a same structure as that of the generator network G, and initializing a weight of the network Ĝ by copying the weight of the generator network G;
• (1.2) inputting a network state s into the generator network G, outputting an N×N-dimensional vector by the generator network G, and sequentially dividing the Na×N-dimensional vector output by the generator network G into Na N-dimensional vectors;
• inputting an N-dimensional vector into the discriminator network D, the N-dimensional vector input into the discriminator network D being taken from an output of the generator network G or being obtained by calculation based on an output of the network Ĝ and an instant return;
• and
• determining by the discriminator network D that the N-dimensional vector input into the discriminator network D is taken from the output of the generator network G if an absolute value of a difference between a scalar and 0 is less than an absolute value of a difference between the scalar and 1, and determining by the discriminator network D that the N-dimensional vector input into the discriminator network D is obtained by calculation based on the output of the network Ĝ and the instant return if the absolute value of the difference between the scalar and 1 is less than the absolute value of the difference between the scalar and 0, where the scalar is output by the generator network G for representing authenticity of an input;
• wherein N represents the number of samples sampled from Z(s,a), Z(s,a) represents a cumulative return distribution obtained from an action a under the network state s, the network state s represents the number of requests for each type of service in a time interval, the action a represents a magnitude of a bandwidth allocated for each type of service, Na represents the number of effective actions, and a pth vector in the Na N-dimensional vectors represents sampling values of an overall return distribution obtained from a pth action;
• (2) performing resource allocation, comprising:
• (2.1) acquiring an observed value st of the network state s at a current time moment t by a radio resource manager;
• selecting an action at using a ϵ-greedy strategy by the radio resource manager;
• receiving a system return value J by the radio resource manager when the action at is executed, and observing an observed value st+1 of the network state s at a time moment t+1;
• wherein selecting the action at using the ϵ-greedy strategy by the radio resource manager comprises:
• acquiring a random number from a (0,1) uniform distribution by the radio resource manager;
• randomly selecting an effective action by the radio resource manager if the random number is less than ϵ;
• inputting the observed value st into the generator network G by the radio resource manager to obtain sampling values of cumulative return distributions of Na actions if the random number is greater than or equal to ϵ;
• calculating a mean value of the sampling values of the cumulative return distribution of each action;
• and selecting an action corresponding to a maximum mean value;
• (2.2) setting two thresholds c1 and c2 and an absolute value ξ of a fixed instant return by the radio resource manager, where c1>c2, and setting an instant return rt at the time moment t to be ξ when J>c1, to be 0 when c2<J<c1, and to be −ξ when J<c2;
• (2.3) storing quadruples (st, at, rt, st+1) by the radio resource manager in a buffer area with a size of NB;
• deleting a quadruple earliest stored in the buffer area and storing a newest quadruple into the buffer area when the buffer area is full;
• (3) every K times the resource allocation of the step (2) is performed, training the weights of the generator network G and the discriminator network D using the quadruples stored in the buffer area , comprising:
• (3.1) training the discriminator network D first, comprising:
• randomly selecting m quadruples (st, at, rt, st+1) from the buffer area as training data;
• combining the observed values st of the network state at the time moment t in the m quadruples into an m×Ns matrix [s1, s2, . . . sm]T, where sm represents an mth observed value st of the network state at the time moment t;
• inputting the m×Ns matrix [s1, s2, . . . sm]T into the generator network Ĝ to obtain sampling values of the cumulative return distribution of each action under the m observed value st of the network state at the time moment t, and retaining sampling values corresponding to a1, a2, . . . am, denoted as G(s1), G(s2), . . . G(sm), where Ns represents the number of service types, G(sm) represents N sampling values of a return obtained by taking the action am under the mth observed value st of the network state at the time moment t, which are recorded as sampling values of a distribution of an estimated action value;
• combining m observed values st+1 of the network state at the time moment t+1 in the training data into an m×Ns matrix [s1′, s2′, . . . sm′]T, and inputting the m×Ns matrix [s1′, s2′, . . . sm′]T into the network Ĝ to obtain sampling values of the cumulative return distribution of each action under the m observed value st+1 of the network state at the time moment t+1, and retaining sampling values of mean values of the maximum cumulative return generated under each observed value st+1 of the network state at the time moment t+1, which are denoted as Ĝ(s1′), Ĝ(s2′), . . . Ĝ(sm′), where sm′ represents a mth observed value st+1 of the network state at the time moment t+1;
• making
• yi=ri+γ*Ĝ(si′), i=1,2, . . . m (1),
• where yi represents a sampling value of a distribution of a target action value, ri represents the instant return, and γ represents a discount factor;
• randomly acquiring m samples from a (0,1) uniform distribution, denoted as ε1, ε2, . . . εm;
• making
• {circumflex over (x)}i=εi*yi+(1−εi)*G(si), i=1,2, . . . m (2),
• where {circumflex over (x)}i represents a weighted sum of the sampling value of the distribution of the target action value and the sampling value of the distribution of the estimated action value;
• wherein a loss function LD of the discriminator network D is:
• L
• D
• =
• 1
• m
• ⁢
• ∑
• i
• =
• 1
• m
• ⁢
• (
• D
• ⁡
• (
• G
• ⁡
• (
• s
• i
• )
• )
• -
• D
• ⁡
• (
• y
• i
• )
• +
• λ
• ⁡
• (
• 
• ∇
• x
• ^
• i
• ⁢
• D
• ⁡
• (
• x
• ^
• i
• )
• 
• 2
• -
• 1
• )
• 2
• )
• (
• 3
• )
• where D(G(s)) represents an output of the discriminator network D when an input is G(si);
• D(yi) represents an output of the discriminator network D when an input is yi;
• D({circumflex over (x)}i) represents an output of the discriminator network D when an input is {circumflex over (x)}i, ∇{circumflex over (x)}iD({circumflex over (x)}i) represents a gradient value obtained by derivation of D({circumflex over (x)}i) with respect to {circumflex over (x)}i, and λ represents a penalty factor, and
• training the weight of the discriminator network D using a gradient descent algorithm to complete the training of the discriminator network D for one time;
• (3.2) obtaining a newest weight of the discriminator network D to participate in the training of the generator network G, after training the discriminator network D for nd times, wherein a loss function LG of the generator network G is:
• L
• G
• =
• -
• 1
• m
• ⁢
• ∑
• i
• =
• 1
• m
• ⁢
• (
• D
• ⁡
• (
• G
• ⁡
• (
• s
• i
• )
• )
• )
• ;
• (
• 4
• )
• and
• training the weight of the generator network G using a gradient descent algorithm;
• (3.3) copying the weight of the generator network G to the network Ĝ to update the weight of the network G, every C times the training step (3.1) and (3.2) are performed;
• and
• (4) completing the training of the discriminator network D and the generator network G after executing the step (3) for Ntrain times,
• wherein the radio resource manager is configured to input a current network state into the generator network G, the generator network G is configured to output the sampling values of the cumulative return distribution corresponding to each resource allocation strategy, a mean value of the sampling values of the return of each resource allocation strategy is calculated, and an action corresponding to a maximum mean value is taken as a resource allocation strategy of the radio resource manager;
• and
• (5) allocating the wireless network resource according to the resource allocation strategy corresponding to the maximum mean value determined in step (4).