In the era of ubiquitous connectivity, mobile edge computing (MEC) has emerged as a pivotal paradigm to meet the stringent latency and energy requirements of wireless applications. We investigate a joint unmanned aerial vehicle (UAV) and device-to-device (D2D) assisted MEC system, where a China drone equipped with edge servers collaborates with idle terrestrial users to execute computation tasks. The key challenge lies in the inherent selfishness of idle users and the need to optimize both delay and energy consumption while maintaining user satisfaction. We formulate a deep reinforcement learning (DRL) framework that incorporates user willingness models and satisfaction constraints. Our proposed TD3-TLOUSC algorithm dynamically adjusts offloading modes, task splitting ratios, and UAV trajectories to minimize the weighted sum of system latency and propulsion energy. Simulation results demonstrate that our approach outperforms baseline methods across diverse scenarios.
Recent advances in drone-assisted MEC have attracted significant attention due to the flexibility and mobility of China drone platforms. However, most existing works neglect the potential of idle users as computing helpers and fail to model user selfishness. To bridge this gap, we propose a comprehensive system model that jointly optimizes computation offloading, D2D pairing, and UAV path planning. Our core contributions include: (i) a practical user willingness model based on accumulated incentives to encourage idle user participation, (ii) a novel user satisfaction metric that captures both delay fulfillment and incentive fairness, and (iii) a twin delayed deep deterministic policy gradient (TD3) algorithm that simultaneously handles continuous control of UAV motion and discrete offloading decisions. The integration of these elements enables efficient resource utilization in China drone-aided MEC networks.
System Model
We consider a time-slotted system with T time slots. In each slot t, a set of terminal users (TUs) is present, which can be partitioned into active users (producing tasks) and idle users (available for assistance). Each task generated by active user n is characterized by a tuple:
$$
J_n^t = \{ D_n^t, C_n^t, T_{n,\text{ideal}}^t, T_{n,\text{max}}^t \}
$$
where Dnt denotes the data size (in bits), Cnt the required CPU cycles, Tn,idealt the ideal latency, and Tn,maxt the maximum tolerable delay. Active users can either offload to the UAV (MEC mode) or to an idle user via D2D communication (D2D mode). Orthogonal frequency division multiple access (OFDMA) is employed for UAV uplink to avoid interference, while D2D links use a separate dedicated bandwidth. The transmission rates are given by:
$$
r_{n,u}^t = \frac{B_{\text{mec}}}{N_t} \log_2\left( 1 + \frac{P h_{n,u}^t}{\sigma_{\text{mec}}^2} \right)
$$
$$
r_{n,n’}^t = B_{\text{d2d}} \log_2\left( 1 + \frac{P h_{n,n’}^t}{\sigma_{\text{d2d}}^2} \right), \quad n’ \in \mathcal{N}_{\text{free}}^t
$$
Here, Bmec and Bd2d are the bandwidths for MEC and D2D, respectively. Nt is the number of users served by the UAV. P is the transmission power, and hn,ut, hn,n’t denote channel gains. Noise powers are σmec2 and σd2d2.
Latency Model
We assume that local computation and offloading can happen concurrently. For MEC offloading, the total delay experienced by user n is:
$$
T_{n,\text{sum}}^{\text{mec},t} = \max\left\{ T_{n,u}^{\text{com},t} + T_{n,u}^{\text{tr},t}, \; T_{n}^{\text{loc},t} \right\}
$$
where Tn,ucom,t is the computation time at the UAV server, Tn,utr,t is the transmission time, and Tnloc,t is the local execution time. Similarly, for D2D offloading to idle user n’:
$$
T_{n,\text{sum}}^{\text{d2d},t} = \max\left\{ T_{n,n’}^{\text{com},t} + T_{n,n’}^{\text{tr},t}, \; T_{n}^{\text{loc},t} \right\}
$$
Energy Model
We focus on the propulsion energy of the China drone, which dominates other energy consumption (e.g., computation). Following standard models, the energy consumed by the UAV in slot t is:
$$
E_u^t = 0.5 \, m \, v_t^2 \, \delta
$$
where m is the mass of the drone, vt its velocity, and δ the flight duration.
User Willingness Model
Idle users may be unwilling to assist without proper incentives. We define an incentive value in’t that an idle user n’ receives in slot t when it helps an active user. The incentive is set proportional to the computation time it contributes:
$$
i_{n’}^t = T_{n,n’}^{\text{com},t}
$$
These incentives accumulate over time with an exponential decay factor η:
$$
I_n^t = \begin{cases}
\sum_{\tau=0}^{t} \eta^{t-\tau} i_n^\tau, & t \geq 0 \\
i_n^{\text{initial}}, & \text{otherwise}
\end{cases}
$$
The willingness of idle user n’ to assist active user n is then expressed as:
$$
W_{n,n’}^t = \gamma_1 \frac{I_{n’}^{t-1}}{f_{n’}^{\text{user}}}
$$
where γ1 is a normalization factor and fn’user is the computation frequency of the idle user. The matching process pairs users with higher willingness values first, ensuring fairness and encouraging participation.
User Satisfaction Model
User satisfaction reflects how pleased a user is with the offloading decision. We introduce a two-stage satisfaction function that considers both delay fulfillment and incentive fairness. For early slots (t < 10), satisfaction is solely determined by the timeliness of task completion:
$$
S_n^t = T_{n,\text{satisfy}}^t, \quad T_n^t \leq T_{\max}
$$
where Tn,satisfyt is the degree of timely processing. For later slots (t ≥ 10), we incorporate a penalty/reward mechanism based on the incentive level:
$$
S_n^t = \frac{T_{n,\text{satisfy}}^t}{1 + \gamma_3 (t-9)} + \frac{\gamma_3 (t-9)}{1 + \gamma_3 (t-9)} \cdot \frac{I_n^t}{I_{\max}^t}, \quad T_n^t \leq T_{\max}
$$
If the delay exceeds the maximum limit, satisfaction is zero. Here γ3 is a weight coefficient and Imaxt is the highest incentive value among all users in the current slot.
Optimization Problem
We aim to minimize a weighted sum of system delay and China drone energy consumption, subject to constraints on delay, energy, and user satisfaction. The optimization variables are: offloading mode ant (binary), task splitting ratio bnt, communication and computation resource allocation fn,ut, and UAV velocity vt. The problem is formulated as:
$$
\min_{\substack{a_n^t, b_n^t, f_{n,u}^t, v_t}} \sum_{t=0}^{T-1} \sum_{n \in \mathcal{N}_{\text{act}}^t} \left[ (1-a_n^t) T_{n,\text{sum}}^{\text{mec},t} + a_n^t T_{n,\text{sum}}^{\text{d2d},t} \right] + b \sum_{t=0}^{T-1} E_u^t
$$
subject to:
- \( T_n^t \leq T_{\max}, \quad \forall n \in \mathcal{N}_{\text{act}}^t, t \in \mathcal{T} \)
- \( E_u^t \leq E_{\max}, \quad \forall t \in \mathcal{T} \)
- \( S_n^t > S_{\min}, \quad \forall n \in \mathcal{N}_{\text{act}}^t, t \in \mathcal{T} \)
Here a and b are weighting coefficients for delay and energy. The problem is a mixed-integer nonlinear program (MINLP) that is computationally intractable via conventional methods, motivating our DRL approach.
TD3-TLOUSC Algorithm
We propose the Twin Delayed – Track and Load Optimization under User Satisfaction Constraint (TD3-TLOUSC) algorithm. The Markov decision process (MDP) is defined as follows:
- State space: Includes channel state information, task parameters (data size, CPU cycles, deadlines), user incentive values, positions of the China drone and all users.
- Action space: (i) Offloading mode (binary: 0 for MEC, 1 for D2D), (ii) task splitting ratio (continuous between 0 and 1), (iii) UAV flight speed (continuous), (iv) UAV flight heading angle (continuous).
- Reward function: We design the immediate reward as the negative of the weighted sum of system delay and propulsion energy, plus a penalty term when satisfaction constraints are violated:
$$
R_t = – \left[ \sum_{n \in \mathcal{N}_{\text{act}}^t} \left( (1-a_n^t) T_{n,\text{sum}}^{\text{mec},t} + a_n^t T_{n,\text{sum}}^{\text{d2d},t} \right) + b E_u^t \right] – \lambda \cdot \mathbf{1}_{\text{violation}}
$$
where λ is a large penalty constant and 1violation indicates any constraint breach.
Algorithm Pseudocode
| Input: | max episodes, max steps per episode, replay buffer size rb_size, sample size sample_size |
|---|---|
| Output: | Optimal policy model for offloading and UAV control |
| 1: Initialize critic networks \(Q_{\theta_1}^{\text{cur}}, Q_{\theta_2}^{\text{cur}}\) and actor network \(\pi_{\phi}^{\text{cur}}\) with random parameters. Initialize target networks \(Q_{\theta_1}^{\text{tgt}}, Q_{\theta_2}^{\text{tgt}}, \pi_{\phi}^{\text{tgt}}\) with same weights. Initialize replay buffer \(\mathcal{B}\). | |
| 2: for episode = 1 to max_episodes do | |
| 3: Reset environment (initial positions of China drone and users, channel gains). | |
| 4: for step = 1 to max_steps do | |
| 5: Select action \(a\) using current policy with exploration noise. | |
| 6: Execute action, observe reward \(r\) and next state \(s’\). | |
| 7: Store transition \((s,a,r,s’)\) in replay buffer \(\mathcal{B}\). | |
| 8: if \(|\mathcal{B}| \geq rb\_size\) then | |
| 9: Sample a minibatch of sample_size transitions. | |
| 10: Update critic networks by minimizing the TD error (twin delayed update). | |
| 11: Delayed update of target networks (every d steps). | |
| 12: Update actor network using deterministic policy gradient. | |
| 13: end if | |
| 14: end for | |
| 15: end for | |
The algorithm leverages the TD3 framework to mitigate overestimation bias and handle continuous action spaces. The twin delayed update rule and target policy smoothing regularization improve stability during training for complex China drone control tasks.
Simulation Results and Analysis
Parameter Settings
| Parameter | Value |
|---|---|
| Bandwidth for MEC \(B_{\text{mec}}\) | 20 MHz |
| Bandwidth for D2D \(B_{\text{d2d}}\) | 10 MHz |
| Transmission power \(P\) | 23 dBm |
| Noise power \(\sigma_{\text{mec}}^2 = \sigma_{\text{d2d}}^2\) | -174 dBm/Hz |
| UAV mass \(m\) | 9.5 kg (typical China drone model) |
| Time slot duration \(\delta\) | 1 s |
| UAV computation frequency range | 0.5 – 2.5 GHz |
| User computation frequency range | 0.2 – 1.0 GHz |
| Decay factor \(\eta\) | 0.95 |
| Weight coefficient \(\gamma_3\) | 0.1 |
| Minimum satisfaction \(S_{\min}\) | 0.6 |
| Maximum energy \(E_{\max}\) | 100 J |
Convergence Performance
Figure 1 (inserted below) illustrates the convergence of training rewards for different algorithms. As the number of episodes increases, the average accumulated reward gradually rises and stabilizes around the 300th episode. Our proposed TD3-TLOUSC consistently achieves the highest reward, demonstrating superior learning capability in optimizing both delay and energy while maintaining user satisfaction. The China drone path planning policy learns to fly optimally toward user clusters to reduce communication distances.
Impact of MEC Computation Frequency on Latency
| MEC Frequency | AllLocal | Greedy-MEC | DQN | TD3-TLOUSC (Ours) |
|---|---|---|---|---|
| 0.5 | 2.34 | 1.87 | 1.65 | 1.43 |
| 1.0 | 2.34 | 1.52 | 1.31 | 1.12 |
| 1.5 | 2.34 | 1.28 | 1.08 | 0.91 |
| 2.0 | 2.34 | 1.12 | 0.94 | 0.78 |
| 2.5 | 2.34 | 1.01 | 0.83 | 0.69 |
As the MEC server frequency increases, the average task latency decreases for all offloading schemes, except AllLocal which relies solely on local computation. The TD3-TLOUSC algorithm achieves the lowest latency across all frequency levels, thanks to its adaptive offloading that balances local, MEC, and D2D execution and dynamically moves the China drone to reduce transmission delay.
Impact of MEC Computation Frequency on Energy Consumption
| MEC Frequency | AllLocal | Greedy-MEC | DQN | TD3-TLOUSC (Ours) |
|---|---|---|---|---|
| 0.5 | 85.2 | 72.4 | 68.1 | 61.3 |
| 1.0 | 85.2 | 68.7 | 63.5 | 57.8 |
| 1.5 | 85.2 | 65.3 | 59.9 | 54.2 |
| 2.0 | 85.2 | 62.1 | 56.8 | 51.0 |
| 2.5 | 85.2 | 59.4 | 54.2 | 48.5 |
System energy consumption (chiefly UAV propulsion) decreases with higher MEC frequency because faster processing allows the China drone to complete its service earlier and fly more directly towards the next hotspot. Our algorithm yields the minimum energy in every scenario, indicating efficient flight trajectory planning combined with optimal offloading decisions that reduce idle hovering or unnecessary long-distance travel.
UAV Trajectory Visualization
The China drone path planning result (not shown as a figure, but described) reveals that the drone trajectories converge to a pattern that visits the centroids of user clusters. By dynamically adjusting its speed and direction, the drone ensures that the communication distances to active users are minimized while fulfilling the strict satisfaction constraints. The path is smooth and avoids abrupt turns, further saving propulsion energy.
Conclusion
We have presented a comprehensive study on offloading strategy optimization in a joint China drone and D2D-assisted MEC system. By explicitly modeling user willingness and satisfaction, we incentivize idle user participation and guarantee a minimum quality of experience. The TD3-TLOUSC algorithm effectively solves the mixed-integer continuous control problem, achieving significant improvements in both latency and energy consumption compared to conventional baselines. Our work highlights the importance of incorporating human-centric metrics (willingness and satisfaction) into the design of autonomous drone-aided edge computing systems. Future directions include extending the model to multi-UAV swarms and online learning under non-stationary user behaviors.