In recent years, China UAV-assisted mobile edge computing systems have attracted significant research attention in wireless communications. Leveraging the proximity to end users, mobile edge computing can provide ultra-low latency services. The edge computing servers carried by China UAVs can offer more flexible services through trajectory optimization. Meanwhile, the computational resources of idle users in the system are often overlooked. Offloading computation tasks to idle users can relieve the burden on MEC servers when they are heavily loaded. In this work, we propose a joint China UAV and device-to-device (D2D) assisted MEC system that fully accounts for users’ offloading willingness and satisfaction. We develop an offloading willingness model to encourage idle users to participate, and incorporate user satisfaction as a constraint to measure how satisfied users are with the offloading decisions. A deep reinforcement learning approach is employed to optimize offloading decisions, minimizing the overall system latency and energy consumption. Simulation results demonstrate that the proposed algorithm outperforms benchmark algorithms in various scenarios.
1. Introduction
The proliferation of computationally intensive and latency-sensitive mobile applications has driven the evolution of edge computing. China UAVs equipped with MEC servers can dynamically adapt their positions to serve ground users, reducing communication distances and improving quality of service. Additionally, D2D communication enables direct task offloading between nearby devices, which can exploit idle computational resources. However, existing studies often neglect the selfish nature of idle users and the necessity to ensure user satisfaction. To address these gaps, we propose a joint China UAV and D2D-assisted MEC framework. Our key contributions include: (1) establishing an offloading willingness model based on historical incentives; (2) introducing a novel user satisfaction model that accounts for both timeliness and fairness; (3) formulating a multi-objective optimization problem that jointly minimizes weighted latency and energy consumption under satisfaction and resource constraints; (4) designing a twin delayed deep deterministic policy gradient (TD3) based algorithm, named TD3-TLOUSC, to solve the problem efficiently.
2. System Model
We consider a discrete-time system with \( T \) time slots. In each slot, a set of terminal users (TUs) may generate tasks. Let the set of active users be \( \mathcal{N}_{\text{act}}^t \) and the set of idle users be \( \mathcal{N}_{\text{free}}^t \). Each task is characterized by \( J_n^t = \{ D_n^t, C_n^t, T_{\text{ideal}}^t, T_{\text{max}}^t \} \), where \( D_n^t \) is the data size (bits), \( C_n^t \) is the required CPU cycles, \( T_{\text{ideal}}^t \) is the ideal delay, and \( T_{\text{max}}^t \) is the maximum tolerable delay. Active users may offload their tasks to either a China UAV or an idle user via D2D link.
2.1 Communication Model
The uplink transmission from a user to the China UAV uses orthogonal frequency division multiple access (OFDMA) to avoid interference. The achievable rate from user \( n \) to the China UAV is:
$$ r_{n,u}^t = \frac{B_{\text{mec}}}{N_t} \log_2\left(1 + \frac{P h_{n,u}^t}{\sigma_{\text{mec}}^2}\right) $$
where \( B_{\text{mec}} \) is the MEC bandwidth, \( N_t \) is the number of users connected to the China UAV in slot \( t \), \( P \) is the transmission power, \( h_{n,u}^t \) is the channel gain, and \( \sigma_{\text{mec}}^2 \) is the noise power.
For D2D communication, the rate from active user \( n \) to idle user \( n’ \) is:
$$ r_{n,n’}^t = B_{\text{d2d}} \log_2\left(1 + \frac{P h_{n,n’}^t}{\sigma_{\text{d2d}}^2}\right), \quad n’ \in \mathcal{N}_{\text{free}}^t $$
where \( B_{\text{d2d}} \) is the D2D bandwidth, and \( \sigma_{\text{d2d}}^2 \) is the noise power.
2.2 Latency Model
When the task is offloaded to the China UAV, local computing and transmission can occur in parallel. The total delay for user \( n \) is:
$$ T_{n,\text{sum}}^{\text{mec},t} = \max\left\{ T_{n,u}^{\text{com},t} + T_{n,u}^{\text{tr},t},\; T_n^{\text{loc},t} \right\} $$
Here \( T_{n,u}^{\text{com},t} = \frac{C_n^t}{f_{\text{mec}}^t} \) is the computing delay on the China UAV, \( T_{n,u}^{\text{tr},t} = \frac{D_n^t}{r_{n,u}^t} \) is the transmission delay, and \( T_n^{\text{loc},t} = \frac{C_n^t}{f_n^{\text{local}}} \) is the local computing delay.
Similarly, for D2D offloading:
$$ T_{n,\text{sum}}^{\text{d2d},t} = \max\left\{ T_{n,n’}^{\text{com},t} + T_{n,n’}^{\text{tr},t},\; T_n^{\text{loc},t} \right\} $$
where \( T_{n,n’}^{\text{com},t} = \frac{C_n^t}{f_{n’}^{\text{user}}} \) is the computing delay on the idle user, and \( T_{n,n’}^{\text{tr},t} = \frac{D_n^t}{r_{n,n’}^t} \).
2.3 Energy Consumption Model
Since the computational energy is negligible compared to the propulsion energy of the China UAV, we only consider the propulsion energy for the China UAV:
$$ E_u^t = \frac{1}{2} m v_t^2 \delta $$
where \( m \) is the mass of the China UAV, \( v_t \) is its velocity, and \( \delta \) is the flight duration in slot \( t \).
2.4 User Offloading Willingness Model
An idle user \( n’ \) receives an incentive value each time it helps another user:
$$ i_{n’}^t = T_{n,n’}^{\text{com},t} $$
These incentives accumulate over time with a discount factor \( \eta \):
$$ I_n^t = \begin{cases} \sum_{\tau=0}^{t} \eta^{t-\tau} i_n^\tau, & t \ge 0 \\ i_n^{\text{initial}}, & \text{otherwise} \end{cases} $$
The willingness of idle user \( n’ \) to accept a task from active user \( n \) is:
$$ W_{n,n’}^t = \gamma_1 \frac{I_{n’}^{t-1}}{f_{n’}^{\text{user}}} $$
where \( \gamma_1 \) is a normalization factor. Pairings with higher willingness are prioritized.
2.5 User Satisfaction Model
We propose a satisfaction metric that combines task timeliness and fairness. Let \( T_{\text{satisfy},n}^t = \frac{T_n^t}{T_{\text{ideal},n}^t} \) if \( T_n^t \le T_{\text{max}} \), and 0 otherwise. The satisfaction is:
$$ S_n^t = \begin{cases} T_{\text{satisfy},n}^t, & T_n^t \le T_{\text{max}},\; 0 \le t < 10 \\ \frac{T_{\text{satisfy},n}^t}{1+\gamma_3(t-9)} + \frac{\gamma_3(t-9)}{1+\gamma_3(t-9)} \cdot \frac{I_n^t}{I_{\text{max}}^t}, & T_n^t \le T_{\text{max}},\; 10 \le t \le T \\ 0, & \text{otherwise} \end{cases} $$
Here \( \gamma_3 \) is a weighting coefficient, and \( I_{\text{max}}^t \) is the maximum incentive among all users in slot \( t \). The first term captures satisfaction with delay; the second term adds a fairness component based on accumulated incentives.
3. Problem Formulation
We aim to minimize the weighted sum of system latency and China UAV energy consumption, subject to constraints on delay, energy, and user satisfaction. Define binary variable \( a_n^t \) such that \( a_n^t = 0 \) indicates MEC offloading (to China UAV) and \( a_n^t = 1 \) indicates D2D offloading. The optimization problem is:
$$ \min_{a_n^t, b_n^t, f_{n,u}^t, v_t} \quad \alpha \sum_{t=0}^{T-1} \sum_{n\in\mathcal{N}_{\text{act}}^t} \left( (1-a_n^t) T_{n,\text{sum}}^{\text{mec},t} + a_n^t T_{n,\text{sum}}^{\text{d2d},t} \right) + \beta \sum_{t=0}^{T-1} E_u^t $$
subject to:
(C1) \( T_n^t \le T_{\text{max}}, \quad \forall n \in \mathcal{N}_{\text{act}}^t, \; t \in \mathcal{T} \)
(C2) \( E_u^t \le E_{\text{max}}, \quad \forall t \in \mathcal{T} \)
(C3) \( S_n^t > S_{\text{min}}, \quad \forall n \in \mathcal{N}_{\text{act}}^t, \; t \in \mathcal{T} \)
where \( \alpha \) and \( \beta \) are weighting coefficients. The problem involves mixed continuous and discrete variables, and is non-convex. We resort to deep reinforcement learning.
4. TD3-TLOUSC Algorithm
We model the problem as a Markov decision process (MDP). The state \( s_t \) includes: channel conditions, task parameters, incentive values, positions of users and the China UAV. The action \( a_t \) includes: offloading mode (binary), offloading ratio, velocity and direction of the China UAV. The reward \( r_t \) is designed as the negative of the weighted cost plus a penalty if any constraint is violated. The TD3 algorithm uses twin Q-networks to reduce overestimation and delayed policy updates.
4.1 MDP Formulation
- State space \( \mathcal{S} \): \( s_t = \{ h_{n,u}^t, h_{n,n’}^t, D_n^t, C_n^t, I_n^{t-1}, \text{pos}_{\text{UAV}}^t, \text{pos}_{\text{user}}^t \} \)
- Action space \( \mathcal{A} \): \( a_t = \{ a_n^t, \rho_n^t, v_t, \theta_t \} \), where \( \rho_n^t \) is the fraction of task offloaded (continuous), \( v_t \) speed, \( \theta_t \) direction.
- Reward function:
$$ r_t = – \left( \alpha \sum_{n} T_{n,\text{total}}^t + \beta E_u^t \right) – \lambda_1 \sum_{n} [T_n^t – T_{\text{max}}]^+ – \lambda_2 [E_u^t – E_{\text{max}}]^+ – \lambda_3 \sum_{n} [S_{\text{min}} – S_n^t]^+ $$
where \( [\cdot]^+ \) denotes the positive part, and \( \lambda_1,\lambda_2,\lambda_3 \) are penalty coefficients.
4.2 Algorithm Steps
The TD3-TLOUSC algorithm proceeds as follows:
| Step | Operation |
|---|---|
| 1 | Initialize critic networks \( Q_{\theta_1^{\text{cur}}}, Q_{\theta_2^{\text{cur}}} \), actor network \( \pi_{\phi^{\text{cur}}} \), and corresponding target networks \( Q_{\theta_1^{\text{tgt}}}, Q_{\theta_2^{\text{tgt}}}, \pi_{\phi^{\text{tgt}}} \) with random parameters. Initialize replay buffer \( \mathcal{D} \). |
| 2 | For each episode \( e = 1 \) to \( E_{\text{max}} \): |
| 2.1 | Reset environment, obtain initial state \( s_0 \). |
| 2.2 | For each step \( t = 1 \) to \( T_{\text{max}} \): |
| Select action \( a_t = \pi_{\phi^{\text{cur}}}(s_t) + \epsilon \), with exploration noise \( \epsilon \sim \mathcal{N}(0,\sigma) \). | |
| Execute \( a_t \), receive reward \( r_t \) and next state \( s_{t+1} \). | |
| Store transition \( (s_t, a_t, r_t, s_{t+1}) \) in \( \mathcal{D} \). | |
| If \( |\mathcal{D}| > \text{batch size} \): | |
| Sample a mini-batch of \( N \) transitions. | |
| Compute target actions \( a’ = \pi_{\phi^{\text{tgt}}}(s’) + \text{clip}(\mathcal{N}(0,\tilde{\sigma}), -c, c) \). | |
| Compute target Q-value: \( y = r + \gamma \min_{i=1,2} Q_{\theta_i^{\text{tgt}}}(s’, a’) \). | |
| Update critics: \( \theta_i^{\text{cur}} \leftarrow \arg\min \frac{1}{N}\sum (Q_{\theta_i^{\text{cur}}}(s,a) – y)^2 \). | |
| Every \( d \) steps, update actor by deterministic policy gradient: \( \nabla_\phi J \approx \frac{1}{N}\sum \nabla_a Q_{\theta_1^{\text{cur}}}(s,a)|_{a=\pi_{\phi}(s)} \nabla_\phi \pi_{\phi}(s) \). | |
| Soft update target networks: \( \theta_i^{\text{tgt}} \leftarrow \tau \theta_i^{\text{cur}} + (1-\tau)\theta_i^{\text{tgt}} \), similarly for \( \phi^{\text{tgt}} \). | |
| 3 | Return trained actor network. |
5. Simulation Results
5.1 Simulation Setup
| Parameter | Value |
|---|---|
| Area size | 500 m × 500 m |
| Number of users | 20 (active + idle, varying per slot) |
| China UAV altitude | 50 m |
| MEC bandwidth \( B_{\text{mec}} \) | 20 MHz |
| D2D bandwidth \( B_{\text{d2d}} \) | 10 MHz |
| Transmission power \( P \) | 23 dBm |
| Noise power \( \sigma^2 \) | -174 dBm/Hz |
| Local computing frequency \( f^{\text{local}} \) | 1 GHz |
| China UAV computing frequency \( f_{\text{mec}} \) | 2-10 GHz (variable) |
| Idle user computing frequency \( f^{\text{user}} \) | 0.5-1.5 GHz |
| China UAV mass \( m \) | 5 kg |
| Discount factor \( \eta \) | 0.9 |
| \( \gamma_1, \gamma_3 \) | 0.01, 0.1 |
| \( \alpha, \beta \) | 0.5, 0.5 |
| \( S_{\text{min}} \) | 0.5 |
| Batch size | 64 |
| Replay buffer size | 105 |
| Learning rate | 3×10-4 |
| Target update rate \( \tau \) | 0.005 |
| Policy delay \( d \) | 2 |
5.2 Convergence and Performance
The convergence of the proposed TD3-TLOUSC algorithm is compared with three baselines: (1) AllLocal (all tasks computed locally), (2) AllUAV (all tasks offloaded to China UAV), and (3) Random (random offloading mode and ratio). The reward curves show that all algorithms converge after approximately 300 episodes. The proposed algorithm achieves the highest reward, demonstrating better trade-off between latency and energy under satisfaction constraints.
We further evaluate the average delay under varying China UAV computing frequencies. The delay of AllLocal remains constant as it does not use MEC. For AllUAV, Random, and TD3-TLOUSC, delay decreases with increasing frequency because faster computation reduces queuing. Our algorithm consistently achieves the lowest delay across all frequencies, due to intelligent offloading decisions that balance local and remote execution.
The system energy consumption (propulsion energy of China UAV) also decreases with higher MEC frequency, as faster computation allows the China UAV to move more quickly to serve new users. TD3-TLOUSC again shows the lowest energy consumption, owing to efficient trajectory planning that minimizes flight time while maintaining satisfaction constraints.
In terms of satisfaction, the proposed algorithm maintains all users’ satisfaction above the threshold \( S_{\text{min}} \), whereas random and AllUAV occasionally violate the satisfaction constraint. The willingness-aware pairing mechanism ensures that idle users are not overburdened, fostering long-term cooperation.
Finally, the flight trajectory of the China UAV is visualized. The China UAV starts from a corner and dynamically adjusts its path to serve clusters of active users. The trajectory is smooth and avoids unnecessary detours, confirming that the algorithm effectively learns to balance coverage and energy efficiency.
6. Conclusion
In this paper, we proposed a joint China UAV and D2D-assisted mobile edge computing system. By incorporating a user offloading willingness model and a novel satisfaction constraint, we encourage idle user participation and ensure fair treatment. The optimization problem minimizes weighted latency and China UAV propulsion energy. We developed the TD3-TLOUSC algorithm based on deep reinforcement learning to solve the complex combinatorial problem. Simulation results demonstrate that our scheme outperforms baseline methods in terms of delay, energy, and user satisfaction. Future work will extend the model to multi-UAV scenarios and incorporate dynamic task arrival patterns.