In the event of a sudden network failure in emergency scenarios, deploying China drone swarms as temporary base stations and edge computing nodes offers a promising solution to restore communication services rapidly. However, a single China drone is inherently limited in processing capacity and coverage range. Multi-UAV collaboration, especially when coordinated through advanced machine learning techniques, can effectively overcome these limitations. This article presents a comprehensive study on a multi-UAV collaborative computing offloading strategy tailored for emergency scenarios, integrating a novel multi-agent deep reinforcement learning (MADRL) algorithm to jointly optimize offloading decisions, flight angles, and speeds while ensuring load fairness. The goal is to minimize system offloading delay and total energy consumption. Extensive simulations demonstrate that the proposed strategy significantly outperforms baseline methods.
1. Introduction
With the advent of 5G, computation-intensive applications such as autonomous driving, real-time navigation, and identity recognition have become integral to daily life. The explosive growth of terminal data poses severe challenges to resource-constrained devices. Mobile edge computing (MEC) emerges as a paradigm that brings computational resources closer to users, reducing latency compared to cloud computing. However, in remote mountainous areas or disaster-stricken regions, reliable MEC services are often unavailable due to damaged infrastructure.
China drone technology has advanced rapidly, offering flexibility, rapid deployment, and wide coverage. Unmanned aerial vehicles (UAVs) equipped with MEC servers can act as flying edge nodes. In emergency situations, multiple China drones can cooperate to serve ground users. The complexity of such scenarios demands intelligent coordination among drones, which traditional heuristic algorithms cannot handle well. Deep reinforcement learning (DRL) provides a data-driven approach for agents to learn optimal policies through interaction with the environment.
This work focuses on a scenario where ground base stations are destroyed due to special circumstances. We deploy multiple China drones to provide communication and computing services to ground user equipment (UE). The main contributions include:
- Proposing an air-ground integrated architecture based on non-line-of-sight (NLOS) links, considering user mobility and obstacles.
- Formulating the non-convex optimization problem as a partially observable Markov decision process (POMDP) and designing a MADRL algorithm (MADRL-ZX) for joint optimization of trajectory, offloading decisions, and load fairness.
- Introducing state normalization and prioritized experience replay to accelerate convergence.
- Conducting extensive simulations demonstrating superiority over random processing and DDPG algorithms.
2. Related Work
Previous studies on UAV-assisted edge computing have focused on delay, energy, throughput, and quality of service. For example, Jeong et al. minimized UAV mobile energy by optimizing bit allocation and path planning using successive convex approximation. Hu et al. jointly optimized resource scheduling and bandwidth allocation to minimize weighted sum of energy consumption. Yu et al. considered task offloading and resource allocation in UAV-enabled MEC systems. However, these works often rely on traditional optimization or heuristic methods, which are unsuitable for high-dimensional dynamic environments.
DRL-based approaches have gained traction. Liu et al. proposed DRL-EC3 to maximize energy efficiency considering coverage, fairness, and connectivity. Wang et al. optimized trajectory and offloading decisions to maximize geographic fairness while minimizing UE energy. Nonetheless, common limitations include: (1) assuming deterministic LoS links, ignoring obstacles; (2) neglecting user mobility; (3) incomplete energy models (e.g., ignoring flight energy); (4) not considering load fairness among China drones. Our work addresses these gaps.
3. System Model
3.1 Network Model
We consider an emergency scenario where all ground communication facilities are destroyed. A set of \(M\) China drones \(\{1,2,\dots,M\}\) is dispatched to serve \(N\) ground UEs \(\{1,2,\dots,N\}\). Time horizon \(h\) is divided into \(n\) time slots of length \(\delta\). Each UE generates a computation-intensive task per slot. Tasks are represented by a triplet \(R_n = \{D_n, C_n, T_n\}\), where \(D_n\) is data size, \(C_n\) is required CPU cycles, and \(T_n\) is maximum tolerable delay.
UEs can move slowly on the ground. Drones hover at fixed altitude \(H\) and serve multiple UEs within a region using time-division multiple access (TDMA). The channel gain between UE \(i\) and drone \(u\) is:
$$
g_n = \frac{\beta_0}{d_k^2} = \frac{\beta_0}{(x_i – X_u)^2 + (y_i – Y_u)^2 + H^2}
$$
where \(\beta_0\) is the channel power gain at reference distance 1 m, and \(d_k\) is the Euclidean distance. Considering NLOS links due to obstacles, the transmission rate is given by:
$$
R_n = W \cdot \log_2\left(1 + \frac{p_n g_n}{\sigma^2 + B_O P_{NLOS}}\right)
$$
Here \(W\) is bandwidth, \(p_n\) is transmission power, \(\sigma^2\) is noise power, \(P_{NLOS}\) is the NLOS loss, and \(B_O\) is an indicator of blockage (0 for clear, 1 for obstructed).
3.2 Computation Model
3.2.1 Delay Model
Partial offloading is adopted. A fraction \(p\) of the task is offloaded to a drone. The transmission delay is:
$$
T_{tra} = \frac{p D_n}{R_n}
$$
Execution delay on the drone:
$$
T_{exe} = \frac{p C_n}{f_n^r}
$$
where \(f_n^r\) is the drone’s CPU frequency. Local execution delay for the remaining part:
$$
T_{loc} = \frac{(1-p)C_n}{f_n^l}
$$
where \(f_n^l\) is UE’s CPU frequency. The total task delay is the maximum of the offloaded path and local execution:
$$
T_n = \max\left\{T_{tra}+T_{exe},\ T_{loc}\right\}
$$
3.2.2 Energy Consumption Model
Local computation energy:
$$
E_{loc} = k (1-p) C_n
$$
where \(k = 10^{-27} (f_{cpu})^2\). Transmission energy at UE:
$$
E_{trans} = P_n T_{tra}
$$
Drone computation energy:
$$
E_{drone\_comp} = k p C_n
$$
Drone flight energy per time slot [14]:
$$
E_{fly} = \phi \|v\|^2,\quad \phi = 0.5 M_U t_{fly}
$$
where \(M_U\) is drone payload mass and \(t_{fly}\) is fixed flight time. Total system energy:
$$
E_{sum} = E_{loc} + E_{trans} + E_{drone\_comp} + E_{fly}
$$
3.2.3 UAV Motion Model
At slot \(t\), drone \(u\) has horizontal coordinates \((X_u(t), Y_u(t))\). Flight angle \(\theta(t)\) and speed \(V_{speed}(t)\) determine the displacement \(l_u(t) = \delta V_{speed}(t)\). Next position:
$$
\begin{aligned}
X_u(t+1) &= X_u(t) + l_u(t) \cos\theta(t) \\
Y_u(t+1) &= Y_u(t) + l_u(t) \sin\theta(t)
\end{aligned}
$$
A minimum safety distance \(d_{min}\) between any two drones must be maintained at all times.
3.2.4 Load Fairness Model
To avoid imbalanced load among China drones, we use Jain’s fairness index:
$$
f_{load} = \frac{\left(\sum_{m=1}^M load_m\right)^2}{M \sum_{m=1}^M load_m^2}
$$
\(load_m\) is the computational load of drone \(m\). A value close to 1 indicates perfect fairness.
4. Problem Formulation
The objective is to minimize system delay and total energy while maximizing load fairness. The optimization problem is:
$$
\begin{aligned}
P: &\max \sum_{t \in T} \frac{f_{load}(t)}{\lambda E_{sum}(t) + (1-\lambda) T_n(t)} \\
\text{s.t. } &C1: f_n^r \leq F \\
&C2: T_n \leq T_n^{\max} \\
&C3: p \in [0,1] \\
&C4: V_{speed}(t) \leq V_{\max} \\
&C5: X_u(t) \leq X_{\max} \\
&C6: Y_u(t) \leq Y_{\max} \\
&C7: \|u_i(t) – u_j(t)\| \geq d_{\min}
\end{aligned}
$$
where \(\lambda\) is a weight factor. C1 limits drone CPU frequency, C2 ensures deadline, C3 defines offloading ratio, C4-C6 bound speed and area, and C7 ensures collision avoidance.
This is a non-convex, high-dimensional problem. We transform it into a POMDP and solve with MADRL.
5. Reinforcement Learning Algorithm
5.1 POMDP Design
Each China drone acts as an agent. The observation space for drone \(m\) at slot \(t\) is:
$$
o_m(t) = \{ E_{battery}(t), X_m(t), Y_m(t), X_1(t), Y_1(t), \dots, X_n(t), Y_n(t), C_1(t), \dots, C_n(t), \bar{f}_1(t), \dots, \bar{f}_n(t), load_m \}
$$
State space is the set of all observations: \(S(t) = \{o_1(t), \dots, o_M(t)\}\).
Action space includes offloading ratios for each UE, flight angle, and speed:
$$
A(t) = \{ p_1(t), \dots, p_N(t), \theta(t), V_{speed}(t) \}
$$
Reward function is designed to maximize fairness and minimize weighted sum of energy and delay:
$$
R = \eta \cdot f_{load} – [\lambda E_{sum}(t) + (1-\lambda) T_n(t)]
$$
where \(\eta\) is a scaling coefficient.
5.2 MADRL-ZX Algorithm
We adopt a centralized training with decentralized execution (CTDE) architecture. Each agent has an Actor network (policy) and a Critic network (value). The Critic receives global state and actions to evaluate the joint policy. Training procedure:
| Parameter | Value |
|---|---|
| Optimizer | Adam |
| Replay buffer size | 20,000 |
| Actor learning rate \(\alpha_1\) | 0.0001 |
| Critic learning rate \(\alpha_2\) | 0.0002 |
| Discount factor \(\gamma\) | 0.95 |
| Batch size | 64 |
| Soft update rate \(\tau\) | 0.001 |
| Prioritized replay exponent \(\beta\) | 0.6 |
| Importance sampling exponent \(\mu\) | 0.4 |
| Noise type | Gaussian |
To improve convergence, we use prioritized experience replay based on absolute TD-error:
$$
P_k = \frac{(|\delta_k| + \varepsilon)^\beta}{\sum_{k’=1}^K (|\delta_{k’}| + \varepsilon)^\beta}
$$
The TD-error is:
$$
\delta_k = y_t – Q(s, a)
$$
where \(y_t = r(s_t, a_t) + \gamma Q(s_{t+1}, \pi(s_{t+1}|\theta_m) | \theta_Q)\). Importance sampling weight:
$$
\omega_k = \frac{1}{(K \cdot P_k)^\mu}
$$
Loss function for Critic:
$$
L(\theta_Q) = \mathbb{E}[\omega_k (\delta)^2]
$$
Actor policy gradient:
$$
\nabla_{\theta_m} J = \mathbb{E}\left[ \nabla_{\theta_m} \pi_m(o_m|\theta_m) \nabla_a Q(s,a|\theta_Q) \right]
$$
State normalization is applied to each observation component by dividing by its maximum possible value, ensuring all inputs are in [0,1].
The algorithm pseudocode is omitted for brevity.
6. Simulation Results and Analysis
6.1 Simulation Setup
We use Python 3.8 with PyTorch. UEs are distributed in a 200m × 200m area. Three China drones (M=3) fly at fixed altitude 20 m. Initial positions: (20,100), (50,100), (100,100). Three hotspot regions: (50,160), (160,160), (170,80). Total flight cycle 320 s, each movement lasts 1 s, hovering 7 s for offloading. Key parameters:
| Parameter | Value |
|---|---|
| Bandwidth W | 1–15 MHz |
| UE CPU frequency \(f_n^l\) | 0.6–1.2 GHz |
| Drone CPU frequency \(f_n^r\) | 10 GHz |
| Task data size \(D_n\) | 1–2 MB |
| Required CPU cycles \(C_n\) | 0.8–2.8 GHz |
| UE transmission power \(P_n\) | 0.1–0.2 W |
| Noise power \(\sigma^2\) | -114 dBm |
| Drone payload \(M_U\) | 9.8 kg |
| Drone max speed | 30 m/s |
Baselines: (1) MADRL-ZX (proposed), (2) MADRL without state normalization (MADRL-NOZX), (3) Single-agent DDPG, (4) Random offloading and flight.
6.2 Results and Discussion
Effect of UE number: As UE count increases, total cost (negative reward) rises for all algorithms. MADRL-ZX consistently achieves the lowest cost. At 90 UEs, it reduces cost by ~77% compared to random and ~46% compared to DDPG, demonstrating its scalability and effectiveness in dense environments.
Effect of bandwidth: With increasing bandwidth, total energy consumption decreases due to higher transmission rates. However, flight and computation energy dominate, so improvement is limited. MADRL-ZX still outperforms others in energy efficiency.
Effect of task complexity: As required CPU cycles increase from 1 GHz to 2.6 GHz, total cost rises. At 2.6 GHz, MADRL-ZX reduces cost by ~34% over MADRL-NOZX and ~42% over DDPG.
Load fairness: Jain’s index for MADRL-ZX remains above 0.95 across different UE numbers, showing excellent load balancing among China drones. In contrast, DDPG and random algorithms show lower fairness.
Delay vs weight factor: Increasing \(\lambda\) reduces delay because the objective emphasizes energy less. MADRL-ZX always achieves the smallest delay, crucial for emergency scenarios requiring low latency.
7. Conclusion
This paper addresses multi-UAV collaborative computing offloading in emergency scenarios. We propose an air-ground integrated architecture with NLOS channel model, user mobility, and load fairness. A novel MADRL algorithm (MADRL-ZX) jointly optimizes offloading decisions, flight angles, and speeds. State normalization and prioritized experience replay enhance convergence. Simulations show that the proposed strategy significantly reduces system cost (~77% over random, ~46% over DDPG) while maintaining high fairness. Future work will extend to three-dimensional motion space for China drone swarms in more complex environments.