The coordination of unmanned aerial vehicles (UAVs) in China drone swarms plays a pivotal role in modern reconnaissance-strike-battle damage assessment (BDA) missions. However, the high dimensionality of state space, coupled with complex interdependencies among tasks, resource compatibility, and multiple operational constraints, poses significant challenges for real-time, high-quality scheduling. This paper proposes an agile, low-complexity task scheduling algorithm that integrates heterogeneous graph neural networks (HGNN) with proximal policy optimization (PPO) to address these challenges. We first model the China drone swarm scheduling problem as a heterogeneous disjunctive graph (HDG), which encodes heterogeneous entities (tasks and UAVs) and multi-semantic constraints into fixed-dimensional graph embeddings. This representation compresses high-dimensional dynamic states into compact, learnable features. The scheduling decision process is then formulated as a Markov decision process (MDP) under the constraint of a feasible candidate set. By leveraging PPO for end-to-end optimization, our method generates high-quality cooperative task sequences that maximize mission revenue while minimizing travel cost and damage risk. Extensive simulations demonstrate that the proposed HGNN-PPO method achieves superior performance in terms of both scheduling efficiency and comprehensive mission quality compared to traditional genetic algorithms (GA), graph attention network with greedy search (GAT+Greedy), and PPO without graph representation (PPO-NOGNN). The approach scales effectively to large-scale China drone swarm scenarios, achieving a solution time of only 1.89 seconds for 200 UAVs and 30 targets, significantly outperforming GA (16.50 seconds). The results validate the effectiveness and superiority of our method for agile, multi-constraint scheduling in China drone swarms.
Introduction
In modern military and civilian applications, China drone swarms are increasingly deployed to execute complex reconnaissance-strike-BDA missions. The ability to rapidly and adaptively schedule heterogeneous UAVs to perform interdependent tasks under tight constraints is critical for mission success. However, the scheduling problem is inherently a combinatorial optimization task with multiple objectives—maximizing cumulative target value, minimizing total travel distance, and mitigating exposure risks—all while satisfying temporal dependencies, time windows, fuel limits, and resource compatibility. Traditional optimization methods, such as genetic algorithms, often suffer from high computational complexity and poor scalability, especially as the number of UAVs and targets grows. Deep reinforcement learning (DRL) offers a promising alternative by learning end-to-end policies that map states to actions. Nevertheless, naive DRL approaches often fail to capture the structural relationships among tasks and resources, leading to suboptimal decisions. To address these limitations, we propose a novel framework that combines heterogeneous graph neural networks (HGNN) with proximal policy optimization (PPO). By representing the scheduling state as a heterogeneous disjunctive graph, we explicitly model task precedence, resource compatibility, and concurrent execution constraints. The HGNN encoder leverages multi-head attention to aggregate information across diverse node types (operation nodes and UAV nodes), producing rich embeddings that guide the PPO policy. Our method is designed for agile, low-complexity online scheduling of China drone swarms, balancing solution quality with computational efficiency.
The main contributions of this work are as follows:
- We propose a unified scheduling framework for China drone swarms based on a heterogeneous disjunctive graph, enabling event-triggered rolling-horizon decisions that adapt to dynamic environments.
- We formulate the scheduling action as an operation-UAV matching edge and introduce a constraint-aware candidate edge filtering mechanism that reduces the feasible action space while guaranteeing feasibility, thus lowering decision complexity.
- We design an HGNN with multi-head attention for interpretable representation learning of heterogeneous entities and relations, and jointly learn a graph-level value function to enhance policy generalization and training stability.
- We implement a stable training mechanism for PPO under the feasible candidate set constraint, incorporating adaptive learning rate scheduling and dynamic clipping to suppress overly large policy updates, improving sample efficiency and convergence speed.
Problem Formulation
Scenario Description
Consider a China drone swarm consisting of multiple UAVs stationed at several bases. A set of suspected targets is identified, each requiring a sequential chain of operations: reconnaissance (R), strike (S), and battle damage assessment (B). UAVs are heterogeneous in terms of speed, maximum range, and capability type (reconnaissance, strike, assessment). The mission is to assign each operation to a compatible UAV and determine the execution order, such that all constraints are met and an aggregated objective function is optimized. The problem is formulated as a mixed-integer programming model.
Notation and Constraints
Let the set of bases be $$\mathcal{B} = \{B_1, B_2, \ldots, B_{N_B}\}$$, and the set of UAVs be $$\mathcal{R} = \{R_1, R_2, \ldots, R_{N_V}\}$$. Each UAV $$R_k$$ is characterized by its maximum range $$L_k$$, speed $$V_k$$, capability type $$C_k$$, and current availability. The set of targets is $$\mathcal{M} = \{M_1, M_2, \ldots, M_{N_T}\}$$, each with value $$C_{M_i}$$, threat level $$\chi_{M_i}$$, and location $$(X_{M_i}, Y_{M_i})$$. The set of operations is $$\mathcal{O} = \{O_i^j \mid \forall i \in \mathcal{M}, \forall j \in \{R,S,B\}\}$$, with an auxiliary start operation S and end operation E. Each operation $$O_i^j$$ has processing time $$\tau(O_i^j)$$ and a permissible time window $$[t_{\min}(O_i^j), t_{\max}(O_i^j)]$$. The decision variable $$x_{i,k}^j \in \{0,1\}$$ indicates whether operation $$O_i^j$$ is assigned to UAV $$R_k$$.
The key constraints are:
- Fuel constraint: The total distance flown by UAV $$R_k$$ must not exceed its maximum range: $$D_k \leq L_k$$, where $$D_k = \sum_{O_i^j} d_{i,k}^j x_{i,k}^j$$.
- Resource allocation: Each operation must be assigned to exactly one UAV: $$\sum_{R_k} x_{i,k}^j = 1$$. Furthermore, the UAV’s capability must match the operation requirement: $$x_{i,k}^j = 0$$ if $$C_k \neq \text{req}(O_i^j)$$.
- Time windows: The start time $$t_s(O_i^j)$$ of each operation must satisfy $$t_{\min}(O_i^j) \leq t_s(O_i^j) \leq t_{\max}(O_i^j)$$.
- Task precedence: For each target $$M_i$$, the three operations must be executed in order R → S → B: $$t_f(O_i^R) \leq t_s(O_i^S)$$ and $$t_f(O_i^S) \leq t_s(O_i^B)$$.
Objective Function
The overall objective is to maximize the weighted sum of mission revenue minus travel cost and damage risk:
$$ J = \omega_S J_S – \omega_D J_D – \omega_R J_R $$
where:
- Revenue: $$J_S = \sum_{M_i} C_{M_i} g_{M_i}$$, with $$g_{M_i} = \prod_{j} \delta_i^j$$ indicating that all three operations for target $$M_i$$ are scheduled (closed-loop completion).
- Travel cost: $$J_D = \sum_{R_k} \frac{D_k}{L_k}$$.
- Damage risk: $$J_R = \sum_{M_i} \sum_{j} \sum_{R_k} C_{M_i} x_{i,k}^j \rho_{i,k}^j$$, where $$\rho_{i,k}^j = 1 – e^{-\kappa \chi_{M_i} t_{b}(O_i^j)}$$ is the damage probability of UAV $$R_k$$ executing operation $$O_i^j$$, with exposure time $$t_b$$.
Methodology
Heterogeneous Disjunctive Graph Representation
To capture the complex relational structure of the scheduling problem, we embed the state into a heterogeneous disjunctive graph (HDG) $$\mathcal{G}_t = (\mathcal{V}, \mathcal{N}, \mathcal{E}_c, \mathcal{E}_d)$$. The node set $$\mathcal{V}$$ consists of operation nodes (each operation $$O_i^j$$ plus virtual start/end) and UAV nodes (each UAV $$R_k$$). The edges $$\mathcal{E}_c$$ represent task precedence (directed arcs from S to E forming chains for each target), and the edges $$\mathcal{E}_d$$ (undirected) represent candidate assignment edges between operations and UAVs that satisfy compatibility, time windows, and other constraints. At each decision step, the agent selects a candidate edge to make it a fixed assignment, and the graph is updated accordingly (removing conflicting edges). This dynamic graph structure serves as the state representation for the MDP.
The raw features for each node and edge are defined as follows:
- Operation node features (12 dimensions): scheduling status (0/1), number of neighboring UAVs, actual start time, number of unsatisfied predecessors, expected finish time, target location (x,y), target value, required UAV type (one-hot), time window lower bound, time window upper bound, processing duration.
- UAV node features (10 dimensions): capability type (one-hot), maximum range, speed, availability (0/1), number of neighboring operations, base ID, current position (x,y), current task end time, cumulative traveled distance.
- Edge features (2 dimensions): damage probability $$\rho_{i,k}^j$$ and flight distance $$d_{i,k}^j$$.
The HDG representation allows the agent to observe the entire state in a structured, fixed-dimensional manner through graph neural network encoding.
HGNN Architecture for Policy and Value
We design a heterogeneous graph neural network that takes the HDG as input and outputs action probabilities and state value. The architecture consists of three main components: (1) feature transformation, (2) multi-head graph attention encoder, and (3) dual output heads (policy and value).
- Feature transformation: Since operation and UAV nodes have different raw feature dimensions, we apply a type-specific linear transformation to map them into a common hidden space of dimension $$d=11$$: $$\mathbf{x}_{O_i^j} = g_t(\boldsymbol{\mu}_{O_i^j})$$ and $$\mathbf{x}_{R_k} = g_t(\mathbf{v}_{R_k})$$.
- Multi-head graph attention encoder: We stack $$L$$ layers of multi-head graph attention (GAT) layers. In each layer, for a node $$i$$, we compute attention coefficients with its neighbors $$j \in \mathcal{N}(i)$$:
$$e_{ij} = \text{LeakyReLU}(\mathbf{a}^T [\mathbf{W}\mathbf{x}_i \| \mathbf{W}\mathbf{x}_j])$$
$$\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{q \in \mathcal{N}(i)} \exp(e_{iq})}$$
The output of a single head is:
$$\mathbf{y}_i = \text{ELU}\left(\sum_{j \in \mathcal{N}(i)} \alpha_{ij} \mathbf{W}\mathbf{x}_j\right)$$
We use $$K=4$$ independent attention heads and concatenate their outputs: $$\mathbf{h}_i^{(l)} = \|_{k=1}^K \text{ELU}\left(\sum_{j} \alpha_{ij}^{(k)} \mathbf{W}^{(k)}\mathbf{x}_j\right)$$. After stacking $$L$$ layers (we use $$L=3$$), we obtain the final node embeddings. - Node feature fusion: A residual dropout with ReLU is applied to refine the embeddings: $$\tilde{\mathbf{h}}_i = \text{Dropout}(\text{ReLU}(\mathbf{W}_f \mathbf{h}_i^{(L)} + \mathbf{b}_f))$$.
- Graph-level representation: We use global attention pooling to obtain a graph embedding:
$$\beta_i = \text{softmax}(\tanh(\mathbf{q}^T \tilde{\mathbf{h}}_i + b))$$
$$\mathbf{h}_{\text{graph}} = \sum_i \beta_i \tilde{\mathbf{h}}_i$$ - Policy head (action selection): For each candidate assignment edge $$E_{i,k}^j$$, we construct an edge representation:
$$\boldsymbol{\chi}_{i,k}^j = [\tilde{\mathbf{h}}_{O_i^j} \| \tilde{\mathbf{h}}_{R_k} \| \boldsymbol{\lambda}_{i,k}^j]$$
A two-layer MLP with ReLU and dropout outputs a score:
$$f_{\pi}(E_{i,k}^j) = \text{MLP}_{\pi}(\boldsymbol{\chi}_{i,k}^j)$$
The policy over valid actions is obtained by masked softmax:
$$\pi(E_{i,k}^j \mid \mathcal{G}_t) = \frac{\exp(f_{\pi}(E_{i,k}^j)) \cdot m_{i,k}^j}{\sum_{E’ \in \mathcal{E}_t^{\text{valid}}} \exp(f_{\pi}(E’))}$$ - Value head: A two-layer MLP on the graph embedding outputs the state value: $$V_{\phi}(\mathcal{G}_t) = \text{MLP}_v(\mathbf{h}_{\text{graph}})$$.
Markov Decision Process Formulation
The scheduling process is modeled as a finite-horizon MDP $$(\mathcal{S}, \mathcal{A}, P, r, \gamma)$$. State $$s_t = \mathcal{G}_t$$ is the current HDG. Action $$a_t$$ is the selection of a candidate assignment edge. The transition is deterministic: the environment fixes the selected edge, updates UAV availability, times, and removes conflicting edges, leading to $$\mathcal{G}_{t+1}$$. Reward $$r_t$$ is designed to reflect the optimization objective in an incremental form:
$$ r_t = \lambda_p r_t^p + \lambda_b r_t^b – \lambda_d r_t^d – \lambda_r r_t^r $$
where:
- $$r_t^p = \sum_{M_i} \sum_{j} \zeta_i^j \Delta\delta_i^j(t)$$ is the incremental operation revenue ($$\zeta_i^j$$ is the operation value weight).
- $$r_t^b = \sum_{M_i} C_{M_i} \Delta g_{M_i}(t)$$ is the extra reward for completing a closed-loop (all three operations of a target).
- $$r_t^d = \sum_{R_k} \frac{\Delta D_k(t)}{L_k}$$ is the incremental normalized travel cost.
- $$r_t^r = \sum_{M_i} \sum_{j} \sum_{R_k} C_{M_i} \Delta\rho_{i,k}^j(t)$$ is the incremental damage risk penalty.
This design provides dense, informative gradient signals for PPO training.
Training with PPO
We train the HGNN policy and value parameters using the clipped PPO algorithm. The training procedure involves iterative interaction with the environment (simulating China drone swarm scenarios) and parameter updates. Key details include:
- Action mask: At each state, we filter candidate edges using the constraint set (precedence, time windows, fuel, capability) to produce the valid action set. The policy distribution is computed only over this set, preventing illegal actions.
- Advantage estimation: Generalized Advantage Estimation (GAE) with $$\lambda=0.95$$ and discount factor $$\gamma=0.99$$ is employed to reduce variance: $$\hat{A}_t = \sum_{l=0}^{\infty} (\gamma\lambda)^l \delta_{t+l}$$, where $$\delta_t = r_t + \gamma V(s_{t+1}) – V(s_t)$$.
- PPO loss:
$$L^{\text{CLIP}}(\theta) = \mathbb{E}_t \left[ \min\left( r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right]$$
with probability ratio $$r_t(\theta) = \frac{\pi_{\theta}(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$$ and clip range $$\epsilon=0.2$$. - Value loss: $$L^V(\phi) = \mathbb{E}_t \left[ (V_{\phi}(s_t) – \hat{R}_t)^2 \right]$$, where $$\hat{R}_t = \hat{A}_t + V_{\text{old}}(s_t)$$.
- Entropy bonus: $$H(\pi_{\theta}(\cdot|s_t))$$ to encourage exploration.
- Total loss: $$L = L^{\text{CLIP}} – c_v L^V + c_e H$$.
- Adaptive learning rate: We start with a warm-up phase, then decay the learning rate by half if the average reward does not improve for 50 consecutive episodes. Gradient clipping with maximum norm 0.5 is employed to stabilize training.
The training algorithm is summarized in Algorithm 1 (offline training) and Algorithm 2 (online application).
We insert the image of a China drone here to illustrate the practical context of our work:
Simulation Results
Experimental Setup
We evaluate our method on China drone swarm mission scenarios of different scales:
- Small: 2 bases, 12 UAVs, 2 targets.
- Medium: 2 bases, 30 UAVs, 5 targets.
- Large: 10 bases, 200 UAVs, 30 targets.
The UAVs are heterogeneous (reconnaissance, strike, BDA types). Key parameters are summarized in Tables 1–3 (adapted from the original paper).
| ID | Type | Base | Speed | Max Range | Position (x,y) |
|---|---|---|---|---|---|
| 1 | Recon | B1 | 29.76 | 371.73 | (-3.52, -6.98) |
| 2 | Recon | B1 | 27.61 | 361.25 | (-2.69, -8.84) |
| 3 | Assess | B1 | 28.27 | 367.73 | (-1.64, -5.19) |
| 4 | Recon | B2 | 29.41 | 634.31 | (292.48, 144.46) |
| 5 | Recon | B2 | 23.32 | 517.00 | (301.71, 140.99) |
| 6 | Strike | B2 | 21.77 | 442.54 | (295.79, 142.89) |
| 7 | Recon | B2 | 26.85 | 520.40 | (298.20, 152.30) |
| 8 | Strike | B2 | 24.90 | 480.10 | (305.10, 146.70) |
| 9 | Recon | B1 | 30.20 | 410.80 | (-6.10, -3.40) |
| 10 | Assess | B1 | 28.60 | 560.00 | (4.30, -7.90) |
| Base | Position (x,y) |
|---|---|
| B1 | (0, 0) |
| B2 | (300, 150) |
| ID | Position | Value | Threat | R window | R duration | S window | S duration | B window | B duration |
|---|---|---|---|---|---|---|---|---|---|
| 1 | (120,80) | 12 | 0.35 | [10,60] | 8 | [15,70] | 5 | [20,80] | 6 |
| 2 | (520,260) | 18 | 0.75 | [5,50] | 10 | [10,60] | 7 | [15,70] | 8 |
Training Results
We trained the HGNN-PPO model using the adaptive scheduling algorithm. The training curves for different scales are shown in (the figure omitted in text). The average reward steadily increases and converges, demonstrating stable learning. The loss curves show that after approximately 150 episodes, the variance decreases significantly, indicating stable convergence. The learning rate schedule reduces from an initial value when reward plateaus, ensuring both exploration and convergence.
Online Scheduling and Performance Comparison
We compare our HGNN-PPO method against three baselines: (1) Genetic Algorithm (GA), (2) PPO without graph representation (PPO-NOGNN), and (3) GAT with greedy search (GAT+Greedy). To present a fair multi-indicator comparison, we normalize the results for each scale using the min-max normalization:
- For revenue (higher better): $$S_{\text{rev}} = (x – \min) / (\max – \min)$$
- For travel cost and damage risk (lower better): $$S_{\text{cost}} = (\max – x) / (\max – \min)$$
The results are summarized in Table 4.
| Method | Scale | Time (s) | Revenue | Travel Cost | Risk |
|---|---|---|---|---|---|
| HGNN-PPO | 12 | 0.62 | 0.85 | 1.00 | 0.90 |
| 30 | 1.74 | 1.00 | 1.00 | 1.00 | |
| 200 | 1.89 | 1.00 | 1.00 | 1.00 | |
| GA | 12 | 1.85 | 1.00 | 0.73 | 0.00 |
| 30 | 6.90 | 0.62 | 0.80 | 0.00 | |
| 200 | 16.50 | 0.00 | 0.71 | 0.00 | |
| PPO-NOGNN | 12 | 0.71 | 0.20 | 0.00 | 0.40 |
| 30 | 2.10 | 0.00 | 0.00 | 0.29 | |
| 200 | 2.64 | 0.29 | 0.00 | 0.58 | |
| GAT+Greedy | 12 | 0.48 | 0.00 | 0.45 | 1.00 |
| 30 | 1.36 | 0.25 | 0.50 | 0.71 | |
| 200 | 1.47 | 0.43 | 0.43 | 0.74 |
Key observations:
- HGNN-PPO consistently achieves the highest or near-highest normalized scores across all scales and indicators. In the large-scale scenario (200 UAVs, 30 targets), it attains the maximum scores (1.0) for revenue, travel cost, and damage risk, while maintaining a solution time of only 1.89 seconds—significantly faster than GA (16.50 s).
- Without the graph representation, PPO-NOGNN performs poorly. Its inability to capture structural task relationships leads to low revenue and high travel cost, especially in medium and large scales. This underscores the importance of the HDG representation for China drone swarm scheduling.
- GAT+Greedy is fast (1.47 s for large scale) but yields lower multi-indicator quality. Its greedy nature limits the ability to balance rewards and constraints, resulting in normalized scores around 0.4–0.74, far below the 1.0 achieved by HGNN-PPO.
- GA can achieve high revenue in small scales but suffers from high computational cost and poor scalability. In large scales, GA’s revenue score drops to 0.0 (relative worst), and its damage risk score is also 0.0 within the normalized range, indicating it fails to control risk effectively.
Additionally, we performed an online scheduling experiment for a small scenario (12 UAVs, 2 targets) to illustrate the generated schedule. The resulting Gantt chart (not shown here) demonstrates that the China drone swarm completes the reconnaissance-strike-BDA chains within the time windows, respecting all precedence constraints and resource availability. The event-triggered rolling-horizon mechanism successfully adapts to time-window openings.
Conclusion
In this paper, we have proposed an agile, low-complexity task scheduling algorithm for China drone swarms by integrating heterogeneous graph neural networks with proximal policy optimization. The key innovation is the hybrid approach that structures the scheduling state as a heterogeneous disjunctive graph, enabling compact representation of multi-entity and multi-constraint relations, and then learns a policy via PPO under a feasible candidate action set. Extensive simulations across various scales demonstrate that our HGNN-PPO method significantly outperforms traditional GA, graph-free PPO, and GAT-greedy approaches in terms of solution quality (revenue, travel cost, damage risk) and computational efficiency. The method scales gracefully to large fleets of China drone swarms (200 UAVs, 30 targets) with a solution time of under 2 seconds, making it suitable for real-time online scheduling. Future work will focus on incorporating stronger disturbance handling, lightweight online adaptation mechanisms, and extending to multi-objective Pareto optimization.