The rapid advancement of Unmanned Aerial Vehicle (UAV) technology has significantly expanded their operational scope. UAV drones offer distinct advantages in performing missions that are dull, dirty, or dangerous for human operators. However, the capabilities of a single UAV drone are inherently limited when confronting complex, large-scale, or multi-faceted missions. To address this, the concept of UAV swarms—multiple UAV drones operating in a coordinated manner—has gained considerable traction. Efficient task allocation within such heterogeneous UAV swarms, where drones possess varied capabilities, is paramount to maximizing mission success, optimizing resource utilization, and ensuring the robustness of the entire system. This problem becomes exponentially more challenging under real-world conditions characterized by environmental uncertainties such as variable wind and precipitation, which directly impact UAV performance.
Traditional approaches to multi-UAV drone task allocation can be broadly categorized into centralized and distributed methods. Centralized algorithms, such as Genetic Algorithms (GA) and Particle Swarm Optimization (PSO), leverage global information to find optimal or near-optimal solutions. While effective for moderate-scale problems in controlled settings, their computational complexity often becomes a bottleneck for large, dynamic UAV swarms, and they present a single point of failure. Distributed methods, including auction-based mechanisms and coalition formation games, enhance scalability and robustness by enabling UAV drones to make local decisions. Nevertheless, these methods often rely on predefined models or heuristic rules that may lack the adaptability to handle unforeseen environmental disturbances or complex, evolving mission constraints effectively.
Reinforcement Learning (RL), and particularly Deep Reinforcement Learning (DRL), offers a promising paradigm shift. Instead of relying on explicit programming or rigid rules, a DRL agent learns optimal decision-making strategies through continuous interaction with its environment. This makes it exceptionally well-suited for dynamic and uncertain scenarios typical of UAV drone operations. By framing the task allocation problem as a Markov Decision Process (MDP), the system can learn policies that are both flexible and high-performing. The Deep Q-Network (DQN) algorithm marked a breakthrough by successfully combining Q-learning with deep neural networks, enabling RL to solve problems with high-dimensional state spaces. Subsequent innovations like Double DQN (addressing Q-value overestimation), Dueling DQN (separating state value and action advantage), and Prioritized Experience Replay (improving sample efficiency) have further advanced the field.
In this work, we focus on the critical challenge of fast and efficient task allocation for a heterogeneous UAV drone swarm operating under environmental uncertainty. We propose a novel algorithm named Double Dueling DQN with Prioritized Experience Replay (D3QPER). Our approach integrates the strengths of Double DQN and Dueling DQN within a stable training framework, enhanced by prioritized sampling of past experiences. This integration aims to accelerate learning, improve policy robustness, and deliver rapid allocation decisions. The primary contributions of this work are as follows:
1. We formulate the heterogeneous UAV swarm task allocation problem under environmental uncertainty as a Markov Decision Process. The model incorporates key constraints including UAV capability matching, safety requirements, and a penalty for resource over-provisioning.
2. We design the D3QPER algorithm, which synergistically combines a dueling network architecture to reduce action-selection noise, a double Q-learning mechanism to mitigate value overestimation, and a prioritized experience replay buffer to focus learning on the most informative experiences.
3. Through extensive simulations, we demonstrate that our D3QPER algorithm achieves superior convergence speed and stability compared to baseline DRL methods (DQN, Double DQN, Dueling DQN). It efficiently generates valid task allocation schemes for UAV drones with minimal resource waste and exhibits strong scalability as the number of tasks increases.
1. Problem Formulation and MDP Modeling
We consider a scenario where a ground control station commands a heterogeneous swarm of UAV drones. The swarm comprises N distinct types of UAVs, with a sufficient number of drones available for each type. A set of M independent tasks, $\mathcal{T} = \{T_1, T_2, …, T_M\}$, arrives simultaneously. Each task $T_m$ has specific requirements defined by a triple: total required attack capability $\text{att}(T_m)$, total required detection capability $\text{det}(T_m)$, and a defensive threshold $\text{def}(T_m)$ that a single UAV drone assigned to this task must meet for its own safety. Each UAV drone of type j possesses inherent capabilities: attack power $\text{att}(U_j)$, defense level $\text{def}(U_j)$, and detection power $\text{det}(U_j)$. A key factor is that the actual efficiency $e_j^m$ of a type-j UAV drone when executing task $m$ is not constant; it is stochastically affected by the local environmental conditions, namely wind speed $w_m$ and rainfall $r_m$ at the task location. The core decision is to determine, for each task, the composition of the UAV drone team—specifically, the number $a_j^m$ of type-j drones to allocate—such that the task is completed successfully with minimal resource expenditure.
1.1 Constraints and Objective Function
Any valid allocation for task $m$ must satisfy the following constraints. First, the collective capabilities of the assigned UAV drone team must meet or exceed the task’s demands, adjusted for their environmental efficiency:
$$ \sum_{j=1}^{N} a_j^m \cdot e_j^m \cdot \text{det}(U_j) > \text{det}(T_m) $$
$$ \sum_{j=1}^{N} a_j^m \cdot e_j^m \cdot \text{att}(U_j) > \text{att}(T_m) $$
Second, for the safety of every individual UAV drone, its defensive capability must be sufficient to withstand the task’s threat level:
$$ e_j^m \cdot \text{def}(U_j) > \text{def}(T_m) \quad \text{for any } j \text{ where } a_j^m > 0. $$
Merely satisfying these constraints can lead to wasteful overallocation of UAV drones. Therefore, our objective is not just feasibility but optimality, minimizing the “resource waste.” We define a performance index $J_m$ for task $m$ as the sum of squared excesses in attack and detection capabilities:
$$ J_m = \left( \sum_{j=1}^{N} a_j^m \cdot e_j^m \cdot \text{att}(U_j) – \text{att}(T_m) \right)^2 + \left( \sum_{j=1}^{N} a_j^m \cdot e_j^m \cdot \text{det}(U_j) – \text{det}(T_m) \right)^2 $$
A successful and efficient allocation is one where $J_m$ is below a predefined threshold $\text{Th}$. The optimization problem is to find the allocation matrix $[a_j^m]$ that minimizes this waste across tasks, subject to the constraints above.
1.2 Markov Decision Process Formulation
To solve this using reinforcement learning, we model the sequential decision-making process for a single task as a Markov Decision Process (MDP). The agent’s goal is to learn a policy for selecting UAV drone types to build a team.
State Space $\mathcal{S}$: The state $s$ for a task is a vector encapsulating all necessary information for decision-making:
$$ s = \left( \text{att}(T_m), \text{det}(T_m), \text{def}(T_m), w_m, r_m \right) $$
This includes the task’s intrinsic requirements and the environmental conditions affecting UAV drone efficiency.
Action Space $\mathcal{A}$: The action is the selection of a UAV drone type to add to the current team for the task. With N drone types, $\mathcal{A} = \{U_1, U_2, …, U_N\}$.
Reward Function $\mathcal{R}$: The reward provides immediate feedback on the quality of an action. We design a shaped reward function:
$$ R(s, a) =
\begin{cases}
-15, & \text{if chosen UAV drone violates safety constraint } (e_j^m \cdot \text{def}(U_j) \leq \text{def}(T_m)) \\
-1, & \text{if action is valid but task is not yet complete} \\
+15, & \text{if task is completed successfully AND } J_m < \text{Th} \\
0, & \text{if task is completed but } J_m \geq \text{Th (inefficient allocation)}
\end{cases} $$
This function strongly penalizes dangerous assignments, provides a small negative step cost to encourage efficiency, gives a high positive reward for successful and efficient completion, and zero reward for successful but wasteful completion.
State Transition Probability $\mathcal{P}_{ss’}^a$: The stochastic efficiency $e_j^m$ of a UAV drone under environmental conditions $(w_m, r_m)$ defines the transition dynamics. When a drone of type $j$ is added, the remaining required capabilities for the task are reduced by $e_j^m \cdot \text{att}(U_j)$ and $e_j^m \cdot \text{det}(U_j)$. The environmental conditions $w_m$ and $r_m$ remain part of the state until the task is closed.
Discount Factor $\gamma$: We use $\gamma = 0.9$ to balance the importance of immediate versus future rewards.
2. The D3QPER Algorithm for UAV Task Allocation
The core of our solution is the Double Dueling DQN with Prioritized Experience Replay (D3QPER) algorithm. It integrates several advanced DRL techniques to create a stable and efficient learner for the UAV drone allocation problem.
2.1 Algorithmic Components
Dueling Network Architecture: A standard DQN uses a single stream of fully-connected layers to output Q-values for all actions. A Dueling DQN splits the network into two separate streams: one estimates the state-value function $V(s; \theta)$, representing how good it is to be in state $s$, and the other estimates the action-advantage function $A(s, a; \theta)$, representing how much better taking action $a$ is compared to the average action in that state. These are then combined to produce the Q-value:
$$ Q(s, a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \left( A(s, a; \theta, \alpha) – \frac{1}{|\mathcal{A}|} \sum_{a’} A(s, a’; \theta, \alpha) \right) $$
where $\theta$ denotes the common network parameters, and $\alpha$, $\beta$ are the parameters of the advantage and value streams, respectively. This separation allows the network to learn which states are valuable without having to learn the effect of each action in every state, leading to more stable and efficient policy learning, which is crucial for the diverse state space encountered in UAV drone allocation.
Double Q-Learning: Standard DQN is known to suffer from overoptimistic Q-value estimates due to the max operation in the target: $y = r + \gamma \max_{a’} Q(s’, a’; \theta^-)$. Double DQN decouples the action selection from the action evaluation. It uses the online network $\theta$ to select the best action for the next state, but uses the target network $\theta^-$ to evaluate its Q-value:
$$ y_{\text{Double}} = r + \gamma Q\left(s’, \arg\max_{a’} Q(s’, a’; \theta); \theta^-\right) $$
This simple change significantly reduces overestimation bias, leading to more stable training and often better final policies for the UAV drone agent.
Prioritized Experience Replay (PER): Not all experiences in the replay buffer are equally valuable for learning. PER assigns a priority $p_i$ to each transition $(s_i, a_i, r_i, s_{i+1})$, typically based on the magnitude of its Temporal-Difference (TD) error $\delta_i = |y_i – Q(s_i, a_i; \theta)|$. Transitions with larger TD error are more surprising to the current network and thus have more learning potential. The sampling probability for transition $i$ is $P(i) = p_i^\alpha / \sum_k p_k^\alpha$, where $\alpha$ controls the prioritization strength ($\alpha=0$ uniform sampling). To correct for the bias introduced by non-uniform sampling, importance-sampling weights $w_i = (N \cdot P(i))^{-\beta}$ are applied to the updates, which are annealed over time ($\beta$ increases to 1). This mechanism allows our DRL agent for UAV drones to learn more effectively from critical or surprising experiences.
2.2 Integrated D3QPER Framework
Our D3QPER algorithm synthesizes these components. It employs two neural networks: an online network and a target network, both with the dueling architecture. The online network parameters $\theta$ are updated at every step, while the target network parameters $\theta^-$ are periodically synchronized with $\theta$. The experience replay buffer stores transitions with their associated TD error priorities.
The training step proceeds as follows: A batch of transitions is sampled from the buffer according to priorities. For each transition, the target $y$ is computed using the Double Q-learning rule combined with the target dueling network:
$$ y = r + \gamma Q\left(s’, \arg\max_{a’} Q(s’, a’; \theta); \theta^-, \alpha^-, \beta^-\right) $$
The loss for the batch is the mean squared error between $y$ and the current online Q-value, weighted by the importance-sampling weights $w_i$:
$$ \mathcal{L}(\theta) = \frac{1}{B} \sum_{i=1}^{B} w_i \left[ y_i – Q(s_i, a_i; \theta, \alpha, \beta) \right]^2 $$
This loss is then used to update the parameters $\theta$ of the online network via gradient descent. The priorities of the sampled transitions in the buffer are subsequently updated with their new TD errors.
By combining these techniques, D3QPER aims to provide a learning agent that is sample-efficient (thanks to PER), stable and accurate in its value estimates (thanks to Double DQN), and capable of robust policy learning in complex state spaces (thanks to the Dueling architecture). This makes it particularly suitable for learning optimal task allocation strategies for heterogeneous UAV drone swarms.
3. Simulation Experiments and Results
We conducted extensive simulation experiments to evaluate the performance of the proposed D3QPER algorithm for UAV drone task allocation.
3.1 Experimental Setup and Parameters
We consider a heterogeneous UAV drone swarm with N=5 distinct drone types. Their inherent capabilities are summarized in Table 1.
| UAV Type (j) | Attack $\text{att}(U_j)$ | Defense $\text{def}(U_j)$ | Detection $\text{det}(U_j)$ |
|---|---|---|---|
| Type 1 | 25 | 30 | 30 |
| Type 2 | 40 | 35 | 10 |
| Type 3 | 15 | 20 | 35 |
| Type 4 | 45 | 25 | 5 |
| Type 5 | 20 | 15 | 35 |
The environmental efficiency $e_j^m$ for each UAV drone type under different wind ($w_m$) and rain ($r_m$) levels (scaled 1-5, with 5 being most severe) was defined based on realistic performance degradation models. For instance, Type 1 drones have an efficiency of 0.9 in mild conditions but drop to 0.85 in severe weather ($w_m$ or $r_m$ = 5). Type 4 and 5 drones are more susceptible, with efficiencies as low as 0.7 in the worst conditions. This stochastic efficiency is modeled as a normal distribution around these nominal values.
The DRL models were implemented using TensorFlow. Key hyperparameters for the D3QPER algorithm are listed in Table 2.
| Hyperparameter | Value | Hyperparameter | Value |
|---|---|---|---|
| Training Episodes | 2500 | Discount Factor ($\gamma$) | 0.9 |
| Replay Buffer Size | 10000 | PER Exponent ($\alpha$) | 0.6 |
| Target Update Freq. | 500 steps | Learning Rate | 0.005 |
| Batch Size | 32 | Exploration $\epsilon$ (initial/final) | 1.0 / 0.01 |
3.2 Performance Analysis
Allocation Effectiveness: After training, the D3QPER policy was tested on 1600 random task sets. Each task had randomly generated requirements and environmental conditions. The algorithm successfully generated valid allocations for all tasks. A sample of allocations for 10 different tasks demonstrates the policy’s adaptability. The composition of the UAV drone team varies significantly based on the specific needs of each task. For example, a task with high detection needs but low defense threat might receive more Type 3 and Type 5 UAV drones, while a high-threat, high-attack task might be assigned more Type 2 and Type 4 drones. The learned policy effectively minimizes resource waste ($J_m < \text{Th}$), with the cumulative reward per 10-task episode consistently stabilizing between 60 and 70, indicating reliable and efficient performance.
Scalability: To test scalability, we evaluated the algorithm on varying numbers of simultaneous tasks: 10, 20, 40, and 50. The results show that the average reward per task remains stable and high across these different scales. This demonstrates that the learned policy generalizes well and the algorithm’s decision-making time per task does not degrade significantly with problem size, a crucial feature for real-world deployment of large UAV drone swarms.
Computational Efficiency: A key objective is fast task allocation. We compared the total computation time required by D3QPER, standard DQN, and Double DQN to generate allocation plans for 100 tasks. The results are conclusive: D3QPER is markedly faster. On average, our D3QPER algorithm computes a suitable UAV drone allocation for a single task in approximately 0.24 seconds. This high speed is attributable to the policy being encapsulated in a neural network, which performs a simple forward pass to make decisions, in contrast to the iterative computation required by traditional optimization algorithms.
Impact of Prioritized Experience Replay: We ablated the PER component to isolate its contribution. Training curves clearly show that D3QPER (with PER) achieves convergence significantly faster than its variant with uniform experience replay. The PER-equipped version learns from surprising or high-error transitions more frequently, accelerating the policy improvement process for the UAV drone allocation agent. The reward curve is also smoother, indicating more stable learning.
Convergence Comparison with Baseline Algorithms: We compared the training performance of D3QPER against three strong baselines: Vanilla DQN, Double DQN, and Dueling DQN (with uniform replay). The plot of average reward per episode against training steps reveals clear differences. Vanilla DQN shows high initial variance and slower convergence due to overestimation and less efficient learning. Both Double DQN and Dueling DQN show improved stability and performance. However, our integrated D3QPER algorithm consistently outperforms all of them. It reaches a higher level of performance more quickly (converging around 850 episodes) and maintains a stable, high reward with minimal variance. This validates our hypothesis that the synergistic combination of double Q-learning, dueling architecture, and prioritized replay creates a superior learner for the complex heterogeneous UAV drone task allocation problem.
4. Conclusion
In this work, we addressed the critical challenge of fast and efficient task allocation for heterogeneous UAV drone swarms operating under realistic environmental uncertainties. We formulated the problem as a Markov Decision Process, incorporating key constraints related to UAV capability matching, operational safety, and resource optimization. Our main contribution is the design and implementation of the D3QPER algorithm, a deep reinforcement learning approach that innovatively integrates a Double DQN framework, a Dueling network architecture, and a Prioritized Experience Replay mechanism.
Extensive simulation results demonstrate the efficacy of our approach. The D3QPER algorithm learns a robust policy that can rapidly generate effective UAV drone team compositions for diverse tasks. It achieves significantly faster convergence and greater stability during training compared to standard DRL baselines. The algorithm exhibits strong scalability, handling an increasing number of tasks without performance degradation. Most importantly, it executes the allocation decision with remarkable speed, averaging only 0.24 seconds per task, which is essential for time-sensitive operations involving UAV drone swarms.
Future work will focus on extending this framework to fully distributed settings where UAV drones themselves make cooperative allocation decisions with limited communication, and on incorporating more dynamic aspects such as task arrival during execution and mid-mission UAV drone attrition.