The deep integration of Artificial Intelligence and unmanned systems has positioned Unmanned Aerial Vehicle (UAV) swarms, leveraging their high mobility, low cost, and distributed coordination advantages, as a core force in complex operational scenarios such as swarming confrontation, cooperative combat, disaster rescue, and intelligent logistics. The Heterogeneous UAV Cooperative Multi-Task Allocation (HUCTA) problem is a pivotal challenge that directly determines the effectiveness of such swarms. Its core conflict lies in the competitive game between limited heterogeneous UAV drone resources and dynamic multi-task demands under multiple optimization objectives, involving complexities such as platform heterogeneity, multi-objective conflicts, multi-constraints, and strong inter-task coupling. Efficiently solving this problem is crucial for task completion rates, resource utilization efficiency, and adaptability to dynamic environments.
Existing research in HUCTA often relies on traditional optimization algorithms like Genetic Algorithms (GA) and Particle Swarm Optimization (PSO). While these methods can achieve local optimization in static scenarios, they struggle to balance conflicting objectives (e.g., excessive pursuit of the shortest range may compromise timeliness) and exhibit insufficient dynamic adaptability. When tasks change abruptly, these methods are prone to resource misallocation and task interruption, leading to low completion rates and severe resource waste in complex environments. Therefore, how to effectively enhance the competitive allocation of limited resources among multiple tasks and objectives in dynamic heterogeneous environments, thereby improving the robustness, flexibility, and responsiveness of UAV drone swarms, remains a critical technical challenge.
In multi-objective optimization research, classic swarm intelligence algorithms like Bee Colony and Ant Colony algorithms have made some progress. However, their application to such multi-objective optimization problems involving mixed variables and multiple constraints is not yet mature. PSO, due to its fast convergence, has been widely adapted for resource game scenarios. From bi-objective optimization (e.g., focusing on total range and total time) to tri-objective optimization (e.g., balancing range, time, and operational effectiveness), and more general multi-objective models like NSGA, researchers continue to incorporate more comprehensive dimensions of resource competition. Yet, their evaluation processes fail to explicitly model and quantify the inherent competition and synergy among these multiple optimization objectives. Traditional algorithms struggle to comprehend the complex dynamic trade-offs, such as sacrificing a small amount of range for significant time reduction or reducing range for nearby assignments which may increase local time. This lack of understanding of the essence of objective conflicts means the search process operates without cognitive guidance, easily becoming dominated by a single objective and falling into its local optimum.
To address the issue of local optima, researchers have introduced population diversity enhancement strategies. These strategies, while broadening the search scope and increasing the number of non-dominated solutions found, are essentially technical patches based on a single evaluation system (fitness value or Pareto ranking) and search mechanism (particle position update). They lack explicit modeling of the nature of multi-objective conflicts and targeted guidance for promoting synergy or compromise among objectives during the search. Some studies have attempted to introduce game theory to model resource competition, reflecting a recognition of conflict relationships. However, their heuristic-based algorithms still treat the task set as static. When dynamic task insertions cause dramatic shifts in the game landscape among objectives, these algorithms similarly cannot adjust strategies online and incrementally to adapt to new multi-objective balance requirements.
In contrast, Deep Reinforcement Learning (DRL), with its end-to-end dynamic decision-making characteristics, offers a new perspective for solving dynamic HUCTA problems. By modeling task allocation as a Markov Decision Process and enabling continuous interaction between the agent and the environment, DRL can perceive real-time changes in resource states and generate allocation strategies. However, conventional DRL relies on random exploration to generate initial experience in the UAV drone task allocation scenario, leading to frequent invalid resource allocations during early exploration. Invalid exploration not only reduces convergence efficiency but can also cause the algorithm to learn suboptimal local patterns dominated by low-quality experience, trapping it in a poor local equilibrium.
To overcome the limitations of existing research, this work proposes a dynamic allocation framework that integrates Game-theoretic Multi-objective Prior knowledge with Deep Reinforcement Learning (GMP-DRL). The main innovations include:
- Explicit Modeling of Multi-Objective Resource Games: Treating UAV drones and tasks as game participants, a potential game model based on Nash Equilibrium is constructed to characterize the competitive and cooperative relationships among matching degree, range, and time, providing a theoretical basis for finding high-quality equilibrium solutions.
- Prior Knowledge-Guided Efficient Exploration: A hybrid single-objective optimal prior sample strategy is designed—comprising matching-degree-optimal, range-optimal, and time-optimal initialization experience pools. This guides the algorithm to cover key regions of the multi-objective Pareto front from the very beginning of training, avoiding invalid exploration and accelerating convergence for the UAV drone swarm.
- DRL-Based Dynamic Decision-Making: Utilizing DRL to perceive state changes in real-time and dynamically generate allocation schemes, ensuring the scheme’s real-time capability and high efficiency in complex scenarios with high-dimensional coupled constraints for the heterogeneous UAV drone team.
Heterogeneous UAV Cooperative Task Allocation Model
In a heterogeneous multi-UAV drone cooperative task allocation scenario, a swarm needs to collaboratively execute reconnaissance, strike, and assessment tasks on multiple ground targets, with strict dependencies among the three task types. Based on this, the problem can be formulated as: comprehensively considering the attribute characteristics of different tasks, the payload differences and operational capability distinctions of UAV drones, and multiple constraints, assign corresponding task sequences to each UAV drone in the swarm and define the execution order for all tasks, ultimately achieving global optimization of several performance metrics to enhance the swarm’s operational efficiency and task execution effectiveness.
Based on the above description, a mathematical model for task allocation is constructed. Let the total number of UAV drones be $N_U$, with the numbers of reconnaissance UAV drones ($U_O$), strike UAV drones ($U_A$), and assessment UAV drones ($U_E$) being $N_O$, $N_A$, and $N_E$, respectively. The UAV drone set can be represented as $U = {u^O_1 , u^O_2 …u^O_{N_O} , u^A_1, u^A_2…u^A_{N_A} , u^E_1 , u^E_2 …u^E_{N_E}}$. The number of ground targets $T$ is $N_T$, and the target set is $T = {t_1, t_2, t_3…t_{N_T}}$. Each target contains three different types of tasks: reconnaissance ($M_O$), strike ($M_A$), and assessment ($M_E$). The cooperative task set for target $t_k$ is $t_k = {m^O_k , m^A_k, m^E_k}$. The entire task allocation process terminates when all three tasks on all targets are completed.
Detailed symbols involved in the model construction are explained in the table below, while the allocation matrix is defined as:
$$S = [s_{ij}]_{N \times M},\quad s_{ij} = \begin{cases} 1, & \text{task $m_j$ is assigned to UAV drone $u_i$} \\ 0, & \text{otherwise} \end{cases}$$
Here, each pair $(u_i , m_j)$ can be viewed as a player choosing strategy $s_{ij} \in {0,1}$.
| Category | Parameter |
|---|---|
| UAV Drone | UAV drone type set $U \in {U_O, U_E, U_A}$ |
| Type $W_i$ of UAV drone $i$ | |
| Location coordinates $L_i$ of UAV drone $i$ | |
| Strike capability $S_i$ of UAV drone $i$ | |
| Reconnaissance capability $R_i$ of UAV drone $i$ | |
| Assessment capability $A_i$ of UAV drone $i$ | |
| Maximum travel distance $V_i$ of UAV drone $i$ | |
| Maximum ammunition resource $P_i$ for strike tasks carried by UAV drone $i$ | |
| Maximum time resource $C_i$ for reconnaissance/assessment tasks owned by UAV drone $i$ | |
| Total flight range $B_i$ of UAV drone $i$ | |
| Flight speed $Q_i$ of UAV drone $i$ | |
| Task sequence $M_i$ assigned to UAV drone $i$ | |
| Target & Task | Target task set $M \in {M_O, M_E, M_A}$ |
| Type $W_j$ of task $j$ | |
| Location coordinates $L_j$ of task $j$ | |
| Strike demand $S_j$ of task $j$ | |
| Reconnaissance demand $R_j$ of task $j$ | |
| Assessment demand $A_j$ of task $j$ | |
| Ammunition demand $P_j$ of strike task $j$ | |
| Time demand $C_j$ of reconnaissance/assessment task $j$ | |
| Start time $ST_j$ of task $j$ | |
| Completion time $ED_j$ of task $j$ |
To enhance the model’s practicality and alignment with real-world scenarios, considering the attribute characteristics of UAV drones, targets, and tasks, the following five key constraints are defined, focusing on resource limitations and task coordination logic:
- Type Matching Constraint: Reconnaissance tasks are only assigned to reconnaissance UAV drones, strike tasks only to strike UAV drones, and assessment tasks only to assessment UAV drones:
$$u_i \in {U_O}, m_j \in {M_O}; \quad u_i \in {U_A}, m_j \in {M_A}; \quad u_i \in {U_E}, m_j \in {M_E};$$ - Task Sequence Constraint: Each target is executed only once, and each UAV drone is assigned at least one target; the three tasks for a target must strictly follow the reconnaissance-strike-assessment sequence:
$$t_j^{M_O} < t_j^{M_A} < t_j^{M_E}$$
where $t_j^{M_O}$, $t_j^{M_A}$, $t_j^{M_E}$ represent the start execution times for the reconnaissance, strike, and assessment tasks of target $j$, respectively. - Maximum Range Constraint: The total flight distance for the task sequence $Seq_{u_i} = {m_1, m_2…m_j}$ executed by UAV drone $u_i$ must not exceed its maximum range:
$$\sum_{m_k \in Seq_{u_i}, k<j} ,="" \leq="" dist(m_k="" li="" m_{k+1})="" v_i$$ - Ammunition Resource Constraint: The ammunition demand of a strike task must not exceed the remaining ammunition of the strike UAV drone:
$$P_j \leq Rem(P_i)$$ - Time Resource Constraint: The time demand of reconnaissance and assessment tasks must not exceed the UAV drone’s remaining reconnaissance or assessment time, respectively:
$$C_j \leq Rem(C_i)$$
Under the premise of satisfying the above constraints, the optimization of the task allocation scheme requires a comprehensive trade-off among task-UAV drone matching degree, resource consumption, and execution efficiency, achieved through a multi-objective utility function. The objective function consists of the following three parts:
- Task-UAV Drone Matching Degree: To address heterogeneity, it is essential to achieve precise matching between UAV drone capabilities and task demands. High-demand tasks should be assigned to high-capability UAV drones to improve task completion rate:
$$fitness = \begin{cases} \sum min (1, \frac{R_i}{R_j}), & u_i \in U_O , m_j \in M_O \\ \sum min (1, \frac{S_i}{S_j}), & u_i \in U_A, m_j \in M_A \\ \sum min (1, \frac{A_i}{A_j}), & u_i \in U_E , m_j \in M_E \end{cases}$$ - Total Range Cost: Measured by the total flight range of UAV drones to quantify resource consumption, minimized by normalization to the interval $[0,1]$, where $max\_voyage$ is the maximum possible range for that type of UAV drone executing all tasks:
$$voyage = \sum_{u_i \in {U_O,U_A,U_E}} 1 – \frac{B_i}{max\_voyage}$$ - Total Completion Time Cost: Considering load balancing to shorten total task duration, normalized to $[0,1]$, where $max\_time$ is the maximum allowable time and $Q_i$ is the UAV drone’s flight speed:
$$time = \sum_{u_i \in {U_O,U_A,U_E}} 1 – \frac{B_i}{max\_time \cdot Q_i}$$
Heterogeneous UAV Cooperative Task Allocation Algorithm
In the complex scenario of UAV drone swarms performing multi-task cooperative operations, traditional methods exhibit significant limitations in adapting to dynamic environments and balancing multi-objective optimization. Deep Reinforcement Learning, with its neural network-based, end-to-end online learning characteristics, provides a new pathway to address this problem. However, conventional DRL, in the context of heterogeneous UAV drone multi-objective, strongly-constrained scenarios, relies on random exploration to generate experience samples during the initialization phase, lacking prior knowledge of task and UAV drone characteristics, making it highly susceptible to ineffective exploration. Therefore, this chapter focuses on a Deep Reinforcement Learning method based on the fusion of Game-Theoretic prior knowledge.
Nash Equilibrium Prior Sample Initialization Strategy
To mitigate the issues of blind exploration and susceptibility to single-objective local optima in conventional DRL for heterogeneous UAV drone multi-task allocation, this paper leverages the Nash equilibrium properties of potential games to propose a multi-dimensional optimal objective prior sample guidance mechanism. This mechanism is integrated into the DRL initialization phase to significantly improve convergence quality and speed. Compared to GA and PSO, where random initialization of populations can lead to searches being confined to local regions, the hybrid tri-objective prior experience pool designed in this paper can cover key Pareto regions of multi-objective optimization during the initialization phase. This allows the algorithm’s subsequent search to start from multiple high-value regions rather than random, ineffective ones, thereby significantly reducing the risk of falling into a single-objective local optimum. This paper constructs a multi-objective utility function for each player $(u_i , m_j)$ based on game theory:
$$S_M^* = \mathop{argmax}\limits_{S} \sum_{i,j} M_{ij}(S),$$
$$S_L^* = \mathop{argmin}\limits_{S} \sum_{i,j} L_{ij}(S),$$
$$S_T^* = \mathop{argmin}\limits_{S} \sum_{i,j} T_{ij}(S),$$
where $L_{ij}(S)$ is the additional range cost for UAV drone $u_i$ to execute task $m_j$, $T_{ij}(S)$ is the corresponding completion time cost, $M_{ij}(S)$ is the task-UAV drone matching degree score, $\alpha, \beta, \gamma \geq 0$ are objective weights with $\alpha + \beta + \gamma = 1$, and the combined utility is:
$$U_{ij}(S) = -[\alpha L_{ij}(S) + \beta T_{ij}(S)] + \gamma M_{ij}(S)$$
Definition (Global Potential Function):
$$\Phi(S) = \sum_{p=1}^{N} \sum_{q=1}^{M} U_{pq}(S)$$
Lemma: Based on the constructed multi-objective utility function and global potential function, the heterogeneous UAV drone task allocation game is an exact potential game. Furthermore, any pure-strategy Nash Equilibrium of this game is exactly equivalent to a local maximum point of the global potential function $\Phi(S)$. That is:
- If allocation strategy $S^*$ is a pure-strategy Nash Equilibrium of the game, then $S^*$ must be a local maximum point of $\Phi(S)$.
- If allocation strategy $S^*$ is a local maximum point of $\Phi(S)$, then $S^*$ must be a pure-strategy Nash Equilibrium of the game.
Within this framework, the convergence process of the potential game drives $S$ towards a local maximum point, but its quality heavily depends on the initial strategy $S^{(0)}$. Therefore, this paper introduces three types of single-objective prior samples into the DRL initialization phase. By constructing a prior knowledge guidance mechanism across the dimensions of matching degree, range, and time, and designing differentiated UAV drone selection algorithms, efficient initialization of the experience replay buffer is achieved.
| Algorithm 1: Initialize Game Experience Replay Buffer |
|---|
|
Input: UAV drone set $\boldsymbol{U}$, Task set $\boldsymbol{M}$, Exploration rate $\boldsymbol{\epsilon}$ Output: Initialized experience replay buffer $\boldsymbol{D}$ 1. Solve for matching-maximization strategy $\boldsymbol{S_M^*}$ 2. Solve for flight-distance-minimization strategy $\boldsymbol{S_L^*}$ 3. Solve for completion-time-minimization strategy $\boldsymbol{S_T^*}$ 4. Store $\{\boldsymbol{S_M^*}, \boldsymbol{S_L^*}, \boldsymbol{S_T^*}\}$ into experience replay buffer $\boldsymbol{D}$ 5. For episode $\boldsymbol{e} = 1$ to $\boldsymbol{N}$: 6. Initialize task environment and UAV drone states 7. For step $\boldsymbol{t} = 1$ to $\boldsymbol{T}$: 8. If $\boldsymbol{\epsilon > \epsilon_{threshold}}$: 9. Randomly select action $\boldsymbol{a_t}$ from $\{\boldsymbol{S_M^*}, \boldsymbol{S_L^*}, \boldsymbol{S_T^*}\}$ 10. Else: 11. Select $\boldsymbol{a_t = \arg\max Q(s_t, a)}$ 12. Train the Deep Reinforcement Learning algorithm 13. End For 14. End For |
During the initialization phase of DRL, this paper first solves the three single-objective problems for optimal matching degree, optimal range, and optimal time, respectively. The corresponding optimal allocation strategies $S_M^*$, $S_L^*$, $S_T^*$ are loaded into the experience replay buffer. Subsequently, during the high-exploration phase of early training, actions are randomly selected with equal probability from these three prior strategies, ensuring the algorithm perceives multiple Pareto-ascent paths simultaneously. When the exploration rate $\epsilon$ gradually decays below a threshold, it switches to primarily using the optimal action output by the DRL network. Through this hybrid sampling mechanism of prior-driven and network-decision parallelism, the initial strategy set $S^{(0)}$ already covers multiple local high points of the potential function across different dimensions. Subsequent best-response update iterations allow the algorithm to climb along multiple high-value gradients in the potential function space simultaneously and faster, reaching a local maximum point with a high overall potential function value—i.e., a high-quality Nash Equilibrium point for the global potential.
Deep Reinforcement Learning Algorithm
Aiming at the limitations of traditional PSO and GA in dynamic environment adaptability—not only relying on fixed task sets but also struggling to capture the dynamic interplay of constraints such as UAV drone type limitations, remaining resources, and task sequence dependencies—this paper proposes a sequence decision model based on Deep Reinforcement Learning. The core of multi-task allocation for UAV drone swarms is a dynamic Partially Observable Markov Decision Process (POMDP). DRL is employed to achieve an end-to-end mapping from states to actions via a policy network. The deep neural network automatically extracts features related to the resource constraints of heterogeneous UAV drones. Combined with experience replay and target networks to handle the temporal dependencies of dynamic environments, it adapts to complex scenarios with real-time coupling of high-dimensional constraints.
| Algorithm 2: GMP-DRL Task Allocation Algorithm |
|---|
|
Input: UAV drone set $\boldsymbol{U}$, Task set $\boldsymbol{M}$, Number of episodes $\boldsymbol{N}$, Time steps $\boldsymbol{T}$ Output: Optimized UAV-Task assignment policy $\boldsymbol{\pi^*}$ 1. Initialize Q-network with random parameters $\boldsymbol{\theta}$ 2. Initialize target Q-network $\boldsymbol{\theta^- \leftarrow \theta}$ 3. Initialize experience replay buffer $\boldsymbol{D}$ (using Algorithm 1) 4. For episode $\boldsymbol{e} = 1$ to $\boldsymbol{N}$: 5. Initialize simulation environment and task states 6. For step $\boldsymbol{t} = 1$ to $\boldsymbol{T}$: 7. Construct state $\boldsymbol{s_t = \{M_j, U_1, …, U_n\}}$, including task and UAV drone attributes 8. Apply temporal mask to filter invalid actions violating task order: Reconnaissance $\rightarrow$ Strike $\rightarrow$ Assessment 9. Select action $\boldsymbol{a_t}$ via the initialization game experience algorithm (Algorithm 1) 10. Execute action $\boldsymbol{a_t}$, update UAV drone and task states 11. Calculate reward $\boldsymbol{R_t}$ 12. Observe next state $\boldsymbol{s_{t+1}}$ 13. Store transition $(\boldsymbol{s_t, a_t, R_t, s_{t+1}})$ into buffer $\boldsymbol{D}$ 14. Sample mini-batch from $\boldsymbol{D}$ and compute gradient descent loss $\boldsymbol{L_t}$ 15. Update parameters $\boldsymbol{\theta}$ using Stochastic Gradient Descent 16. Periodically update target network: $\boldsymbol{\theta^- \leftarrow \theta}$ 17. End For 18. End For 19. Return the learned optimal task assignment policy $\boldsymbol{\pi^*}$ |
Therefore, the DQN algorithm, which possesses dynamic Markov properties, is selected to address the task reallocation problem. DQN, as an unsupervised learning method, enables an agent to continuously interact with a dynamic environment, take corresponding actions to receive rewards, and learn the environment’s patterns until an optimal policy is obtained. However, its foundational algorithm requires breakthroughs in constraint adaptability and objective synergy for UAV drone scenarios. Key issues to address include how to accurately model heterogeneous resources and task demands in high-dimensional state spaces and how dynamic task flows can drive online policy learning. Accordingly, this paper designs the DRL framework specifically for UAV drone swarm characteristics:
- State Space $S$: The DRL method adopts a centralized architecture. After training, the model is deployed at a ground control station (simulated as a logical decision node in experiments). In real scenarios, UAV drones upload their status via communication links to the control station, which fuses the heterogeneous states of UAV drones and the dynamic states of tasks to form the environmental state, constructing a high-dimensional state vector:
$$S_t = {M_j, U_1, U_2, U_3, … U_n}$$
where $M_j$ and $U_i$ represent the attribute sets of task $m_j$ and UAV drone $u_i$, respectively. - Action Space $A$: A task-type temporal masking mechanism is designed to directly filter out actions that violate the hard constraints of task sequence types (reconnaissance before strike before assessment). This front-loads constraint handling from reward punishment to action generation, reducing invalid exploration:
$$A_t = {1, 2, 3 … n}; \quad \text{subject to: } t_j^{M_O} < t_j^{M_A} < t_j^{M_E}$$
where $A_t$ denotes the action of selecting a UAV drone for a task, and $t_j$ indicates strict adherence to the reconnaissance-strike-assessment sequence. - Reward Function $R$: Different from traditional single rewards, a hierarchical structure comprising a basic reward, constraint penalties, and multi-objective reward is constructed. The basic reward drives task completion, constraint penalties prevent hard constraint violations, and the multi-objective reward adapts to scenario requirements:
$$R_t = r_{base} + r_{penalty} + r_{multi}$$
where $r_{base}$ is the basic task completion reward, $r_{penalty}$ is the penalty for constraint violation, and $r_{multi}$ is the multi-objective reward. - Value Function: The action-value function $Q^\pi(s, a)$ is defined as the expected return under policy $\pi$ after taking action $a$ in state $s$:
$$Q^\pi(s, a) = \mathbb{E}_\pi[G_t | S_t = s, A_t = a]$$
where the return $G_t$ is the discounted sum of future rewards: $G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$, with $\gamma$ as the discount factor. The update follows the Bellman equation, and DQN uses a neural network to approximate the Q-function, minimizing the loss:
$$Loss(\theta) = \mathbb{E} \left[ \left( (R + \gamma \max_{a’} Q(s’, a’; \theta^-)) – Q(s, a; \theta) \right)^2 \right]$$
where $\theta$ are the parameters of the online network and $\theta^-$ are the parameters of the target network, which are periodically synchronized to stabilize training.
Simulation Experiments and Result Analysis
To demonstrate the algorithm’s generalization capability across different task scales, this study constructs diverse mission scenarios to simulate dynamic task allocation environments. Three scale scenarios are established: small (5 UAV drones & 5 targets, 15 tasks), medium (10 UAV drones & 10 targets, 30 tasks), and large (20 UAV drones & 20 targets, 60 tasks). Each scenario includes reconnaissance, strike, and assessment type UAV drones. Each target contains 3 tasks, making the total number of tasks three times the number of UAV drones. Targets are randomly distributed within a 100×100 simulation map. The computing platform has 32GB RAM, an NVIDIA GeForce GTX 1080Ti GPU, and the deep learning framework is implemented using PyTorch.
The state space dimension is dynamically adjusted based on the scenario scale, containing current task features and UAV drone attribute features for the specific scale. The action space dimension matches the number of UAV drones. The action selection strategy employs an $\epsilon$-greedy method, with $\epsilon$ initialized to 1.0 and decayed by a factor of 0.995 per episode until reaching 0.01. The discount factor $\gamma$ is set to 0.99. To alleviate correlation between states, an experience replay buffer with a capacity of 10,000 is constructed to store samples $(s, a, r, s’)$ generated during training, with mini-batches of 64 randomly sampled for parameter updates. The proposed algorithm is trained for 2000 episodes, with the target network parameters synchronized every 10 episodes.
DRL and GMP-DRL Convergence Comparison
The cumulative reward convergence process for DRL and GMP-DRL in the large-scale (20×20) scenario shows significant differences. Regarding convergence speed, DRL relies on random exploration to learn the task allocation policy through trial and error. Facing the curse of dimensionality in the high-dimensional state space, its reward curve oscillates violently in the early stages, requiring approximately 1600 episodes to gradually stabilize. In contrast, GMP-DRL, with its fine-grained task matching and game-prior mechanism, guides the agent to prioritize exploring high-value areas of the action space that satisfy task type matching and range constraints. This effectively reduces无效 exploration, achieving convergence in only about 980 episodes—a 39% improvement in convergence speed over DRL. In terms of post-convergence performance, GMP-DRL’s cumulative reward stabilizes around 0.82, significantly higher than DRL’s 0.75. Moreover, the fluctuation amplitude of GMP-DRL’s reward curve is much smaller than that of DRL, demonstrating stronger stability. In conclusion, in large-scale complex scenarios, GMP-DRL’s advantages in convergence speed and stabilized performance are more pronounced, validating the adaptability and efficiency of its task allocation mechanism for high-dimensional constrained environments.
Algorithm Performance Comparison and Analysis
To verify the superiority of the proposed algorithm, its performance is compared against seven different optimization algorithms: 1) Random Algorithm (RANDOM): assigns tasks completely randomly. 2) Round-Robin Algorithm (RR): assigns tasks in a fixed cyclical order based on type matching. 3) Genetic Algorithm (GA): simulates biological evolution through selection, crossover, and mutation. 4) Particle Swarm Optimization (PSO): treats each solution as a particle updating its position and velocity based on personal and global best experiences. 5) Hybrid Strategy Multi-Objective PSO (HS-MOPSO): incorporates strategies like constrained particle dynamic selection and dominance-based individual selection during initialization. 6) Deep Reinforcement Learning (DRL): combines deep neural networks with Q-learning, modeling task allocation as an MDP. 7) Game-theoretic Multi-objective Prior-knowledge DRL (GMP-DRL): the proposed algorithm. For fairness, core parameters for PSO, MOPSO, and GA are set uniformly: maximum iterations=100, population size=50, learning factors $C_1=C_2=0.45$, mutation probability=0.8.
The comparison analyzes multiple metrics: task-resource matching degree, total UAV drone range, total task completion time, and task completion rate. In the medium (10×10) scenario, GMP-DRL achieves a task-resource matching degree of 0.95, the shortest total range (8.7% shorter than PSO, 18.1% shorter than RR). Its total time is significantly lower than RANDOM and PSO. Although slightly higher than MOPSO’s time, GMP-DRL balances high matching degree and constraint satisfaction. Its key advantage is achieving a 100% task completion rate while ensuring high matching度. It is the only algorithm that satisfies all constraints. In contrast, traditional algorithms exhibit clear limitations. RANDOM and RR lack intelligent optimization. RR achieves the highest matching度 due to its strict type-matching mechanism but does not consider fine-grained matching of capability levels. MOPSO has the shortest total time, benefiting from its population diversity strategies in multi-objective optimization, but at the cost of a low completion rate of only 0.77. The total range for PSO and MOPSO is similar but higher than GMP-DRL, indicating their static planning struggles to adapt path optimization during dynamic task execution and balance multiple objectives with dynamic constraints. The total completion time for DRL and GMP-DRL is similar, but there is a significant difference in total range; DRL is 8.4% higher than GMP-DRL. DRL, lacking prior knowledge of multi-objective game, finds it difficult to balance time efficiency and range optimization, whereas GMP-DRL, by integrating game-theoretic priors, significantly reduces resource consumption without sacrificing temporal performance.
In summary, through the fusion of game-prior guidance and DRL dynamic decision-making, GMP-DRL achieves a globally optimal balance among resource matching, energy consumption, efficiency, and constraint satisfaction in medium-scale scenarios. To further explain the root causes of the aforementioned performance differences, the following analysis delves into allocation result details and the range-time trade-off.
PSO and GMP-DRL Allocation Result Comparison
Corresponding to the task completion rate differences discussed earlier, this section compares the details of the allocation sequences. The task execution sequences reveal significant differences between MOPSO and GMP-DRL in UAV drone task assignment strategy. MOPSO, lacking a precise matching mechanism between UAV drone types and task demands, exhibits type mismatches, such as a reconnaissance UAV drone being assigned a strike task. For instance, the sequence for a reconnaissance UAV drone $u^O_2$ includes a strike task $m^A_8$, and a strike UAV drone $u^A_1$ is assigned an assessment task $m^E_7$ before its strike task $m^A_{10}$. For target $t_7$, the assessment task is scheduled before the reconnaissance and strike tasks, causing some strike and assessment tasks to be unable to proceed due to missing prerequisites. Consequently, MOPSO achieves only a 77% completion rate. In contrast, GMP-DRL’s task sequences strictly adhere to temporal logic and type matching. For example, $u^O_1$ executes $m^O_2 \rightarrow m^O_9 \rightarrow m^O_{10}$, building a complete dependency chain for subsequent strike UAV drone $u^A_1$ executing $m^A_6 \rightarrow m^A_9 \rightarrow m^A_{10}$ and assessment UAV drone $u^E_1$ executing $m^E_2 \rightarrow m^E_4 \rightarrow m^E_6 \rightarrow m^E_{10}$. GMP-DRL achieves a 100% task completion rate. Its task sequences ensure precise matching between heterogeneous UAV drone capabilities and task demands while resolving resource conflicts and temporal contradictions, highlighting its capability for global optimization in complex multi-task cooperative scenarios.
Algorithm Range and Time Trade-off Comparison
Regarding the range and time metrics discussed earlier, this section further analyzes the algorithms’ balance. A scatter plot showing the distribution relationship between total UAV drone range and total task completion time for each algorithm reveals distinct patterns. RANDOM and RR points are scattered randomly across the space, showing no aggregation trend or movement towards efficient regions, as neither considers range or time optimization. GA and PSO points show a tendency to cluster in specific areas due to their algorithmic mechanisms but remain somewhat离散 due to local optima influence. MOPSO, introducing strategies to optimize population convergence and diversity, somewhat alleviates PSO’s local optimum issue, and its points distribute more towards the efficient region. However, its 77% task completion rate indicates insufficient adaptability in complex task scenarios. In-depth analysis shows that while MOPSO pursues the optimization objective of minimizing total completion time, it does not fully consider constraint conditions during task execution, leading to some tasks being interrupted or delayed due to unmet constraints. This phenomenon indicates that MOPSO still has room for improvement in the balance and stability of multi-objective optimization. In contrast, DRL and GMP-DRL algorithms show outstanding performance in balancing range and task completion time. GMP-DRL, in particular, has a lower completion time than DRL, with most points concentrated in regions with relatively short range and reasonably low task completion time. By combining deep reinforcement learning with task dynamic matching strategies, and enabling the agent to learn effective task allocation policies, GMP-DRL better adapts task characteristics to UAV drone resources, effectively controlling UAV drone range while ensuring task completion efficiency, achieving a superior balance between range and time.
Furthermore, regarding task allocation time consumption, GMP-DRL demonstrates excellent performance across all three scenario scales, with its average time consumption significantly lower than that of GA, PSO, and DRL algorithms. Although slightly higher than RANDOM and RR, the差距 is not substantial. The efficiency of GMP-DRL primarily benefits from the end-to-end decision-making mechanism of deep reinforcement learning, which can directly output allocation results based on environmental states, thereby avoiding the substantial computational overhead generated by the iterative search processes of traditional optimization algorithms. Simultaneously, GMP-DRL can quickly adapt to changes in tasks and resources in dynamic environments, highlighting its advantages in real-time response and robustness, providing a more practical solution for complex task allocation scenarios.
Algorithm Comparison in Small and Large Scenarios
In the small (5×5) scenario, GMP-DRL shows significant comprehensive performance advantages: task-resource matching degree reaches 0.84, total range is 1067.47 (slightly higher than PSO but with a 61.5% improvement in matching degree), total time is 905.82, and task completion rate is 0.87, outperforming RANDOM, PSO, and other algorithms. RR and GA achieve a matching degree of 0.97, but their optimization of total time and range is insufficient. MOPSO has the shortest time but is less practical due to its low matching degree and completion rate. In the large (20×20) scenario, GMP-DRL’s matching degree improves to 0.92, total range of 4018.72 is optimal (2.5% shorter than PSO), and total time is 3461.80, balancing MOPSO’s efficiency with high matching demand. Although its total range is slightly higher than DRL’s, its total completion time is significantly lower than DRL’s, and it achieves the highest task completion rate of 92%. GMP-DRL efficiently generates multiple high-quality allocation solutions by integrating game-theoretic priors, maintaining stable task execution capability even in complex environments with intertwined constraints.
In conclusion, through the fusion of game-prior guidance and deep reinforcement learning, GMP-DRL achieves balanced optimization of matching degree, range, time, and completion rate across small, medium, and large-scale scenarios, with extremely low allocation time consumption. It is better suited to the practical requirements of dynamic task allocation for heterogeneous UAV drones compared to traditional algorithms.
Conclusion
This paper addresses the heterogeneous multi-UAV drone multi-objective allocation problem by proposing the GMP-DRL method, which integrates game-theoretic priors with deep reinforcement learning. It effectively overcomes the limitations of traditional algorithms regarding local optima, insufficient dynamic adaptability, and multi-objective balancing. By constructing an optimization model that combines task-UAV drone resource matching degree, range cost, and time cost, and integrating a hybrid single-objective optimal prior sample initialization strategy with a deep reinforcement learning mechanism, the proposed method demonstrates excellent performance across three heterogeneous scenarios: small, medium, and large. Particularly in the medium-scale scenario, it achieves a 100% task completion rate and is the only algorithm that satisfies all constraints, outperforming traditional algorithms while maintaining stability under complex constraints.
The experimental results fully validate the effectiveness and superiority of the proposed method in multi-objective balancing and dynamic adaptability. Although this research has achieved阶段性 results, the ability to handle dynamically emerging tasks has not been fully verified, and the UAV drone model does not account for equipment failures and communication delays in extreme environments. Future research could further extend to real-time response mechanisms for dynamic task emergence scenarios, enhancing the ability to handle extreme situations such as task priority changes and sudden UAV drone failures. Exploring extended applications for multi-swarm cooperative allocation could provide more universal solutions for the intelligent collaboration of heterogeneous UAV drone systems.