In the era of rapid advancements in artificial intelligence and unmanned aerial vehicle (UAV) swarm technology, multi-UAV drone confrontation has emerged as a critical research hotspot in military applications. As an extension of UAV swarm technology, intelligent algorithms are employed to control a group of UAV drones in aerial combat scenarios. This paper presents a comprehensive study on the application of an improved multi-agent deep deterministic policy gradient (MADDPG) algorithm integrated with an attention mechanism for collaborative UAV drone countermeasure tasks. The primary objective is to enhance the global situational awareness of each UAV drone through point-to-point information exchange, particularly under the constraint that each drone can only observe its local environment. By dynamically exchanging information with neighboring drones, the proposed method achieves superior cooperative performance.
The core innovation lies in leveraging the attention mechanism to selectively focus on the most relevant information from other UAV drones, thereby improving decision-making efficiency and accuracy. This approach reduces the cognitive load on each UAV drone by filtering out irrelevant data and amplifies the impact of task-critical observations. Extensive experiments are conducted in a custom-built reinforcement learning environment based on a multi-agent particle framework. The red team consists of eight UAV drones controlled by the proposed algorithm, while the blue team employs a rule-based engagement strategy. Results demonstrate that the attention-augmented MADDPG algorithm increases the red team’s win rate by 12 percentage points, from approximately 60% to 72%, and accelerates convergence during training. This work provides a viable pathway for deploying intelligent UAV drone swarms in complex adversarial environments.
1. Introduction
The rapid development of UAV drone technologies has revolutionized modern warfare, enabling the execution of reconnaissance, strike, electronic warfare, and other missions with unprecedented efficiency. Single-UAV systems often suffer from limitations in coverage, redundancy, and adaptability, prompting a shift toward multi-UAV drone swarms. By coordinating multiple drones, tasks can be distributed and executed collaboratively, leading to enhanced operational flexibility. However, the complexity of adversarial scenarios—characterized by high dynamics, partial observability, and intense competition—poses significant challenges to traditional rule-based control methods.
Reinforcement learning (RL), especially deep reinforcement learning (DRL), has emerged as a powerful paradigm for training intelligent agents through interaction with the environment. In multi-agent systems, multi-agent deep reinforcement learning (MADRL) extends RL to handle multiple interacting agents. Among these, the multi-agent deep deterministic policy gradient (MADDPG) algorithm adopts a centralized training with decentralized execution (CTDE) framework, which alleviates the non-stationarity problem common in multi-agent environments. Nevertheless, standard MADDPG still struggles with scalability and efficiency in large-scale UAV drone confrontations, primarily because each agent’s policy relies solely on its local observations, leading to suboptimal coordination.
To address these limitations, we introduce an attention mechanism into the MADDPG framework. The attention mechanism allows each UAV drone to dynamically assign weights to the observations of other drones based on relevance, effectively creating a weighted global state representation. This enables each drone to focus on mission-critical information while ignoring noise, thereby improving collaboration and decision-making. Our contributions are threefold:
- We propose an attention-based MADDPG (ATT-MADDPG) algorithm tailored for multi-UAV drone countermeasure tasks.
- We construct a realistic simulation environment mimicking red-blue UAV drone confrontations, with both sides employing different strategies.
- We conduct extensive experiments, demonstrating that ATT-MADDPG achieves a 12% higher win rate and faster convergence compared to standard MADDPG.
The remainder of this paper is organized as follows. Section 2 reviews related work on MADDPG and attention mechanisms. Section 3 details the proposed algorithm. Section 4 describes the simulation environment and experimental setup. Section 5 presents and analyzes experimental results. Finally, Section 6 concludes the paper and outlines future research directions.
2. Related Work
2.1 Multi-Agent Deep Deterministic Policy Gradient (MADDPG)
MADDPG is an extension of the Deep Deterministic Policy Gradient (DDPG) algorithm for multi-agent settings. It adopts the CTDE paradigm: during execution, each agent uses its own policy network based on local observations; during training, a centralized critic network has access to the joint state and joint actions of all agents. This structure mitigates the non-stationarity caused by other agents’ changing policies. Formally, for an environment with \(N\) agents, the deterministic policy of agent \(i\) is given by:
$$
a_i = \mu_{\theta_i}(o_i)
$$
where \(o_i\) is the local observation of agent \(i\) and \(\theta_i\) represents the policy network parameters. The centralized critic for agent \(i\), denoted \(Q_i(x, a_1, \dots, a_N)\), estimates the expected return using the joint state \(x\) and joint actions. The policy gradient for agent \(i\) is:
$$
\nabla_{\theta_i} J(\theta_i) = \mathbb{E}_{x \sim \mathcal{D}} \left[ \nabla_{\theta_i} \mu_{\theta_i}(o_i) \cdot \nabla_{a_i} Q_i(x, a_1, \dots, a_N) \big|_{a_i = \mu_{\theta_i}(o_i)} \right]
$$
where \(\mathcal{D}\) is the replay buffer. The critic is updated by minimizing the Bellman residual:
$$
\mathcal{L}_i = \mathbb{E}_{x, a_1, \dots, a_N, r_i, x’ \sim \mathcal{D}} \left[ \left( Q_i(x, a_1, \dots, a_N) – y_i \right)^2 \right]
$$
$$
y_i = r_i + \gamma Q_i'(x’, a_1′, \dots, a_N’), \quad a_j’ = \mu_{\theta_j’}(o_j’)
$$
Here, \(r_i\) is the reward, \(\gamma\) is the discount factor, and \(Q_i’\) and \(\mu_{\theta_j’}\) are target networks for stable training.
2.2 Attention Mechanism
The attention mechanism, inspired by human visual attention, allows a model to dynamically weigh the importance of different parts of the input. In multi-agent RL, attention enables each agent to selectively aggregate information from other agents. Given the observations \(\{o_j\}_{j=1}^N\), the attention weight \(\alpha_{ij}\) from agent \(i\) to agent \(j\) is computed as:
$$
\alpha_{ij} = \frac{\exp(\text{score}(o_i, o_j))}{\sum_{k \neq i} \exp(\text{score}(o_i, o_k))}
$$
A common choice for the score function is the dot product: \(\text{score}(o_i, o_j) = o_i^\top W o_j\), where \(W\) is a learnable weight matrix. The aggregated observation for agent \(i\) is then:
$$
\tilde{o}_i = \sum_{j \neq i} \alpha_{ij} o_j
$$
This weighted combination allows agent \(i\) to focus on the most relevant peers for the current task.
3. Proposed Method: Attention-Augmented MADDPG
We augment the standard MADDPG with an attention module that processes the observations of all other UAV drones before feeding them into the policy network. The architecture is shown conceptually in the following steps:
- Each UAV drone \(i\) first obtains its local observation \(o_i\).
- It receives observations from all other drones in its communication range (assumed fully connected for simplicity in this study).
- The attention module computes the weighted aggregated observation \(\tilde{o}_i\) using the formulas above.
- The combined state \([o_i, \tilde{o}_i]\) (or simply \(\tilde{o}_i\) if only aggregated is used) is fed into the policy network to produce action \(a_i\).
- During training, the centralized critic still uses the full joint state and actions.
The pseudocode for the ATT-MADDPG algorithm is provided in Table 1. Note that we use a distance-based attention variant where the attention weight is inversely proportional to the Euclidean distance between UAV drones:
$$
\alpha_{ij} = \frac{d_{ij}^{-1}}{\sum_{k \neq i} d_{ik}^{-1}}
$$
where \(d_{ij}\) is the distance between drone \(i\) and drone \(j\). This simple heuristic works well in practice and avoids additional learnable parameters.
| Step | Description |
|---|---|
| 1 | Initialize \(N\) agents with replay buffer \(\mathcal{D}\), actor networks \(\mu_{\theta_i}\), critic networks \(Q_i\), and target networks. |
| 2 | For episode = 1 to M do: |
| 3 | Reset environment; get initial observations \(o_i\) for all \(i\). |
| 4 | For step = 1 to T do: |
| 5 | For each agent \(i\): |
| 6 | Compute distance to all other agents: \(d_{ij}\). |
| 7 | Compute attention weights: \(\alpha_{ij} = \frac{d_{ij}^{-1}}{\sum_{k \neq i} d_{ik}^{-1}}\). |
| 8 | Compute aggregated observation: \(\tilde{o}_i = \sum_{j \neq i} \alpha_{ij} o_j\). |
| 9 | Select action: \(a_i = \mu_{\theta_i}([o_i, \tilde{o}_i]) + \mathcal{N}_t\) (exploration noise). |
| 10 | Execute all actions; receive rewards \(r_i\) and next observations \(o_i’\). |
| 11 | Compute next aggregated observations \(\tilde{o}_i’\) using same attention. |
| 12 | Store transition \(([o_i, \tilde{o}_i], a_i, r_i, [o_i’, \tilde{o}_i’])\) in \(\mathcal{D}\). |
| 13 | For each agent \(i\): |
| 14 | Sample a minibatch of transitions from \(\mathcal{D}\). |
| 15 | Compute target \(y_i = r_i + \gamma Q_i'([o_i’, \tilde{o}_i’], a_1′, \dots, a_N’)\) where \(a_j’ = \mu_{\theta_j’}([o_j’, \tilde{o}_j’])\). |
| 16 | Update critic by minimizing \(\frac{1}{B}\sum (Q_i([o_i, \tilde{o}_i], a_1, \dots, a_N) – y_i)^2\). |
| 17 | Update actor using policy gradient: \(\nabla_{\theta_i} J \approx \frac{1}{B}\sum \nabla_{a_i} Q_i(\cdots) \nabla_{\theta_i} \mu_{\theta_i}([o_i, \tilde{o}_i])\). |
| 18 | Soft update target networks: \(\theta_i’ \leftarrow \tau \theta_i + (1-\tau)\theta_i’\), \(Q_i’ \leftarrow \tau Q_i + (1-\tau)Q_i’\). |
| 19 | End for |
| 20 | End for |
| 21 | End for |
The attention module introduces minimal computational overhead while significantly improving information flow. By forcing each UAV drone to aggregate data from all others based on proximity, the agents effectively share a compressed global view, enabling better coordinated maneuvers during combat.
4. Simulation Environment
To evaluate the proposed algorithm, we built a custom multi-UAV drone confrontation environment using a multi-agent particle simulator. The environment simulates two opposing teams: red and blue, each composed of eight UAV drones operating in a 2D planar space of size 10,000 m × 10,000 m. The motion dynamics of each drone are governed by:
$$
v_{t+1} = v_t + a_t \Delta t
$$
$$
\theta_{t+1} = \theta_t + \omega_t \Delta t
$$
$$
x_{t+1} = x_t + v_t \cos(\theta_t) \Delta t
$$
$$
y_{t+1} = y_t + v_t \sin(\theta_t) \Delta t
$$
where \(v\) is speed, \(a\) is acceleration, \(\theta\) is heading angle, and \(\omega\) is angular velocity. Each drone is constrained by maximum speed, maximum acceleration, and maximum turn rate. The state space for each drone includes its own position, velocity, heading, as well as the relative positions and velocities of all detected friendly and enemy UAV drones within a limited sensor range (set to 3000 m). The action space consists of two continuous commands: acceleration and turn rate.
The reward function is designed to encourage offensive behavior and survival:
- Killing an enemy UAV drone: +100 reward.
- Being killed by enemy: 0 reward (no additional penalty).
- Exiting the designated boundary or exceeding speed limits: -10 penalty per step.
- Small negative reward to discourage aimless wandering: -0.1 per step.
Both teams start with random positions within their half of the field. The red team uses our trained ATT-MADDPG policy; the blue team uses a simple rule-based strategy: each blue drone selects the nearest red drone as its target and flies towards it at maximum speed, engaging when within weapon range. This provides a challenging but consistent opponent for evaluation.
The simulation is implemented in Python using PyTorch for neural networks. The training parameters are summarized in Table 2.
| Parameter | Value |
|---|---|
| Number of agents (each team) | 8 |
| Replay buffer capacity | 10,000 |
| Batch size | 512 |
| Discount factor (\(\gamma\)) | 0.99 |
| Learning rate (actor & critic) | 0.001 |
| Hidden layer size | 64 |
| Target network update rate (\(\tau\)) | 0.01 |
| Maximum steps per episode | 500 |
| Total training episodes | 20,000 |
| Exploration noise (Ornstein-Uhlenbeck) | \(\sigma=0.1\) |
| Weapon range | 500 m |
5. Experimental Results and Analysis
5.1 Training Performance
We conducted two sets of experiments: (a) red team using standard MADDPG, and (b) red team using ATT-MADDPG (our proposed method). All other conditions remain identical. The win rate over 20,000 training episodes is recorded every 100 episodes. The win rate is defined as the proportion of episodes in which all blue UAV drones are eliminated before the red team is wiped out or the maximum steps are reached.
The convergence curves are shown in the following table, summarizing the win rate at key episode intervals (average over 5 independent runs with different random seeds).
| Episode | MADDPG Win Rate (%) | ATT-MADDPG Win Rate (%) |
|---|---|---|
| 0 | 12.3 | 13.1 |
| 1,000 | 31.5 | 36.8 |
| 2,000 | 42.1 | 56.4 |
| 3,000 | 59.8 | 68.2 |
| 5,000 | 60.3 | 71.5 |
| 10,000 | 60.1 | 72.0 |
| 15,000 | 60.2 | 71.8 |
| 20,000 | 60.0 | 72.1 |
From the data, standard MADDPG converges to approximately 60% win rate after 3,000 episodes, while ATT-MADDPG reaches 72% and converges faster (around 2,000 episodes). The absolute improvement is 12 percentage points. The faster convergence indicates that the attention mechanism fosters more effective learning by reducing the variance in the gradient estimates and providing more informative state representations.
5.2 Qualitative Analysis
We further analyzed the behavior of UAV drones during episodes. Standard MADDPG often results in drones clustering together or following a single leader, making them vulnerable to flanking attacks. In contrast, ATT-MADDPG induces more intelligent formation: drones spread out to cover more area, yet coordinate attacks on isolated blue targets. The attention weights effectively cause each red drone to pay more attention to blue drones that are in danger of being flanked, thereby enabling rapid reinforcement.
One interesting observation is that the distance-based attention works well in this scenario because proximity is a strong indicator of relevance in combat. However, in more complex settings with heterogeneous capabilities, a learned attention mechanism may be necessary.
5.3 Robustness and Scalability
We also tested the trained ATT-MADDPG policy against a slightly perturbed blue team (e.g., varying speed limits). The win rate remained above 68%, indicating reasonable robustness. Furthermore, we attempted to scale the number of UAV drones per side to 16. The ATT-MADDPG algorithm still trains, but with increased computational cost. Preliminary results show a similar improvement trend, though the absolute win rate gap narrows to about 8%. This suggests that the attention mechanism becomes even more critical as swarm size grows, but the current distance-based attention may need refinement for larger networks.
6. Conclusion and Future Work
In this paper, we presented an attention-augmented multi-agent deep reinforcement learning algorithm, ATT-MADDPG, specifically designed for multi-UAV drone countermeasure tasks. By incorporating a simple distance-based attention mechanism into the observation preprocessing step, each UAV drone gains enhanced situational awareness without requiring full communication of raw observations. The experimental results on a simulated red-blue confrontation scenario demonstrate that ATT-MADDPG improves the win rate by 12% compared to standard MADDPG and converges faster. The approach effectively addresses the partial observability challenge inherent in distributed UAV drone operations.
Despite the promising results, several limitations remain. First, the distance-based attention is heuristic and may not be optimal in all scenarios. Future work could explore learnable attention mechanisms that adapt to dynamic threat levels. Second, we assumed an ideal communication channel with no latency or packet loss; real-world deployments must consider communication constraints. Third, the current environment is limited to 2D and simple kinematics. Extending to 3D with more realistic physics and sensor models is needed for practical validation. Finally, the scalability to larger swarms (e.g., 50+ UAV drones) requires further algorithmic improvements, such as hierarchical attention or graph neural network embeddings.
In conclusion, our work confirms that integrating attention mechanisms into multi-agent RL provides a viable pathway for enhancing coordination and performance in UAV drone combat. With continued refinements, these techniques can contribute to the development of autonomous and resilient drone swarms for future military and civilian applications.