The rise of unmanned aerial vehicles (UAVs), or drones, has revolutionized numerous fields, offering unparalleled flexibility, rapid deployment, and the ability to undertake high-risk missions. While single-UAV systems are powerful, multi-UAV systems provide superior robustness, fault tolerance, and mission capability, particularly in complex tasks like search and rescue, area coverage, and, critically, cooperative target capture or pursuit. In such capture missions, a swarm of defender UAVs must collaboratively track, encircle, and neutralize a dynamic intruder drone, demanding sophisticated coordination and real-time decision-making under uncertainty.
Traditional methods for multi-drone coordination often rely on precise mathematical models, such as leader-follower structures or consensus-based control. However, these approaches can struggle with the dynamic, unpredictable nature of real-world environments and adversarial targets. Deep Reinforcement Learning (DRL) has emerged as a promising alternative, enabling drones to learn optimal policies through interaction without explicit environmental models. Multi-agent DRL frameworks, like the Multi-Agent Deep Deterministic Policy Gradient (MADDPG), allow a team of drones to learn coordinated behaviors in a centralized-training-with-decentralized-execution paradigm. Yet, challenges remain: the high dimensionality of joint observations and actions can slow training, and learned policies may occasionally produce unsafe or inefficient maneuvers that violate the fundamental geometric principles of successful encirclement.
To address these limitations, our research team developed a novel integrated approach for multi-UAV capture. We propose the MP-MADDPG-CSF algorithm, which synergistically combines a Max-Pooling enhanced MADDPG network with a Capture Strategy Filter (CSF). The core innovation lies in how we process information and safeguard actions. First, the Max-Pooling mechanism allows each pursuing drone to distill the most critical information from its neighbors, filtering out redundant data and accelerating policy learning. Second, the CSF acts as a safety and performance governor. It sits atop the neural network’s output, analytically correcting any proposed action that would lead the drone out of a mathematically defined “capture set”—the state conditions guaranteeing successful target interception. This ensures that all drones consistently execute maneuvers adhering to proven capture strategies, significantly boosting mission success rates and efficiency.
In this article, we detail the formulation of the multi-drone pursuit problem, present the architecture of our MP-MADDPG-CSF method, and validate its superiority through comprehensive simulations and benchmark comparisons. Our results demonstrate that this method not only learns faster and more stably than conventional MADDPG but also achieves a significantly higher capture success rate and shorter mission completion time.
1. Problem Formulation and System Modeling
1.1 Multi-UAV Pursuit Scenario
We consider a defensive mission where a team of \(N\) autonomous UAVs protects a designated zone. An intruder UAV attempts to penetrate this zone. The defender drones, initially patrolling, must detect, pursue, and cooperatively capture the intruder before it reaches the protected area. Capture is defined as all pursuers maneuvering to satisfy specific geometric conditions relative to the target simultaneously. The scenario involves a single intruder drone (\(T\)) and multiple pursuer drones (\(P_i\)). The key condition for a successful capture, derived from pursuit-evasion theory, involves constraints on the relative velocity and position at the initiation of the pursuit phase (\(t_0\)):
The relative speed must be bounded:
$$ \|\mathbf{v}_{T/P}(t_0)\| \leq \sqrt{u_m (\rho_a + \rho_p) (1 – \frac{\rho_p}{\rho_a})} $$
where \(\mathbf{v}_{T/P}\) is the relative velocity vector, \(u_m\) is the pursuer’s max acceleration, \(\rho_a\) is the intruder’s detection range, and \(\rho_p\) is the pursuer’s identification range.
The pursuer must be within a critical distance band:
$$ R_s \leq \|\mathbf{P}_a – \mathbf{P}_t\| \leq R_c, \quad t_c \leq t $$
Here, \(R_s\) is a safety radius to avoid collision, and \(R_c\) defines the maximum effective pursuit range. \(\mathbf{P}_a\) and \(\mathbf{P}_t\) denote pursuer and target positions, respectively.
To orchestrate the swarm, we implicitly define roles. The first pursuer(s) to effectively engage the target becomes the “primary attacker,” focusing on direct pursuit. The others act as “secondary supporters,” positioning themselves to cut off escape routes and ensure the intruder remains within the collective capture envelope.
1.2 UAV Kinematic Model
We model each drone’s motion in a 2D plane, ignoring pitch and roll dynamics. The kinematic equations are:
$$ \frac{d}{dt}\begin{bmatrix} X \\ Y \\ \psi \\ V \end{bmatrix} = \begin{bmatrix} v \cos \psi \\ v \sin \psi \\ \omega + \eta_\psi \\ a + \eta_v \end{bmatrix} $$
where \((X, Y)\) is the position, \(\psi \in [-\pi, \pi]\) is the heading/yaw angle, \(v\) is the linear speed, and \(\omega\) is the angular velocity (yaw rate). The control inputs are the acceleration \(a\) (affecting \(v\)) and \(\omega\). The terms \(\eta_\psi\) and \(\eta_v\) represent Gaussian random disturbances in the yaw and speed channels, respectively: \(\eta_\psi \sim \mathcal{N}(0, \sigma_\psi^2)\), \(\eta_v \sim \mathcal{N}(0, \sigma_v^2)\). This can be written as an affine nonlinear system:
$$ \dot{\mathbf{x}} = f(\mathbf{x}) + g(\mathbf{x})\mathbf{u}, \quad \mathbf{x} = [X, Y, \psi, V]^T, \quad \mathbf{u} = [\omega, a]^T $$
1.3 Partially Observable Markov Decision Process (POMDP) Model
The multi-drone pursuit is modeled as a POMDP, defined by the tuple \(M = (\mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{O}, \mathcal{Z}, \mathcal{R}, \gamma)\).
- State Space \(\mathcal{S}\): The global state for drone \(i\) includes its own and others’ status, plus the target’s status:
$$ \mathbf{s}^i = [x_p^i, y_p^i, \psi_p^i, v_p^i, x_t, y_t, v_t, \psi_t, \text{neighbors’ states}…] $$ - Action Space \(\mathcal{A}\): Each drone’s action is \(\mathbf{a}^i = [\omega^i, a^i]\).
- Observation Space \(\mathcal{O}\): Each drone receives a partial observation:
$$ \mathbf{o}^i = [\mathbf{o}_s^i, \mathbf{o}_g^i, \mathbf{o}_o^i, \mathbf{o}_n^i] $$
representing its own state, other drones’ states, the intruder’s state, and neighboring drones’ states, subject to sensor noise. - Reward Function \(\mathcal{R}\): We design a composite reward to guide cooperative behavior:
$$ R^i = R_t + R_d + R_a $$- Capture Reward \(R_t\): A large positive reward if the drone is within capture range (\(\|\mathbf{P}_a – \mathbf{P}_t\| \leq R_c\)).
- Distance Reward \(R_d\): A negative reward proportional to the distance to the target, \(-\alpha \|\mathbf{P}_a – \mathbf{P}_t\|\), encouraging approach.
- Collision Avoidance Reward \(R_a\): A large negative penalty if the drone gets too close to the target or other drones (\(\|\mathbf{P}_a – \mathbf{P}_t\| \leq R_s\)).
2. The MP-MADDPG-CSF Capture Method
2.1 Overall Network Architecture
The MP-MADDPG-CSF framework consists of two main stages: a policy learning stage using the MP-MADDPG network, and a policy refinement stage using the CSF. During execution, each drone uses its actor network (part of MP-MADDPG) to generate a nominal action based on its local observations. This action is then passed to the CSF module. The CSF checks if this action would keep the drone within the safe “capture set.” If not, it solves a quadratic program to find the closest feasible action that satisfies the capture strategy constraints. This final, corrected action is then sent to the drone’s controller.
2.2 Max-Pooling MADDPG (MP-MADDPG)
The standard MADDPG algorithm can suffer from inefficiency due to the high-dimensional input of concatenated observations from all agents. Our MP-MADDPG enhances the critic network by incorporating a Max-Pooling operation to process information from neighboring drones more effectively.
Observation Embedding and Max-Pooling: First, the local observation components are passed through separate fully-connected (FC) embedding layers:
$$ \mathbf{e}_s^i = \text{FC}(\mathbf{o}_s^i), \quad \mathbf{e}_g^i = \text{FC}(\mathbf{o}_g^i), \quad \mathbf{e}_o^i = \text{FC}(\mathbf{o}_o^i), \quad \mathbf{e}_n^i = \text{FC}(\mathbf{o}_n^i) $$
The neighbor embeddings \(\mathbf{e}_n^i\) (which could be a set of vectors from multiple neighbors) are then processed by a Max-Pooling layer:
$$ \tilde{\mathbf{e}}_n^i = \text{MaxPool}(\mathbf{e}_n^i) $$
This operation selects the maximum value across each feature dimension from the set of neighbor embeddings, effectively focusing on the most salient information from any neighbor and creating a fixed-size output regardless of the number of neighbors.
Feature Fusion and Network Updates: The processed features are concatenated and fused:
$$ \mathbf{f}^i = \text{FC}(\mathbf{e}_s^i \| \mathbf{e}_g^i \| \mathbf{e}_o^i \| \tilde{\mathbf{e}}_n^i) $$
This fused feature vector \(\mathbf{f}^i\) is used as input to both the actor and critic networks. The critic \(Q^i\) now estimates the value of the joint action, considering the pooled neighbor information: \(Q^i(\mathbf{s}^i, \mathbf{a}^i, \text{MaxPool}(\mathbf{a}_{N_i}))\). The networks are updated using the standard MADDPG policy gradient and temporal-difference loss, but with this richer, pooled input representation leading to more efficient learning.
2.3 Capture Strategy Filter (CSF) Design
The CSF is a safety assurance module based on control barrier function (CBF) principles. Its goal is to ensure the drone’s state remains within a “capture set” \(\mathcal{C}\), defined as states from which successful capture is guaranteed.
Capture Set Definition: We define a capture strategy function \(h(\mathbf{x})\):
$$ h(\mathbf{x}) = R_c^2 – \|\mathbf{p} – \mathbf{p}_t\|^2 – \alpha \cos^2(\theta_v) $$
where \(\mathbf{p}\) and \(\mathbf{p}_t\) are pursuer and target positions, \(R_c\) is the capture radius, \(\theta_v\) is the angular difference between the pursuer’s velocity vector and the ideal pursuit direction, and \(\alpha\) is a weighting factor. The capture set is \(\mathcal{C} = \{\mathbf{x} | h(\mathbf{x}) \geq 0\}\).
Action Correction via Quadratic Programming: After the actor network proposes an action \(\mathbf{u}^i_{\text{RL}}\), the CSF checks if applying this action would keep \(h(\mathbf{x}) \geq 0\) (i.e., the derivative \(\dot{h} \geq -\kappa(h)\) for a class-\(\mathcal{K}\) function \(\kappa\)). If the action is safe, it is passed through. If not, the CSF solves the following optimization in real-time to find the safest, closest admissible action:
$$ \mathbf{u}^{i*} = \arg\min_{\mathbf{u}} \|\mathbf{u} – \mathbf{u}^i_{\text{RL}}\|^2 $$
$$ \text{subject to: } L_f h(\mathbf{x}) + L_g h(\mathbf{x}) \mathbf{u} + \kappa(h(\mathbf{x})) \geq 0 $$
This constraint, derived from the CBF condition, enforces that the chosen action \(\mathbf{u}\) does not cause the drone to leave the capture set \(\mathcal{C}\). The solution \(\mathbf{u}^{i*}\) is the filtered action sent to the drone.
3. Experimental Setup and Evaluation
We conducted extensive simulation experiments to validate the MP-MADDPG-CSF algorithm for multi-UAV capture missions.
3.1 Training Configuration
The training environment is a 200m x 100m rectangular area with a protected circular zone (radius 20m) at the origin. Four defender drones are randomly initialized around the protected zone, and one intruder drone starts in the opposite corner. The intruder employs random evasion strategies. The drones’ speeds are limited to [0, 3] m/s. We train for 50,000 episodes, with 200 steps per episode. Key hyperparameters for the reinforcement learning agents are summarized below:
| Parameter | Value |
|---|---|
| Total Training Episodes | 5×10⁴ |
| Steps per Episode | 200 |
| Actor Learning Rate | 1×10⁻³ |
| Critic Learning Rate | 1×10⁻³ |
| Hidden Layer Dimension | 64 |
| Discount Factor (\(\gamma\)) | 0.95 |
| Replay Buffer Size | 5×10⁵ |
| Batch Size | 512 |
3.2 Evaluation Metrics
We compare our MP-MADDPG-CSF against three baselines: the original MADDPG, our MP-MADDPG (without CSF), and a state-of-the-art method CEL-MADDPG (which uses curriculum learning). The evaluation is based on:
- Average Episode Reward (\(G\)): The mean cumulative reward per episode, indicating overall policy quality.
$$ G = \frac{1}{N_{ep}} \sum_{ep=1}^{N_{ep}} \sum_{t=0}^{T} R_t $$ - Capture Success Rate (\(P_{capture}\)): The percentage of test episodes where all pursuers satisfy the capture conditions (Eq. 1 & 2) within the time limit.
$$ P_{capture} = \frac{\text{Successful Episodes}}{\text{Total Test Episodes}} \times 100\% $$ - Average Task Completion Time (\(T\)): The mean time, in seconds, from the start of pursuit to successful capture in successful episodes.
$$ T = \frac{1}{N_{success}} \sum_{i=1}^{N_{success}} (T_{capture}^i – T_{pursuit}^i) $$
4. Results and Analysis
4.1 Training Performance
The learning curves for the average reward are shown below. The MP-MADDPG-CSF algorithm converges faster and to a higher average reward than the baselines. The Max-Pooling mechanism accelerates learning by reducing irrelevant information, while the CSF ensures the policy learns from and converges to safer, more effective maneuvers. The MP-MADDPG curve shows faster convergence than vanilla MADDPG but is less stable and effective than the full MP-MADDPG-CSF, underscoring the value of the filter.
4.2 Quantitative Testing Results
After training, we performed 1,000 test episodes for each algorithm. The quantitative results are compelling:
| Algorithm | Capture Success Rate (\(P_{capture}\)) | Avg. Completion Time (\(T\)) | Avg. Episode Reward (\(G\)) |
|---|---|---|---|
| MP-MADDPG-CSF (Ours) | 94.20% | 2.01 s | 4.022 |
| CEL-MADDPG | 89.50% | 2.34 s | 3.754 |
| MP-MADDPG | 83.50% | 2.75 s | 3.553 |
| MADDPG | 76.10% | 3.27 s | 3.375 |
Our MP-MADDPG-CSF method achieves a 94.2% success rate, which is a 23.8% relative improvement over the vanilla MADDPG. Furthermore, it completes capture tasks in 2.01 seconds on average, which is 38.5% faster than MADDPG. This demonstrates a significant enhancement in both mission reliability and efficiency. The MP-MADDPG variant shows that the pooling mechanism alone improves performance, but the integration with the CSF filter delivers the most robust and high-performing solution, also outperforming the curriculum-based CEL-MADDPG.
4.3 Qualitative Analysis and Robustness
Trajectory analysis reveals the superior behavior of drones using our method. In a “4-vs-1” scenario, MP-MADDPG-CSF drones quickly coordinate: two act as primary attackers engaging the target directly, while the others position themselves as supporters to block escape paths. The CSF ensures their approach velocities and angles are optimal for capture once in range. In contrast, drones using plain MADDPG or MP-MADDPG exhibit more hesitancy, larger turning arcs, and occasional maneuvers that violate the capture strategy, leading to longer pursuit times or failures. We also tested in a “3-vs-1” scenario (simulating drone loss) and in environments with static obstacles. The MP-MADDPG-CSF algorithm maintained high performance, demonstrating its adaptability and robustness to different swarm sizes and environmental complexities.
4.4 Simulation in Robotic Operating System (ROS)/Gazebo
To validate practical applicability, we implemented the trained policy in a high-fidelity Gazebo simulation with PX4-autopilot-enabled UAV models using ROS. The drones successfully executed the learned cooperative capture behavior in real-time, transitioning from patrol to a coordinated pursuit and final encirclement, confirming the deployability of the MP-MADDPG-CSF policy on realistic drone platforms.
5. Conclusion
In this work, we presented a novel integrated framework, MP-MADDPG-CSF, to address the challenging problem of multi-UAV cooperative target capture. The method synergizes a multi-agent deep reinforcement learning backbone enhanced with a Max-Pooling mechanism for efficient information processing and a Capture Strategy Filter based on control-theoretic principles for safety and performance guarantee. The Max-Pooling operation allows each drone to focus on the most critical information from its neighbors, speeding up training convergence. The CSF acts as a supervisory layer, analytically correcting the neural network’s output to ensure all actions adhere to a mathematically defined capture strategy, thereby drastically increasing mission success rates.
Our extensive simulations demonstrate that the proposed algorithm significantly outperforms standard MADDPG and other advanced variants like CEL-MADDPG. It achieves a notably higher capture success rate (94.2%), a substantially shorter task completion time, and more stable learning. The integration of learning-based flexibility with model-based safety filters provides a powerful paradigm for developing reliable and efficient autonomous multi-drone systems for complex missions like perimeter defense and dynamic target interception. Future work will focus on extending the method to handle multiple intruders, more complex 3D environments, and real-world flight tests with robust communication constraints.