The rapid convergence of artificial intelligence and unmanned aerial vehicle (UAV) technologies has ushered in a new era of aerial threats. Drone swarms, leveraging their collective scale and coordinated operational capabilities, present a formidable challenge to traditional air defense architectures. Among the various countermeasures—including electronic warfare, laser interception, and kinetic engagement—anti-aircraft artillery systems remain a crucial and widely deployed solution for anti-drone operations due to their cost-effectiveness, reliable interception power, and technological maturity. However, conventional fire control systems often struggle with low engagement efficiency and poor adaptability when confronting dynamic, multi-target drone assaults.
Existing research has sought to enhance traditional artillery systems through intelligent algorithms, such as hybrid optimization models for target assignment and improvements to genetic algorithms. Preliminary explorations using tabular Q-learning have demonstrated feasibility but are inherently limited by the curse of dimensionality in high-state environments. To address these persistent challenges in complex, dynamic battlefields, we propose a novel multi-agent deep reinforcement learning framework for anti-drone artillery systems. This framework, which we developed based on our research into multi-agent systems, integrates multi-source situational information and a hierarchical multi-objective reward mechanism, termed the Situational-Fused Hierarchical Multi-Objective Multi-agent Reinforcement Learning for Counter-UAV Systems (SF-HIMO). Its goal is to significantly boost the strike efficiency and intelligent decision-making capabilities of anti-drone networks.
1. Problem Formulation for Anti-Drone Artillery Defense
Our research focuses on a kinetic fire engagement system countering a drone swarm. The objective is, given a specific operational environment and conditions, to simulate an incoming hostile drone formation and optimize the dynamic deployment and fire coordination of defending unmanned weapon platforms (UWPs) through a multi-agent reinforcement learning approach.
Consider an anti-drone engagement scenario with a defined length \(L\), width \(W\), and height \(H\). All entities—hostile drones, defending UWPs, and a high-value asset to protect—exist and operate within this volume. The scenario features one defendable asset, \(n\) mobile UWPs (equipped with radar and an anti-aircraft gun), and \(m\) incoming hostile drones. The drones approach the asset from their initial positions at a constant velocity \(v_{UAV}\). The asset has a defensive capacity \(\xi\). Each UWP moves autonomously within the area at speed \(v_{UGV}\). Its weapon has an effective engagement range of \([\rho_{min}, \rho_{max}]\), a radar detection radius of \(\rho_{radar}\), a muzzle velocity \(v_0\), a firing cooldown period \(t_{cool}\), and a lethality factor \(\omega\). A UWP can open fire when a drone enters its engagement range. The overarching goal is to dynamically discover optimal spatial positioning and fire allocation strategies for the UWPs to successfully protect the asset and maximize the neutralization of the threat.
For model simplification, we establish the following assumptions:
- During training, the battlefield is fully observable. During execution, UWPs perceive their surroundings via radar.
- The firing cooldown \(t_{cool}\) is a fixed constant.
- UWPs adhere to a thermal constraint: \(t^{(i,k+1)}_{fire} – t^{(i,k)}_{fire} \geq t_{cool}\), where \(t^{(i,k)}_{fire}\) denotes the firing time of the \(k\)-th round from UWP \(i\).
- Hostile drones move in a straight line at constant velocity toward the asset.
- The mission is considered a failure if any drone reaches the asset.
2. Algorithm Design for Anti-Drone Engagement
In our anti-drone task, each gun-equipped UWP is modeled as an independent agent. The operational flow is as follows: First, the agent obtains initial state information (including basic and multi-source situational data) from the environment. Second, based on its observed state, the agent uses an artificial neural network within the QMIX framework to make decisions (e.g., move or engage a specific drone). The artillery system then executes the joint action, affecting the environment. Finally, the agent receives a reward and observes the new environmental state, repeating the cycle.
2.1 Dec-POMDP Modeling
We model the anti-drone system task as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP), defined by the tuple \(G = \langle S, U, P, r, Z, O, n, \gamma \rangle\). The true state \(s \in S\) encompasses all information about guns, drones, and the asset. At each timestep, each agent \(a \in A \equiv \{1, …, n\}\) selects an action \(u_a \in U\), forming a joint action \(\mathbf{u} \in \mathbf{U} \equiv U^n\). This triggers a state transition according to \(P(s’|s,\mathbf{u}): S \times \mathbf{U} \times S \rightarrow [0,1]\). All agents share a joint reward function \(r(s, \mathbf{u}): S \times \mathbf{U} \rightarrow \mathbb{R}\). \(\gamma \in [0, 1)\) is the discount factor.
Due to partial observability, each agent receives an individual observation \(z \in Z\) from the observation function \(O(s, a): S \times A \rightarrow Z\). Each agent has an action-observation history \(\tau_a \in \Tau \equiv (Z \times U)^*\) and acts according to a policy \(\pi_a(u_a | \tau_a): \Tau \times U \rightarrow [0,1]\). The joint policy \(\pi\) has a joint action-value function:
$$Q^{\pi}(s_t, \mathbf{u}_t) = \mathbb{E}_{s_{t+1:\infty}, \mathbf{u}_{t+1:\infty}} \left[ R_t | s_t, \mathbf{u}_t \right]$$
where \(R_t = \sum_{i=0}^{\infty} \gamma^i r_{t+i}\) is the discounted return.
We adopt the centralized training with decentralized execution (CTDE) paradigm. During training, the learning algorithm has access to the global state \(s\) and all agents’ action-observation histories. During execution, each agent must act based solely on its own local observation history.
2.2 State Space with Fused Multi-Source Situational Information
The state space comprises the local observation space \(O\) for each agent and the global state space \(S\).
Local Observation Space: An agent’s observation of an entity includes basic information (ID, position, velocity, heading) and multi-source situational information. The latter is designed to enhance global situational awareness for anti-drone operations and includes:
- Threat Degree (\(C_i\)): Calculated via a multi-attribute decision algorithm integrating relative distance \(d_i\), speed \(v_i\), height \(h_i\), and course shortcut \(\beta_i\). For a target set \(T = \{T_i | i=1,…,m\}\), we form a feature matrix \(X \in \mathbb{R}^{m \times 4}\) and compute the threat value after vector normalization and weight assignment (here, equal weights):
$$C_i = \frac{D_i^-}{D_i^+ + D_i^- + \epsilon}$$
where \(D_i^+\) and \(D_i^-\) are distances to the positive and negative ideal solutions, and \(\epsilon=10^{-8}\) ensures numerical stability. - Far/Near Boundary (\(\rho_{far}, \rho_{near}\)): Dynamic interception boundaries based on projectile time-of-flight \(t_p\) and target kinematics, accounting for the non-instantaneous nature of artillery fire in anti-drone engagements:
$$\rho_{far} = \sqrt{ \rho_{max}^2 + (v_{target} t_p)^2 + 2 v_{target} t_p \sqrt{\rho_{max}^2 – h^2 – \beta^2} }$$
$$\rho_{near} = \sqrt{ \rho_{min}^2 + (v_{target} t_p)^2 + 2 v_{target} t_p \sqrt{\rho_{min}^2 – h^2 – \beta^2} }$$ - Time-Sensitive Parameters: Projectile time-of-flight \(t_p\) and target time-to-reach-asset \(t_{target}\).
These elements form a five-dimensional situational vector \(S_i = [C_i, \rho_{far}, \rho_{near}, t_p, t_{target}]\). An agent’s observation \(o_i\) is the concatenation of information for all entities. If entity \(j\) is outside the agent’s radar detection radius \(\rho_{radar}\) (where \(d_{ij} = ||p_j – p_i||_2\)), its information is zeroed out:
$$ o_{ij} = \begin{cases}
[\text{Basic Info}, S_j] & \text{if } d_{ij} \leq \rho_{radar} \\
\mathbf{0} & \text{otherwise}
\end{cases} $$
$$ o_i = [o_{i1}, o_{i2}, …, o_{i(m+n+1)}] $$
Global State Space: The global state is the union of all local observations (calculated with \(\rho_{radar} \to \infty\) for situational info): \(s = \cup_{i=1}^{n} o_i\).
2.3 Action Space Based on Macro-Actions
Each UWP agent has a discrete action space of dimension \(5 + m\):
$$A_i = \left\{ \text{No-op}, \text{Move East/South/West/North}, \text{Fire at UAV } 1, …, \text{Fire at UAV } m \right\}$$
To facilitate high-level decision-making in this complex anti-drone task, we encapsulate the continuous firing process into a discrete macro-action. Once a gun locks onto a target, the entire firing sequence is treated as one atomic action:
- Trajectory Prediction & Impact Point Calculation: Predict future drone position: \(p_{predicted} = p_{target} + v_{target} \cdot t_p\).
- Iterative Ballistic Solution: Solve the ballistic equation for time-of-flight \(t\) given relative horizontal range \(R\) and vertical displacement \(\Delta y\):
$$\frac{1}{4} g^2 t^4 + (g \Delta y – v_0^2) t^2 + (\Delta y^2 + R^2) = 0$$
Solved via Newton’s method or gradient descent. - Elevation Angle Calculation: Compute gun elevation \(\alpha\):
$$\sin \alpha = \frac{y_{target} + \frac{1}{2}gt^2}{v_0 t}$$ - Trajectory Simulation & Damage Verification: Simulate projectile path using Verlet integration with initial velocity \(v_{initial} = [v_0 \cos\alpha \cos\theta, v_0 \sin\alpha, v_0 \cos\alpha \sin\theta]^T\) and acceleration \(\lambda_a = [0, -g, 0]^T\).
- Damage Assessment: If the miss distance \(d\) is within the warhead’s lethal radius \(r_{damage}\), the kill probability is:
$$p_{kill} = 1 – \exp(-\rho \cdot \lambda_0)$$
where \(\rho = \exp(-\frac{d^2}{2\sigma^2})\) is fragment density and \(\lambda_0\) is the lethality coefficient.
2.4 Hierarchical Multi-Objective Reward Design
A key contribution for effective anti-drone learning is our hierarchical multi-objective reward function, designed to guide agents in complex, delayed-feedback environments:
$$
\begin{aligned}
R = & \underbrace{R_1 \cdot \mathbb{I}\{first\_target\}}_{\text{Diversity Exploration}} + \underbrace{R_2 \cdot \mathbb{I}\{fire\_action\}}_{\text{}} \\
& + \underbrace{\sum_{i=1}^{M} R_3 \cdot \frac{t_{current} + t^i_{fire}}{t^i_{flight}}}_{\text{Delayed Gratification (Pre-reward)}} + \underbrace{R_4 \cdot \Delta N_{kill}}_{\text{Immediate Feedback (Kill reward)}} + \underbrace{R_5 \cdot \mathbb{I}\{ammo\_save\}}_{\text{Resource Optimization}} \\
& – \underbrace{R_6 \cdot \mathbb{I}\{out\_of\_bounds\}}_{\text{Behavioral Constraints}} – \underbrace{\sum_{j=1}^{N} R_7 \cdot \frac{1}{d_j}}_{\text{}} \\
& + \underbrace{R_8 \cdot \mathbb{I}_{success} – R_9 \cdot \mathbb{I}_{fail}}_{\text{Global Success/Failure}}
\end{aligned}
$$
where \(R_1, …, R_9\) are scaling coefficients, \(M\) is the number of in-flight projectiles, and \(N\) is the number of UWPs.
The five reward modules are:
1. Diversity Exploration: High reward \(R_1\) for first-time target acquisition encourages covering multiple threats.
2. Delayed Gratification & Immediate Feedback: A pre-reward proportional to time-to-impact mitigates sparse rewards inherent in delayed ballistic anti-drone engagements. Fixed kill reward \(R_4\) provides immediate positive feedback.
3. Resource Optimization: Reward \(R_5\) for shots with high kill probability encourages ammunition efficiency.
4. Behavioral Constraints: Penalties for moving out-of-bounds (\(R_6\)) and for inter-agent collisions (inverse distance penalty \(R_7\)).
5. Global Success/Failure: Large positive (\(R_8\)) or negative (\(R_9\)) reward for mission success/failure provides clear terminal guidance.
2.5 Training with QMIX
We train the anti-drone agent collective using QMIX, a value-based multi-agent RL algorithm employing CTDE. QMIX enforces a monotonic relationship between local agent Q-values and the global Q-value through a mixing network \(f_s\) with non-negative weights:
$$Q_{tot}(\boldsymbol{\tau}, \mathbf{u}) = f_s(Q_1(\tau^1, u_1), …, Q_n(\tau^n, u_n)), \quad \frac{\partial f_s}{\partial Q_a} \geq 0 \ \forall a$$
Training uses experience replay with transitions \((\mathbf{s}, \mathbf{u}, r, \mathbf{s}’, d)\). The loss is the mean squared TD error on mini-batches:
$$\mathcal{L}(\theta) = \sum_{i=1}^{b} \left( Q_{tot}(\mathbf{s}, \mathbf{u}; \theta) – y_i \right)^2, \quad y_i = r + \gamma \max_{\mathbf{u}’} Q_{tot}(\mathbf{s}’, \mathbf{u}’; \theta^-)$$
where \(\theta^-\) are the parameters of a target network.
3. Simulation Experiments and Analysis
3.1 Environment and Parameter Setup
We built a simulation in Python based on the defined anti-drone scenario. A defending force with 3 UWPs and 1 central asset must counter 10 incoming drones in an 8km x 8km area. We designed two primary scenarios: attack_south (all drones attack from the south) and attack_four (drones attack from all four sides). An extended attack_large scenario with 6 UWPs vs. 30 drones was also tested. Key parameters are summarized below.
| UWP ID | Speed (m/s) | Range (m) | Start Pos (m) | Cooldown (s) | Lethality | Radar Range (m) | Muzzle Vel. (m/s) |
|---|---|---|---|---|---|---|---|
| 1 | 10 | 700-2500 | (4500, 4000, 0) | 2 | 100 | 3000 | 1050 |
| 2 | 10 | 700-2500 | (4000, 4500, 0) | 2 | 100 | 3000 | 1050 |
| 3 | 10 | 700-2500 | (3500, 4000, 0) | 2 | 100 | 3000 | 1050 |
| Lethal Radius (m) | Fragment Density | Lethality Coeff. (units/m²) |
|---|---|---|
| 15 | 5 | 1.6 |
| Speed (m/s) | Defense Capacity | Value |
|---|---|---|
| 60 | 60 | 1 |
Our SF-HIMO algorithm was implemented on a finetuned QMIX baseline. We compared it against three popular MARL algorithms: Finetuned-QMIX, QPLEX, and QTRAN. For fairness, baselines used a state space without situational fusion and a simpler reward function. All experiments were run with 5 random seeds, and performance was evaluated by the average win rate over the last 200k training steps.
3.2 Experimental Performance
SF-HIMO demonstrated superior performance across all anti-drone scenarios. Visual analysis of the learned policies showed intelligent emergent behaviors: in attack_south, UWPs formed a dynamic triangular defense formation; in attack_four, they maintained a flexible topology to address threats from all directions; and in the large-scale attack_large scenario, they effectively prioritized and cleared drone groups sector by sector.
| Scenario | SF-HIMO (Ours) | Finetuned-QMIX | QPLEX | QTRAN |
|---|---|---|---|---|
| attack_south | 86 ± 3 | 34 ± 4 | 29 ± 8 | 0 |
| attack_four | 88 ± 3 | 72 ± 4 | 69 ± 7 | 0 |
| attack_large | 78 ± 4 | 69 ± 7 | 43 ± 6 | 0 |
The results confirm SF-HIMO’s effectiveness. It achieved the highest win rates and best stability, with an average performance improvement of 48.9% over the baselines. The significant gain in the more challenging attack_south scenario highlights its advantage in complex anti-drone engagements. QTRAN’s complete failure (0% win rate) validates the non-trivial complexity of our anti-drone simulation.
3.3 Ablation Study
We conducted ablation experiments to isolate the contributions of the two core components of our anti-drone framework.
| Scenario | SF-HIMO (Full) | w/o Situational Info (SF-MR) | w/o Hierarchical Rewards (SF-MS) | Baseline (Finetuned-QMIX) |
|---|---|---|---|---|
| attack_south | 86 ± 3 | 73 ± 8 | 45 ± 7 | 34 ± 4 |
| attack_four | 88 ± 3 | 85 ± 5 | 73 ± 2 | 72 ± 4 |
Removing either component degrades performance, confirming that both are essential and complementary for effective anti-drone learning. The larger performance drop when removing situational information (SF-MS) suggests that enhanced global awareness is particularly critical in this domain.
4. Conclusion
In this work, we addressed the critical challenge of optimizing anti-drone artillery systems through a novel multi-agent deep reinforcement learning approach. Our SF-HIMO framework successfully integrates multi-source situational information to elevate global battlefield awareness and employs a sophisticated hierarchical multi-objective reward mechanism to guide agents through the complexities of delayed ballistic engagements and collaborative fire distribution.
The experimental results across multiple challenging anti-drone scenarios demonstrate that SF-HIMO significantly outperforms existing MARL baselines in terms of both mission success rate and learning stability. The ablation studies further validate the indispensable role of each designed component. This research provides a robust and effective AI-driven solution for enhancing the strike efficiency and adaptive decision-making capabilities of next-generation anti-drone defense networks.