Our work focuses on advancing drone technology for multi-unmanned aerial vehicle (UAV) game confrontation. In modern aerial combat, drone technology plays a pivotal role in enabling autonomous, cooperative maneuvers. However, existing decision-making methods often suffer from cumbersome reward design, training instability, and low sample efficiency in complex scenarios. To address these challenges, we propose a novel algorithm, Critic and Meta Enhanced Multi-Agent Deep Deterministic Policy Gradient (CM-MADDPG), which integrates meta-reinforcement learning with multi-agent reinforcement learning. By enhancing the centralized critic with structural improvements and leveraging a meta-strategy for adaptive scheduling during training, our approach significantly boosts convergence speed and sample efficiency without increasing inference computational cost.
In this paper, we model the 3×3 multi-drone confrontation as a discrete-time Markov game in a two-dimensional bounded domain. Both the red team (our agents) and the blue team (random opponents) consist of three fixed-wing UAVs. We adopt the Centralized Training and Decentralized Execution (CTDE) paradigm. During training, the centralized critic has access to full joint observations and actions, while during execution each agent acts based solely on its local observation. The reward function is sparse and includes a base step penalty, chase reward, alignment reward, boundary penalty, formation penalty, and firing settlement. The terminal condition is defined by the cumulative team return, with the winner being the team with higher total score.
The core contributions of our paper are: 1) Introducing mild structural and learning enhancements on the centralized critic side to improve cross-UAV interaction value estimation accuracy and suppress overestimation, thereby making training more stable and achieving higher steady-state performance. 2) Adopting a meta-strategy during training for adaptive scheduling (e.g., dynamically adjusting loss weights and exploration intensity) to improve sample efficiency and convergence speed without adding inference overhead. 3) Comprehensive simulations comparing CM-MADDPG with baseline MADDPG, Critic-MADDPG, and Meta-MADDPG, demonstrating significant improvements in win rate, per-UAV reward, and convergence speed.
1. System Model and Confrontation Environment
1.1 Task and Scenario Description
We construct a two-team multi-UAV confrontation benchmark. The red team (set R) and blue team (set B) each consist of three fixed-wing UAVs. The entire set is N = R ∪ B. The domain is a two-dimensional square Ω = [−S, S]², where S = 1.0. The red team uses the trained intelligent policy, while the blue team employs a random policy during training to create a uniform difficulty baseline. Both teams can shoot and evade, and the scoring rules are symmetric. The objective is to achieve higher cumulative team reward within a finite number of steps.
We set a fixed time step Δt = 0.1, maximum steps per episode Tmax = 100. The reward design includes base scoring, approach and alignment shaping, boundary and formation penalties, and hit/miss/cooling-period firing settlement, forming an interpretable task-scoring loop.
1.2 Operational Space, Time, and Notation
The 2D battlefield Ω = [−1, 1]². Time is discretized as t = 0,1,…,Tmax−1 with step Δt=0.1. Each red UAV is indexed i∈{adv1,adv2,adv3}; each blue UAV is indexed i∈{agt1,agt2,agt3}. The position of UAV i at time t is pi(t) = [xi(t), yi(t)]T, yaw angle ψi(t), and scalar speed vi(t).
Default parameters: min speed vmin=0, max speed vmax=1.0, max angular speed ωmax=2.0, max acceleration step amax=0.2, hit distance threshold rhit=0.15, alignment threshold cmin=0.75, boundary safety factor Ssafe=0.92, team minimum distance dteam=0.25, firing cooldown steps Tcool=5.
1.3 UAV Dynamics and Action Mapping
For any UAV i ∈ N, its state is si(t) = [xi(t), yi(t), ψi(t), vi(t)]T. The dynamics with hard boundary clipping are:
$$ \begin{aligned}
x_i(t+1) &= \text{clip}\big(x_i(t) + v_i(t)\cos\psi_i(t)\,\Delta t,\; -S,\; S\big) \\
y_i(t+1) &= \text{clip}\big(y_i(t) + v_i(t)\sin\psi_i(t)\,\Delta t,\; -S,\; S\big) \\
\psi_i(t+1) &= \psi_i(t) + \Delta\psi_i(t) \\
v_i(t+1) &= \text{clip}\big(v_i(t) + \Delta v_i(t),\; v_{\min},\; v_{\max}\big)
\end{aligned} $$
where Δψi(t) ∈ {−ωmax, 0, ωmax} and Δvi(t) ∈ {−amax, 0, amax}. Each discrete action ai(t) ∈ A={0,1,2,3,4,5} maps to: 0 = hold, 1 = left turn, 2 = right turn, 3 = accelerate, 4 = decelerate, 5 = fire.
1.4 Integrated Perception-Confrontation-Reward Modeling
Under CTDE, each agent receives a local observation of 14 dimensions: the (x,y) coordinates of all 6 UAVs (3 red, 3 blue) plus its own speed and yaw angle.
Target association: For a red UAV i, the opponent set O(i) = B. The nearest enemy distance is:
$$ d_{\min,i}(t) = \min_{j\in O(i)} \| \mathbf{p}_i(t) – \mathbf{p}_j(t) \|_2 $$
The alignment score is:
$$ a_i(t) = \cos\big(\psi_i(t) – \text{angle}(\mathbf{p}_{j^*}(t) – \mathbf{p}_i(t))\big), \quad j^* = \arg\min_{j\in O(i)} \| \mathbf{p}_i(t) – \mathbf{p}_j(t) \|_2 $$
A valid firing requires: action = 5, cooldown = 0, ∃ j ∈ O(i) with distance ≤ rhit and ai(t) ≥ cmin. The reward function is:
$$ r_i(t) = r_{\text{base}} + r_{\text{chase},i}(t) + r_{\text{align},i}(t) + r_{\text{bdry},i}(t) + r_{\text{space},i}(t) + r_{\text{fire},i}(t) $$
Details of each reward component are summarized in Table 1.
| Name | Condition | Value |
|---|---|---|
| Base step penalty rbase | Every step | −0.003 |
| Chase reward rchase | Distance to nearest enemy decreases | 0.015·[dmin(t−1) − dmin(t)] |
| Alignment reward ralign | Heading more toward nearest enemy | 0.008·ai(t) (∈[−1,1]) |
| Boundary penalty rbdry | max(|x|,|y|) > 0.92 | −0.012 |
| Formation penalty rspace | ∃ teammate k ≠ i with ‖pk−pi‖ < 0.25 | −0.012 |
| Hit reward (shooter) | Valid firing hits opponent | +10.0 |
| Hit penalty (victim) | Opponent hits self | −10.0 |
| Miss penalty (shooter) | Firing but not hitting | −0.12 |
| Cooldown firing penalty | Firing while cooling | 0.1 × (−0.12) = −0.012 |
1.5 Episode Termination and Win/Loss
Episode ends when t = Tmax. The team reward is the sum of individual rewards over the episode. Red wins if Rred > Rblue. We record the sliding win rate over 200 episodes to assess convergence and stability.
2. MADDPG Algorithm and the Proposed CM-MADDPG
2.1 MADDPG Algorithm
MADDPG extends DDPG to multi-agent settings under CTDE. Each agent has an actor network πθi that outputs deterministic actions, and a critic network Qi that evaluates the joint state-action value. The critic receives full observations and actions of all agents during training, but is removed at execution. The policy gradient for agent i is:
$$ \nabla_{\theta_i} J(\theta_i) = \mathbb{E}_{s,a\sim D} \big[ \nabla_{\theta_i} \pi_i(a_i|o_i) \; \nabla_{a_i} Q_i(s, a_1,…,a_n) \big] $$
The critic is updated by minimizing the loss:
$$ \mathcal{L}(\theta_i^Q) = \mathbb{E}_{(s,a,r,s’) \sim D} \big[ (Q_i(s,a) – y_i)^2 \big] $$
where yi = ri + γ Qitarget(s‘, a‘) with a‘ from target policies.
2.2 CM-MADDPG Algorithm
We propose CM-MADDPG to overcome the limitations of standard MADDPG under sparse rewards and high-dimensional joint input. Our enhancements are twofold:
2.2.1 Critic Enhancement: Attention + Twin-Q
Instead of using a plain MLP for the centralized critic, we introduce an attention mechanism to let the critic focus on the most relevant agents (both teammates and opponents) for the evaluated agent i. For each agent j, we compute an embedding zj = σ(Wooj + Waaj). Then we compute the context vector ci via scaled dot-product attention:
$$ \mathbf{q}_i = \mathbf{W}_q \mathbf{z}_i, \quad \mathbf{k}_j = \mathbf{W}_k \mathbf{z}_j, \quad \mathbf{v}_j = \mathbf{W}_v \mathbf{z}_j $$
$$ \alpha_{ij} = \frac{e^{\mathbf{q}_i^{\mathsf{T}} \mathbf{k}_j / \sqrt{d}}}{\sum_{\ell} e^{\mathbf{q}_i^{\mathsf{T}} \mathbf{k}_\ell / \sqrt{d}}}, \quad \mathbf{c}_i = \sum_j \alpha_{ij} \mathbf{v}_j $$
The critic value is then Qi(o,a) = f([zi; ci]). To reduce overestimation, we adopt a Twin-Q architecture: two separate critics (Q1, Q2), and the target value uses the minimum of the two:
$$ y_i = r_i + \gamma \, (1-d) \, \min_{m=1,2} Q_i^{m,\text{target}}(\mathbf{o}’, \mathbf{a}’) $$
The critic loss for each m is:
$$ \mathcal{L}_{\text{critic}}^m = \frac{1}{|B|} \sum_{(\mathbf{o},\mathbf{a},r,\mathbf{o}’,d) \in B} \big( Q_i^m(\mathbf{o},\mathbf{a}) – y_i \big)^2 $$
Target networks are updated via soft update: θtarget ← τθ + (1−τ)θtarget.
2.2.2 Meta-Strategy Enhancement: Reptile + Adaptive Scheduling
We apply a Reptile-style meta-update every Kmeta episodes to improve sample efficiency. Let θ be the current policy parameters. We perform a few inner-loop gradient steps to obtain fast parameters θfast, then pull the parameters towards the fast solution:
$$ \boldsymbol{\theta} \leftarrow \boldsymbol{\theta} + \beta (\boldsymbol{\theta}^{\text{fast}} – \boldsymbol{\theta}) $$
Additionally, we adaptively schedule the exploration temperature τexp and loss weights based on a sliding win rate W200 and batch TD-error variance Var(y). The rule is:
$$ \tau_{\text{exp}}(t) = \tau_0 \cdot \frac{1}{2} \big( 1 + \exp(-\rho (W_{200}(t) – \eta)) \big) $$
where ρ and η are hyperparameters. When win rate is low, temperature increases to encourage exploration; when win rate stabilizes, temperature decays to reduce randomness.
2.3 Training Procedure of CM-MADDPG
The pseudo-code is summarized in Table 2.
| Step | Operation |
|---|---|
| 1 | Initialize all actor and critic networks (with attention and Twin-Q) and their target networks. |
| 2 | Initialize replay buffer D. |
| 3 | For each episode = 1 to M: |
| 4 | Reset environment; each agent receives initial observation oi. |
| 5 | For each step t = 0 to Tmax−1: |
| 6 | Select action ai = Gumbel-Softmax(πθi(oi), τexp). |
| 7 | Execute actions, observe reward ri and next observation oi‘. |
| 8 | Store transition (o, a, r, o‘, d) in D. |
| 9 | Sample a mini-batch from D. |
| 10 | Compute targets yi using Eq. (20). |
| 11 | Update both critics by minimizing Eq. (21). |
| 12 | Update actors using deterministic policy gradient with entropy regularization Eq. (22). |
| 13 | Soft update target networks. |
| 14 | End for (per step). |
| 15 | Every Kmeta episodes, perform Reptile meta-update (Eq. 23). |
| 16 | Adaptively schedule τexp using Eq. (25). |
| 17 | End for (per episode). |
3. Simulation Results and Analysis
3.1 Experimental Setup
We implemented the simulation environment in Python using CUDA 12.8 and PyTorch 2.8.0. The red team is trained using four algorithms: MADDPG, Critic-MADDPG (only critic enhancement), Meta-MADDPG (only meta enhancement), and our full CM-MADDPG. The blue team always uses a random policy (random maneuvers and random firing with cooldown constraints). All algorithms share the same hyperparameters as listed in Table 3.
| Name | Value |
|---|---|
| Discount factor γ | 0.95 |
| Soft update coefficient τ | 0.02 |
| Batch size | 1024 |
| Actor learning rate | 1×10−3 |
| Critic learning rate | 1×10−3 |
| Attention/embedding dimension | 32 |
| Meta interval (episodes) Kmeta | 1000 |
| Meta outer step size β | 3×10−3 |
3.2 Win Rate Comparison
Figure 1 (included earlier) shows the win rate curves for the four methods over 30,000 training episodes. The raw per-episode results (faint blue line) are smoothed using a 200-episode moving average (orange line). Key observations:
- MADDPG: The win rate slowly rises from about 0.20 to 0.70–0.75 after 30,000 episodes, with high variance.
- Critic-MADDPG: Win rate reaches 0.70 by episode 1,000 and stabilizes around 0.75–0.85, but periodic fluctuations remain.
- Meta-MADDPG: Faster early convergence – win rate hits 0.85 by episode 500 and stays between 0.85–0.90 with smoother curve.
- CM-MADDPG: Win rate climbs to above 0.95 by episode 1,000 and remains close to 1.0, with minimal variance. This demonstrates the combined benefit of critic enhancement and meta-strategy.
3.3 Individual UAV Rewards
Figures 6–9 (not explicitly numbered here but described in original) show the per-UAV reward evolution. Red UAVs are labeled adv1-3, blue UAVs agt1-3. The key results:
- MADDPG: Both red and blue rewards hover near zero with occasional positive spikes, indicating poor coordination.
- Critic-MADDPG: Red UAVs show more frequent positive spikes; the separation between red and blue averages increases, but variance remains high.
- Meta-MADDPG: Red averages become positive earlier; blue averages drift negative; overall smoother.
- CM-MADDPG: Red UAVs consistently achieve the highest and most frequent positive spikes, and their moving averages remain well above blue, demonstrating stable cooperative advantage.
3.4 Quantitative Performance Summary
We compare the final win rate (average over last 5,000 episodes) and average red team reward per episode. The results are shown in Table 4.
| Algorithm | Final Win Rate (average last 5000 episodes) | Average Red Team Reward |
|---|---|---|
| MADDPG | 0.73 | −0.12 |
| Critic-MADDPG | 0.81 | 0.24 |
| Meta-MADDPG | 0.89 | 0.51 |
| CM-MADDPG (ours) | 0.97 | 1.08 |
Our CM-MADDPG achieves the highest win rate and the best team reward, confirming the effectiveness of the combined enhancements. The inference latency remains identical to MADDPG since only the actor is used at test time.
4. Conclusion
In this work, we presented CM-MADDPG, a novel algorithm for multi-drone game confrontation that synergistically combines critic-side enhancement (attention + Twin-Q) with meta-strategy scheduling. Our approach addresses the challenges of sparse rewards and training instability in multi-UAV scenarios without increasing inference cost. Extensive simulations on a 3×3 benchmark show that CM-MADDPG significantly outperforms MADDPG and its single-enhanced variants in terms of convergence speed, final win rate, and per-UAV reward. The results underscore the potential of integrating structure-aware value modeling with meta-learning for advancing drone technology in autonomous aerial combat.
Future work will extend the framework to 3D environments, stronger opponent policies, and larger-scale multi-UAV formations, further validating the robustness of our method. The advancements in drone technology demonstrated here pave the way for more intelligent, autonomous, and cooperative aerial systems.