In modern military operations, UAV drone swarms have become a pivotal force due to their low cost, high redundancy, and flexible self-organization capabilities. These UAV drone systems are extensively deployed in complex missions such as situational awareness, precision strikes, electronic warfare, and swarm combat. However, achieving efficient coordination among large-scale UAV drone clusters in highly dynamic, uncertain, and adversarial battlefield environments poses significant challenges. Key issues include partial observability, communication constraints, and strong adversarial disturbances, which complicate distributed real-time optimal decision-making and adaptive cooperative strategy generation. Traditional rule-based or centralized control methods often lack flexibility, robustness, and adaptability to unknown situations. To address these limitations, multi-agent reinforcement learning (MARL) offers a promising framework. MARL enables UAV drone agents to learn near-optimal distributed cooperative strategies through autonomous exploration and experience learning, without requiring precise global models, effectively handling high-dimensional state spaces, partial observability, and complex cooperative-competitive relationships among agents.
Among MARL approaches, the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm introduces a centralized training with decentralized execution (CTDE) framework, mitigating environmental non-stationarity by using global critic networks and optimizing value functions for distributed decision-making. Its variant, the Additional Opponent Characteristics MADDPG (AOC-MADDPG) algorithm, enhances robustness by incorporating opponent behavior modeling. However, AOC-MADDPG still faces challenges such as high computational complexity, poor training stability, slow convergence, and weak generalization in high-dimensional state spaces. In this article, I propose an improved AOC-MADDPG algorithm that optimizes network architecture, experience replay mechanisms, noise exploration strategies, and integrates a curiosity module. These enhancements aim to boost convergence speed, generalization ability, and stability, while effectively addressing gradient-related issues in UAV drone cooperative combat scenarios.
The core of the improved algorithm lies in several technical innovations. First, I optimize the network structure by extending depth and width, applying Dropout regularization, and implementing batch normalization. This enhances representation power, prevents overfitting, and accelerates training convergence. The updated Actor and Critic networks for each UAV drone agent are designed to handle complex state-action spaces more efficiently. For instance, the Critic network’s target update is governed by: $$ \theta^{Q}_{t+1} = \tau \theta^{Q}_{t} + (1 – \tau) \theta^{Q}_{t} $$ where $\theta^{Q}_{t}$ represents the Critic network parameters at time $t$, and $\tau \in [0,1]$ is the soft update coefficient. Similarly, the Actor network’s target update follows: $$ \theta^{\mu}_{t+1} = \tau \theta^{\mu}_{t} + (1 – \tau) \theta^{\mu}_{t} $$ ensuring stable policy improvements.
Second, I refine the experience replay mechanism by incorporating adjustable priority parameters and multi-step returns. Priority sampling is based on temporal-difference (TD) error, calculated as: $$ p_i = |Q(O_t, A_t) – Q(O_{t+1}, A_{t+1})| + \epsilon_2 $$ where $\epsilon_2$ is a small constant to avoid zero priority. The sampling probability $P_i$ is then: $$ P_i = \frac{p_i^{\alpha}}{\sum_{k=1}^{s} p_k^{\alpha}} $$ with $\alpha$ as the prioritization exponent. To correct bias, importance sampling weights $\omega_i$ are applied: $$ \omega_i = \left( \frac{s \cdot P_i}{\max_j (s \cdot P_j)} \right)^{-\beta_s} $$ where $\beta_s$ increases from 0.4 to 1 over training. Multi-step returns balance bias and variance by accumulating rewards over $n_s$ steps: $$ R_t^{n_s} = R_t + \gamma R_{t+1} + \gamma^2 R_{t+2} + \cdots + \gamma^{n_s-1} R_{t+n_s-1} $$ with $\gamma$ as the discount factor. This enhances sample efficiency and provides denser feedback in sparse reward environments for UAV drone agents.
Third, I enhance the noise exploration strategy by combining Ornstein-Uhlenbeck (OU) noise and Gaussian noise. The mixed noise $n_{\text{mixed}}$ is given by: $$ n_{\text{mixed}} = \alpha_{\text{OU}} \cdot n_{\text{OU}} + (1 – \alpha_{\text{OU}}) \cdot n_{\text{G}} $$ where $n_{\text{OU}}$ is OU noise with parameters $\theta_{\text{OU}}=0.25$ and $\sigma_{\text{OU}}=0.15$, $n_{\text{G}}$ is Gaussian noise with $\sigma_{\text{G}}=0.07$, and $\alpha_{\text{OU}}=0.4$ is the weight. This mixture promotes smooth exploration and action consistency, aiding UAV drone agents in adapting to non-stationary environments.
Fourth, I introduce an intrinsic curiosity module (ICM) to drive exploration. The ICM includes a feature encoder, forward dynamics model, and inverse dynamics model. The intrinsic reward $r_{\text{in}}^i$ for UAV drone agent $i$ is computed as: $$ r_{\text{in}}^i = e_f^i + \beta_{\text{in}} e_{\text{in}}^i $$ where $e_f^i$ is the forward prediction error: $$ e_f^i = ||\hat{f}(o_{t+1}^i) – f(o_{t+1}^i)||^2 $$ and $e_{\text{in}}^i$ is the inverse dynamics error: $$ e_{\text{in}}^i = ||\hat{a}_t^i – a_t^i||^2 $$ with $\beta_{\text{in}}=0.2$. The total reward $r_t^i$ combines extrinsic reward $r_a^i$ and intrinsic reward: $$ r_t^i = r_a^i + \lambda_{\text{in}} r_{\text{in}}^i $$ where $\lambda_{\text{in}}=0.01$. This mechanism encourages UAV drone agents to explore novel states, mitigating sparse reward problems.
To validate the improved algorithm, I conduct simulation experiments in a multi-UAV drone combat environment. The setup involves red and blue teams, each with 10 UAV drone agents. The red team uses the improved AOC-MADDPG algorithm, while the blue team employs rule-based control strategies derived from expert knowledge. The simulation is implemented in Python 3.8 using OpenAI Gym, with key hyperparameters summarized in Table 1.
| Parameter | Symbol | Value |
|---|---|---|
| Experience Replay Buffer Size | $K$ | $10^6$ |
| Batch Size | $p_B$ | 512 |
| Maximum Steps per Episode | $\text{Maxstep}$ | 300 |
| Hidden Layer Dimension | $h$ | 256 |
| Actor Learning Rate | $\eta_a$ | $10^{-4}$ |
| Critic Learning Rate | $\eta_c$ | $10^{-3}$ |
| Soft Update Coefficient | $\tau$ | 0.01 |
| Discount Factor | $\gamma$ | 0.99 |
| Momentum Decay Coefficient | $\lambda$ | 0.95 |
| Prioritization Exponent | $\alpha$ | 0.6 |
| Importance Sampling Weight (Initial) | $\beta_s$ | 0.4 |
| Multi-step Return Steps | $n_s$ | 3 |
| OU Noise Weight | $\alpha_{\text{OU}}$ | 0.4 |
| OU Noise Standard Deviation | $\sigma_{\text{OU}}$ | 0.15 |
| Gaussian Noise Standard Deviation | $\sigma_{\text{G}}$ | 0.07 |
Reward settings are designed to incentivize positive behaviors in UAV drone combat, as detailed in Table 2. These include rewards for combat outcomes (e.g., destroying enemy UAV drone agents), survival, tactical execution, teamwork, efficiency, and penalties for losses.
| Category | Action | Reward Value |
|---|---|---|
| Combat Outcome | Destroy Enemy UAV Drone | +100 |
| Damage Enemy UAV Drone | +20 | |
| Final Victory | +500 | |
| Survival | Per-step Survival | +1 |
| Maintain Formation Integrity | +5 per step | |
| Tactical Execution | Detect Enemy | +10 |
| Successful Attack | +30 | |
| Successful Jamming | +15 | |
| Complete Role Task | +50 | |
| Teamwork | Concentrated Fire | +20 |
| Maintain Formation | +10 | |
| Cooperative Defense | +15 | |
| Efficiency | Quick Victory | +(500 – steps) |
| Ammunition Conservation | +50 × remaining ratio | |
| Loss Penalty | UAV Drone Destroyed | -100 |
| UAV Drone Damaged | -20 | |
| Formation Broken | -30 |
I conduct 200 episodes for both the baseline AOC-MADDPG and the improved algorithm, with the red team as learning agents and the blue team as rule-based opponents. Performance metrics include average cumulative reward, win rate, survival rate, and average steps per episode. The results demonstrate significant improvements. Under the baseline model, the red team achieves an average cumulative reward of 2100.5, a win rate of 72.5%, a survival rate of 71.5%, and an average step count of 162.8. In contrast, the improved model yields an average cumulative reward of 3277.9, a win rate of 89%, a survival rate of 88.8%, and an average step count of 160.3. This represents a 56% increase in cumulative reward, a 22.8% increase in win rate, a 24.2% increase in survival rate, and a 1.5% reduction in steps, indicating that UAV drone agents learn more efficient strategies without prolonging combat unnecessarily.
The training stability is also enhanced, as evidenced by reduced TD errors, Actor loss, and Critic loss. The proximity of survival rate (88.8%) to win rate (89%) suggests effective credit assignment and robust cooperative strategies among UAV drone agents. The small difference between win rate and survival rate (0.2% vs. 1% in baseline) underscores improved policy consistency and adaptability in dynamic UAV drone engagements.
In summary, the improved AOC-MADDPG algorithm addresses key limitations in UAV drone cooperative combat by integrating network optimizations, advanced experience replay, mixed noise exploration, and curiosity-driven learning. These contributions lead to superior performance in terms of reward accumulation, victory likelihood, agent survivability, and training efficiency. The algorithm provides a stable and scalable solution for multi-UAV drone systems operating in complex adversarial environments. Future work could extend this approach to heterogeneous UAV drone swarms or real-world deployment scenarios, further advancing autonomous combat capabilities.
The mathematical formulations and experimental validations highlight the algorithm’s robustness. For instance, the policy gradient update for UAV drone agent $i$ is computed as: $$ \nabla_{\theta^i} J(\theta^i) = \mathbb{E}_{O_t, A_t} \left[ \nabla_{a^i} Q(O_t, A_t) \nabla_{\theta^i} \mu^i(o^i_t, o^{-i}_t) \right] $$ where $Q(O_t, A_t)$ is the centralized action-value function, and $\mu^i$ is the policy of agent $i$. This ensures coordinated learning across UAV drone agents. Additionally, the batch normalization process for each layer’s activations $x_i$ is expressed as: $$ \hat{x}_i = \frac{x_i – \mu_B}{\sqrt{\sigma_B^2 + \epsilon_1}}, \quad y_i = \gamma_1 \hat{x}_i + \beta_1 $$ where $\mu_B$ and $\sigma_B^2$ are batch mean and variance, and $\gamma_1$, $\beta_1$ are learnable parameters. This accelerates convergence and mitigates internal covariate shift in deep networks for UAV drone state processing.
Overall, the integration of these techniques fosters a more intelligent and adaptive UAV drone swarm capable of handling the intricacies of modern warfare. The repeated emphasis on UAV drone throughout this article underscores its central role in the research, from algorithm design to simulation outcomes. As UAV drone technology continues to evolve, such MARL-based approaches will be crucial for achieving superior air dominance and mission success in contested environments.