In recent years, drone technology has rapidly evolved from single-platform operations to large-scale swarm collaborations. This shift is driven by the need for enhanced survivability, functional complementarity, and distributed intelligence in complex electromagnetic environments. However, when facing intelligent adversaries under constrained communication topologies, such as the star topology, and with only partial observations, the collaborative decision-making for heterogeneous unmanned aerial vehicle (UAV) swarms becomes a critical challenge. This work proposes a novel intelligent decision-making framework based on spatio-temporal spectrum awareness to address the collaborative electronic warfare problem for UAV swarms.
We formulate the cooperative jamming and communication problem as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP). This framework integrates heterogeneous node roles, a closed-loop communication-reconnaissance-jamming interaction mechanism, and an intelligent adversary model. Each UAV in the swarm outputs a continuous action vector that includes parameters for communication center frequency, bandwidth, power, velocity, heading, and for multi-functional units, reconnaissance frequency range and jamming center frequency.
System Model and Problem Formulation
We consider a planar electromagnetic confrontation scenario within a 10km × 10km area. The red and blue swarms are structurally symmetric, each comprising one command-and-communication (C2) UAV and five multi-functional (MF) UAVs. The C2 UAV provides strong communication backbone, while each MF UAV integrates basic communication, broadband signal reconnaissance, and directional electromagnetic jamming. The communication topology is a strict star network: MF UAVs can only communicate with the C2 UAV, which acts as the central information relay.
The state vectors for C2 and MF UAVs are defined by the parameters in the tables below.
| Parameter | Symbol | Description |
|---|---|---|
| Position | $\mathbf{p}$ | 2D spatial coordinates |
| Velocity | $v$ | Speed magnitude |
| Heading | $\phi$ | Direction angle |
| Communication center frequency | $f_{c,center}$ | Carrier frequency for data link |
| Communication bandwidth | $B_c$ | Bandwidth of communication signal |
| Communication power | $P_c$ | Transmission power for comms |
| Antenna gains (Tx/Rx) | $G_{tx}, G_{rx}$ | Antenna directivity gains |
| Parameter | Symbol | Description |
|---|---|---|
| Reconnaissance frequency range | $f_{low}, f_{high}$ | Lower and upper bound of passive sensing |
| Jamming center frequency | $f_{j,center}$ | Center frequency of jamming signal |
| Jamming bandwidth | $B_j$ | Bandwidth of jamming signal |
| Jamming power | $P_j$ | Transmission power for jamming |
The communication quality between nodes $i$ and $j$ is modeled using a sigmoid-based quality factor that depends on the Signal-to-Interference-plus-Noise Ratio (SINR):
$$Q_{ij}(SINR) = \frac{1}{1 + \exp(-k \cdot (SINR_{ij} – SINR_{mid}))}$$
where $SINR_{ij}$ is calculated considering free-space path loss, additive white Gaussian noise, and cumulative interference from enemy jammers. The detection probability for a reconnaissance receiver is also a sigmoid function of the received SNR, modulated by the receiver’s operational state (normal, compressed, or saturated).
Each UAV maintains a belief state about all other platforms. The belief state for an enemy platform $e$ includes an estimated state vector $\hat{\mathbf{s}}_e$ and a detection flag $\text{detected} \in \{0,1\}$. This belief is updated via direct observation, communication relay, and decay over time when no new information is received. In the star topology, all MF UAVs report their local beliefs to the C2 UAV, which fuses them using a weighted average based on link quality $Q_{ic}$, and then distributes the fused global belief back to all MF UAVs.
Objective and Constraints
The optimization goal is to maximize the cumulative electromagnetic superiority $\Omega_{em}$ over the mission horizon $T$, defined as a weighted sum of communication potential $\Psi_{comm}$ and stealth potential $\Psi_{stealth}$:
$$\Omega_{em} = w_{comm} \cdot \Psi_{comm} + w_{stealth} \cdot \Psi_{stealth}$$
$$\Psi_{comm} = \frac{\sum Q_{ij}^{red} – \sum Q_{ij}^{blue}}{\sum Q_{ij}^{red} + \sum Q_{ij}^{blue} + \epsilon_0}$$
$$\Psi_{stealth} = \frac{\sum \Pr_{det}^{red \to blue} – \sum \Pr_{det}^{blue \to red}}{\sum \Pr_{det}^{red \to blue} + \sum \Pr_{det}^{blue \to red} + \epsilon_0}$$
The problem is subject to several constraints:
- Star topology constraint: MF UAVs can only communicate with the C2 UAV.
- Action space constraints: All decision variables (e.g., frequency, power, velocity) must remain within their physical limits.
- Rate of change constraints: The maximum allowed change per time step for each action dimension.
- Spatial constraints: All UAVs must stay within the operational area and maintain a minimum safe distance from each other.
- Spectrum conflict rate: Cumulative interference between own communication and reconnaissance bands must be minimized. This is incorporated as a penalty term in the reward function.
The adversary (blue team) employs a greedy utility-based policy, where each UAV evaluates $K$ candidate actions and selects the one that maximizes its own utility function, which includes communication, stealth, reconnaissance, and jamming components.
Spatio-Temporal Spectrum Awareness Multi-Agent Proximal Policy Optimization
To solve the Dec-POMDP under the star topology and partial observability, we propose the Spatio-Temporal Spectrum Awareness Multi-Agent Proximal Policy Optimization (STSA-MAPPO) algorithm. The core innovation is the STSA policy network, which features a synergistic “parse-sense-memory” processing pipeline.
STSA Policy Network Architecture
The STSA network consists of five key modules:
| Module | Function |
|---|---|
| Observation Parser | Converts raw noisy observations into structured tensors for self-state, friend beliefs, and enemy beliefs. |
| Spectrum Awareness Module | Extracts frequency and power features from beliefs, projects them to a high-dimensional embedding, and applies a self-attention mechanism to capture spectrum occupancy and interference patterns. The output is a spectrum context vector $\mathbf{e}_{freq}^{ctx}$. |
| Spatial Coordination Module | Extracts motion and position features, projects them to a spatial embedding $\mathbf{e}_{spatial}$, and uses a serial cross-attention mechanism. The first stage aggregates friend information using the self-embedding as query and friend embeddings as keys/values. The second stage aggregates enemy information using the updated self-embedding as query and enemy embeddings as keys/values. This yields a spatial context vector $\mathbf{h}_{spatial}$. |
| GRU Memory Unit | Concatenates spectrum and spatial context vectors into $\mathbf{h}_{cat}$, which is then fed into a Gated Recurrent Unit (GRU) to capture temporal dependencies. The GRU state $\mathbf{h}_t$ accumulates historical information to infer adversary strategies. |
| Output MLP | A two-layer multilayer perceptron with tanh activation maps the GRU state $\mathbf{h}_t$ to the continuous action vector, scaled to the physical action bounds. |
The spectrum awareness and spatial coordination modules operate in parallel, effectively decoupling the frequency-domain and spatial-domain features. This design allows each module to suppress irrelevant noise and focus on specific physical relationships. The decision flow within the STSA network per agent is described by:
- Parse observation: $\mathbf{o}_t^i \rightarrow \mathbf{s}_{self}, \mathbf{b}_{friend}, \mathbf{b}_{enemy}$.
- Extract frequency features: $\mathbf{x}_{freq} \rightarrow \mathbf{e}_{freq} = \mathbf{W}_{freq} \mathbf{x}_{freq} + \mathbf{b}_{freq}$.
- Apply self-attention: $\mathbf{e}_{freq}^{ctx} = \text{LayerNorm}(\mathbf{e}_{freq} + \text{softmax}(\mathbf{Q}_{freq} \mathbf{K}_{freq}^\top / \sqrt{D_{embed}}) \mathbf{V}_{freq})$.
- Extract spatial features: $\mathbf{x}_{motion} \rightarrow \mathbf{e}_{spatial} = \mathbf{W}_{spatial} \mathbf{x}_{motion} + \mathbf{b}_{spatial}$.
- Serial attention: $\mathbf{h}_{friend} = \text{LayerNorm}(\mathbf{e}_{spatial} + \text{CrossAttn}(\mathbf{e}_{spatial}, \mathbf{E}_{friend}))$.
- Serial attention (cont’d): $\mathbf{h}_{spatial} = \text{LayerNorm}(\mathbf{h}_{friend} + \text{CrossAttn}(\mathbf{h}_{friend}, \mathbf{E}_{enemy}))$.
- Feature fusion: $\mathbf{h}_{cat} = \text{concat}(\mathbf{h}_{spatial}, \mathbf{e}_{freq}^{ctx})$.
- Temporal update: $\mathbf{h}_t = \text{GRU}(\mathbf{h}_{cat}, \mathbf{h}_{t-1})$.
- Action output: $\mathbf{a}_t^i = \text{tanh}(\text{MLP}(\mathbf{h}_t))$, scaled to physical range.
The STSA-MAPPO algorithm follows the Centralized Training with Decentralized Execution (CTDE) paradigm. During training, a central value network $V_\phi(s)$ accesses the global state to compute advantage estimates using Generalized Advantage Estimation (GAE). The policy network (STSA) is updated via the clipped surrogate objective:
$$L(\theta) = \mathbb{E} \left[ \min\left( \rho_t(\theta) \hat{A}_t, \text{clip}(\rho_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right]$$
where $\rho_t(\theta)$ is the probability ratio between the new and old policies. The reward function combines a global reward (electromagnetic superiority minus spectrum conflict penalty) with local rewards tailored to each UAV’s role, plus a collision penalty.
Simulation Results and Analysis
We evaluate STSA-MAPPO against three baselines: MAPPO, MADDPG, and MASAC. All algorithms share the same environment parameters, which are summarized below.
| Parameter | Value |
|---|---|
| Area | 10 km × 10 km |
| Safe distance | 300 m |
| Simulation steps | 150 s with 0.5 s step |
| Number of C2 UAVs | 1 per team |
| Number of MF UAVs | 5 per team |
| Comm. freq. range (C2) | [1.0, 1.6] GHz |
| Comm. freq. range (MF) | [1.0, 1.6] GHz |
| Recon. freq. range (MF) | [1.0, 1.6] GHz |
| Jamming freq. range (MF) | [1.0, 1.6] GHz |
| C2 UAV speed | [0, 150] m/s |
| MF UAV speed | [0, 200] m/s |
| C2 comm. power | [0, 0.2] W |
| MF comm. power | [0, 0.1] W |
| MF jamming power | 1.0 W |
All algorithms are trained using the Adam optimizer with a learning rate of $1 \times 10^{-5}$, discount factor $\gamma = 0.99$, and PPO clip parameter $\epsilon = 0.2$. The total training steps are $9 \times 10^6$.
The following table summarizes the key performance metrics obtained from the evaluation phase (10 episodes after training).
| Metric | STSA-MAPPO | MAPPO | MADDPG | MASAC |
|---|---|---|---|---|
| Average Electromagnetic Superiority $\Omega_{em}$ | 0.35 | 0.10 | -0.05 | -0.15 |
| Average Spectrum Conflict Rate | < 0.10 | 0.25 | 0.35 | 0.40 |
| Average Effective Link Quality | 0.85 | 0.65 | 0.50 | 0.45 |
| Average Our Detection Rate | 0.30 | 0.15 | 0.10 | 0.08 |
| Average Enemy Detection Rate | < 0.05 | 0.20 | 0.30 | 0.35 |
STSA-MAPPO consistently achieves the highest electromagnetic superiority and the lowest internal spectrum conflict rate. Its ability to maintain high communication link quality (0.85) even under intelligent jamming, while simultaneously keeping the enemy’s detection probability below 5%, demonstrates a superior balance between offensive and defensive electronic warfare operations. The heatmap analysis confirms that the learned communication topology rigorously follows the star structure, with all MF-to-MF links having zero quality.
Conclusion
This work presents a comprehensive framework for intelligent decision-making in UAV swarm electronic countermeasures, explicitly addressing the real-world constraints of star-topology communication and partial observability. The proposed STSA-MAPPO algorithm, with its spatio-temporal spectrum awareness policy network, effectively decouples frequency and spatial information, models temporal dependencies, and enables robust collaborative confrontation against a greedy adversary. Simulation results validate that our approach significantly outperforms state-of-the-art multi-agent reinforcement learning baselines in terms of electromagnetic superiority, spectrum efficiency, and survivability. Future work will extend this framework to symmetric intelligent adversary scenarios, exploring equilibrium strategies and policy robustness in more complex games. The continued advancement of drone technology will be essential to further refine these swarm intelligence algorithms for practical deployment.