We investigate the problem of anti-jamming and covert communication for large-scale China UAV drones networks operating in dynamic and complex electromagnetic environments where malicious jammers and passive eavesdroppers coexist. The increasing deployment of China UAV drones demands robust communication mechanisms that can ensure both reliable data transmission and low probability of detection. To address this challenge, we propose a novel framework that integrates meta-reinforcement learning with mean-field game theory, enabling distributed channel access decisions while jointly optimizing anti-jamming performance and covertness. Our approach is particularly relevant for China UAV drones swarms that require scalable and adaptive communication solutions in contested spectrum environments.
Meta Mean-Field Q-Learning for Anti-Jamming Covert Communication in China UAV Drones
1. Introduction
China UAV drones have emerged as versatile platforms for a wide range of applications, including surveillance, disaster response, and military operations. However, the open nature of wireless communication links makes China UAV drones vulnerable to both malicious jamming and passive eavesdropping. In large-scale deployments, the limited spectrum resources exacerbate interference coupling among China UAV drones, while the presence of intelligent jammers and listeners further complicates the decision-making process. Traditional anti-jamming methods, such as frequency hopping and direct sequence spread spectrum, consume significant spectrum resources and fail to achieve coordinated decision-making among multiple China UAV drones. Deep reinforcement learning offers a promising paradigm for adaptive spectrum access, but scaling to large networks with hundreds of China UAV drones remains challenging due to the exponential growth of the joint action space. Furthermore, existing studies typically overlook the covertness requirement, focusing solely on anti-jamming performance without considering the risk of detection by passive listeners.
We consider a scenario involving N China UAV drones pairs, L jammers, and a single passive listener, Willie, who attempts to detect transmissions across all channels. Each China UAV drones pair consists of a transmitting drone and a receiving drone operating within a designated flight area. The system has access to M orthogonal narrowband channels, and the number of China UAV drones pairs exceeds the available channels, creating competition and congestion. The goal is to maximize the long-term cumulative communication rate while minimizing the probability of detection by Willie. We formulate this problem as a partially observable stochastic game and propose a meta mean-field Q-learning algorithm that incorporates covertness constraints.
Our main contributions are summarized as follows. First, we construct a comprehensive system model for anti-jamming covert communication among China UAV drones, incorporating time-slotted spectrum sensing, Rician fading channels, and a passive listener model. Second, we design a mean-field Q-learning algorithm that reduces the high-dimensional interaction among China UAV drones to a tractable individual-mean-field approximation, enabling distributed policy learning. Third, we introduce a meta-reinforcement learning mechanism that allows China UAV drones to rapidly adapt to new jamming and eavesdropping scenarios, improving generalization across diverse tasks. Fourth, we validate our approach through extensive simulations, demonstrating superior performance in terms of convergence speed, stability, and covertness.
2. System Model
2.1. Multi-UAV Communication System
We consider a communication system comprising N pairs of China UAV drones, L jammers, and one passive listener Willie. The set of China UAV drones pairs is denoted as $$\mathcal{N} = \{1, 2, …, N\}$$, the set of available channels as $$\mathcal{M} = \{1, 2, …, M\}$$, and the set of jammers as $$\mathcal{L} = \{1, 2, …, L\}$$. Each China UAV drones pair n selects a channel $$w_n(t) \in \mathcal{M}$$ at time slot t and transmits with power $$P_n$$. The jammers operate with power $$P_j$$ on channels $$g_j(t) \in \mathcal{M}$$. The movement of China UAV drones follows a Gauss-Markov random process, where the velocity $$v_n(t)$$ and direction $$\theta_n(t)$$ evolve as:
$$v_n(t) = \alpha_1 v_n(t-1) + (1-\alpha_1)\bar{v}_n + \sqrt{1-\alpha_1^2}\sigma_n$$
$$\theta_n(t) = \alpha_2 \theta_n(t-1) + (1-\alpha_2)\bar{\theta}_n + \sqrt{1-\alpha_2^2}\phi_n$$
where $$\alpha_1, \alpha_2 \in [0,1]$$ control the memory of the process, $$\bar{v}_n$$ and $$\bar{\theta}_n$$ are the average velocity and direction, and $$\sigma_n, \phi_n$$ represent random perturbations. The China UAV drones operate at a fixed altitude of 80 m within a 500 m x 250 m horizontal area.
2.2. Channel Model
The channel between a transmitting China UAV drones and its corresponding receiver is modeled using Rician fading, which captures the dominant line-of-sight component and scattered multipath components. The channel gain $$h_n(t)$$ is expressed as:
$$h_n(t) = \sqrt{10^{-\frac{\rho_0}{10}}d_n^{-\eta}}\left[p_n^{\text{LoS}} h_n^{\text{LoS}} + (1-p_n^{\text{LoS}})h_n^{\text{NLoS}}\right]$$
where $$\rho_0 = 20\log_{10}\left(\frac{4\pi f_c d_0}{c}\right)$$ is the path loss at reference distance $$d_0 = 1$$ m, $$d_n$$ is the distance between the transmitter and receiver, $$\eta$$ is the path loss exponent, $$h_n^{\text{LoS}}$$ is the deterministic line-of-sight component, and $$h_n^{\text{NLoS}} \sim \mathcal{CN}(0,1)$$ is the non-line-of-sight component. The probability of line-of-sight is given by:
$$p_n^{\text{LoS}} = \frac{1}{1 + A\exp[-B(\theta_n – A)]}$$
where A and B are environmental parameters, and $$\theta_n$$ is the elevation angle between the transmitting and receiving China UAV drones. The interference coupling among China UAV drones is determined by a distance threshold $$d_{\text{th}}$$, beyond which the interference power is considered negligible. The neighbor set for China UAV drones pair n is defined as:
$$\mathcal{N}_{\text{ne}}(n) = \{i: i \in \mathcal{N}\backslash\{n\}, d_i \leq d_{\text{th}}\}$$
where $$d_{\text{th}} = \left(\frac{P_0\mathbb{E}[h_{i,n}(t)]}{I_{\text{th}}}\right)^{1/\eta}$$ and $$I_{\text{th}}$$ is the interference threshold. This framework captures the spatial dependence of interference among China UAV drones.
2.3. Time-Slotted Spectrum Sensing Model
We design a time-slotted model that integrates spectrum sensing, decision-making, and data transmission for China UAV drones. Each time slot consists of four stages: position update, spectrum sensing, channel selection and transmission, and feedback exchange. During spectrum sensing, each transmitting China UAV drones scans all available channels to obtain the current occupancy state. Based on the sensing results and historical information, the China UAV drones selects an optimal channel and performs data transmission. The receiver sends acknowledgment or negative acknowledgment signals to complete the feedback loop. Willie operates as a passive listener, monitoring all channels and attempting to decode transmissions. The jammers operate asynchronously with respect to the China UAV drones network, introducing additional uncertainty.
The signal-to-interference-plus-noise ratio at Willie for monitoring China UAV drones pair n is given by:
$$\text{SINR}_w^n(t) = \frac{P_n|h_{n,w}(t)|^2}{\sum_{i \in \mathcal{U}_{q_n(t)}} P_i |h_{i,w}(t)|^2 + N_w + I_{\text{jam}}}$$
where $$\mathcal{U}_{q_n(t)} = \{i: w_i(t) = w_n(t), i \in \mathcal{N}, i \neq n\}$$ represents the set of China UAV drones pairs sharing the same channel, $$N_w$$ is the noise power at Willie, and $$I_{\text{jam}}$$ is the jamming power received by Willie. The communication rate for China UAV drones pair n is:
$$C_n(t) = \begin{cases}
B \log_2(1 + \zeta_n(t)), & \zeta_n(t) \geq \zeta_{\text{th}} \\
0, & \zeta_n(t) < \zeta_{\text{th}}
\end{cases}$$
where B is the channel bandwidth, $$\zeta_n(t)$$ is the SINR at the receiver of China UAV drones pair n, and $$\zeta_{\text{th}}$$ is the decoding threshold. The SINR at the receiver is:
$$\zeta_n(t) = \frac{P_n|h_n|^2}{I_n(t) + J_n(t) + N_n}$$
where $$I_n(t) = \sum_{i \in \mathcal{N}_n} P_i |h_{i,n}(t)|^2 \eta(w_i(t), w_n(t))$$ represents interference from other China UAV drones, $$J_n(t) = \sum_{j \in \mathcal{L}} P_j |h_{j,n}(t)|^2 \eta(g_j(t), w_n(t))$$ represents jamming power, and $$\eta(x, y)$$ is an indicator function equal to 1 when x = y and 0 otherwise. The objective is to maximize the long-term cumulative rate while minimizing detection risk.
3. Problem Formulation
3.1. Partially Observable Stochastic Game
We model the multi-China UAV drones anti-jamming covert communication problem as a partially observable stochastic game, defined by the tuple $$\langle \mathcal{S}, \mathcal{A}, \mathcal{O}, P, r, \gamma \rangle$$. The state space $$\mathcal{S}$$ describes the global system state, including positions of all China UAV drones and jammers, channel occupancy, and jamming history. Since individual China UAV drones cannot access the global state, each China UAV drones pair obtains a local observation $$\boldsymbol{o}_t^n$$ at time t, defined as:
$$\boldsymbol{o}_t^n = [F_{t-1}^n, \boldsymbol{J}_{t-1}^n, \boldsymbol{D}_{t}^{j,n}, \rho(\text{SINR}_w^n(t))]$$
where $$F_{t-1}^n \in \{0,1\}$$ indicates the success of the previous transmission, $$\boldsymbol{J}_{t-1}^n \in \{0,1\}^M$$ is the sensed jamming vector, $$\boldsymbol{D}_{t}^{j,n}$$ contains distances to jammers, and $$\rho(\text{SINR}_w^n(t))$$ is the risk function representing the covertness level:
$$\rho(\text{SINR}_w^n(t)) = \frac{1}{1 + \exp[-k(\text{SINR}_w^n(t) – \tau_{\text{th}})]}$$
where k controls the steepness of the sigmoid and $$\tau_{\text{th}}$$ is the detection threshold. The action space for each China UAV drones pair is the set of available channels: $$a_t^n \in \mathcal{M}$$. The joint action vector is $$\boldsymbol{a}_t = [a_t^1, a_t^2, …, a_t^N]$$. The immediate reward for China UAV drones pair n combines the communication rate with a covertness penalty:
$$\tilde{r}_t^n = C_n(t) – \lambda \rho(\text{SINR}_w^n(t))$$
where $$\lambda > 0$$ is the covertness weight coefficient. The cumulative discounted reward over time horizon T is:
$$\tilde{r}_t^n = \sum_{t=0}^T \gamma^t [C_n(t) – \lambda \rho(\text{SINR}_w^n(t))]$$
3.2. Nash Equilibrium Objective
The policy of China UAV drones pair n maps observations to action probabilities: $$\pi_t^n: O \rightarrow \Omega(a_t^n)$$. The value function under joint policy $$\boldsymbol{\pi}_t$$ is:
$$V_{\boldsymbol{\pi}_t^n}^n(\boldsymbol{o}_t^n) = \sum_{t=0}^T \gamma^t \mathbb{E}_{\pi_t^n, p}[\tilde{r}_t^n | o_0^n = o, \pi_t^n]$$
The optimal policy for each China UAV drones pair aims to maximize its own value function. A Nash equilibrium is reached when no single China UAV drones pair can improve its reward by unilaterally deviating from the joint policy. In a finite partially observable stochastic game, a Nash equilibrium exists under mild conditions. The equilibrium condition is:
$$V_{(\boldsymbol{\pi}_t^n)^*}^n(\boldsymbol{o}_t^n) \geq V_{\pi_t^n, (\boldsymbol{\pi}_t^{-n})^*}^n(\boldsymbol{o}_t^n), \quad \forall \pi_t^n$$
where $$(\boldsymbol{\pi}_t^{-n})^*$$ represents the equilibrium policies of all other China UAV drones pairs. Solving for the exact Nash equilibrium is computationally intractable for large N, motivating the use of mean-field approximations.
4. Algorithm Design
4.1. Mean-Field Q-Learning
To address the high-dimensional coupling among China UAV drones, we introduce mean-field Q-learning, which approximates the multi-agent interaction as a two-player game between an individual China UAV drones pair and a virtual agent representing the average effect of all neighbors. The Q-function for China UAV drones pair n is decomposed as:
$$Q^n(\boldsymbol{o}_t^n, \boldsymbol{a}_t^n) = \frac{1}{N_{\text{ne}}^n} \sum_{i \in \mathcal{N}_{\text{ne}}(n)} Q^n(\boldsymbol{o}_t^n, a_t^n, a_t^i)$$
where $$N_{\text{ne}}^n = |\mathcal{N}_{\text{ne}}(n)|$$ is the number of neighbors. The neighbor actions are expressed as $$a_t^i = \bar{a}_t^n + \delta a_{n,i}$$, where $$\bar{a}_t^n = \frac{1}{N_{\text{ne}}^n} \sum_i a_t^i$$ is the mean action and $$\delta a_{n,i}$$ is the fluctuation. Under the assumption of second-order differentiability, the Taylor expansion yields:
$$Q^n(\boldsymbol{o}^n, \boldsymbol{a}^n) \approx Q^n(\boldsymbol{o}^n, a^n, \bar{a}^n)$$
This simplification reduces the complex multi-agent interaction to a tractable mean-field form. The mean-field Q-function $$Q^n(\boldsymbol{o}^n, a^n, \bar{a}^n)$$ is approximated by a neural network with parameters $$\theta^n$$. The Boltzmann policy for action selection is:
$$\pi^n(a^n | \boldsymbol{o}^n, \bar{a}^n) = \frac{\exp(-\eta Q^n(\theta^n; \boldsymbol{o}^n, a^n, \bar{a}^n))}{\sum_{a’ \in \mathcal{M}} \exp(-\eta Q^n(\theta^n; \boldsymbol{o}^n, a’, \bar{a}^n))}$$
where $$\eta$$ is the temperature parameter controlling exploration. The loss function for updating the neural network is:
$$L(\theta^n) = \mathbb{E}_{<\boldsymbol{o}^n, a^n, r^n, \boldsymbol{o}’^n, \bar{a}^n>} [\tilde{r}^n + \gamma \max_{a’} \hat{Q}^n(\theta’^n; \boldsymbol{o}’^n, a’, \bar{a}’^n) – Q^n(\theta^n; \boldsymbol{o}^n, a^n, \bar{a}^n)]^2$$
where $$\hat{Q}^n$$ is the target network with parameters $$\theta’^n$$, which stabilizes training.
4.2. Meta Mean-Field Q-Learning Algorithm
Standard mean-field Q-learning performs well when the jamming environment is stationary, but struggles to generalize when the jammers change their movement patterns across different scenarios. To address this limitation, we propose a meta-learning framework that allows China UAV drones to rapidly adapt to new jamming and eavesdropping environments. Our meta mean-field Q-learning for anti-jamming and covert communication, denoted as MMFQ-ACC, consists of two phases: training and adaptation.
In the training phase, the jammers follow straight-line trajectories, and we sample tasks from a task distribution $$p(\tau)$$. Each task corresponds to a specific jammer trajectory configuration. For each task, China UAV drones pairs interact with the environment and collect experiences. The inner-loop updates optimize the Q-network parameters for individual tasks, while the outer-loop meta-update extracts shared knowledge across tasks. The meta-update rule is:
$$\theta_{\text{meta}} \leftarrow \theta – \frac{\alpha}{B} \sum_{i=1}^{n_s} \nabla_\theta L_i(\theta)$$
where $$\alpha$$ is the learning rate, $$n_s$$ is the number of samples per task, and $$L_i(\theta)$$ is the loss for the i-th sample. The output of the training phase is a meta-initialization $$\theta_{\text{meta}}$$ that captures cross-task knowledge.
In the adaptation phase, the jammers follow circular trajectories that differ from the training phase. This creates a domain shift that tests the generalization capability of the algorithm. Starting from the meta-initialization, each China UAV drones pair performs a small number of gradient updates using new task experiences to adapt the policy. The adaptation update is:
$$\theta^* \leftarrow \theta_{\text{meta}} – \frac{\beta}{n_s’} \sum_{i=1}^{n_s’} \nabla_{\theta_{\text{meta}}} L_i(\theta_{\text{meta}})$$
where $$\beta$$ is the adaptation learning rate and $$n_s’$$ is the number of adaptation samples. The MMFQ-ACC algorithm enables China UAV drones to quickly adapt to new jamming scenarios without requiring full retraining, significantly improving convergence speed and final performance.
5. Simulation Results and Analysis
5.1. Simulation Setup
We evaluate the proposed MMFQ-ACC algorithm through extensive simulations and compare it with three baseline algorithms: probabilistic Q-learning, independent Q-learning, and standard mean-field Q-learning. The simulation parameters are summarized in Table 1.
| Parameter | Value |
|---|---|
| Number of China UAV drones pairs | 100 |
| Available channels | 80 |
| Number of jammers | 2 |
| Number of listeners | 1 |
| Carrier frequency | 2 GHz |
| Channel bandwidth | 1 MHz |
| Transmit power | 1 W |
| Jammer power | 1 W |
| System noise | -115 dBm |
| SINR threshold | 5 dB |
| Covertness weight | 0.5 |
| Discount factor | 0.95 |
| Training episodes | 200 |
| Time slots per episode | 2000 |
The training data is generated online through interactions between China UAV drones pairs and the simulation environment, producing 40 million experience samples. The training phase uses straight-line jammer trajectories, while the adaptation phase employs circular trajectories to test generalization.
5.2. Performance under Different China UAV Drones Scales
Table 2 presents the normalized reward achieved by different algorithms for varying numbers of China UAV drones. The ratio of China UAV drones pairs to available channels is maintained at 1:2 to simulate spectrum scarcity. As the number of China UAV drones increases from 20 to 200, all algorithms show improved performance, but the MMFQ-ACC algorithm consistently achieves the highest reward. For 200 China UAV drones, the improvement over mean-field Q-learning and independent Q-learning is 10.9% and 37.4%, respectively. This demonstrates the scalability of our approach for large-scale China UAV drones networks.
| Number of China UAV drones pairs | MMFQ-ACC | Mean-field Q | Independent Q | Probabilistic Q |
|---|---|---|---|---|
| 20 | 0.72 | 0.69 | 0.58 | 0.62 |
| 60 | 0.84 | 0.79 | 0.65 | 0.70 |
| 100 | 0.91 | 0.85 | 0.70 | 0.76 |
| 200 | 0.95 | 0.89 | 0.74 | 0.81 |
5.3. Performance under Different Spectrum Availability
Table 3 investigates the impact of available channels on algorithm performance with 100 China UAV drones pairs. When the number of channels is 40, spectrum resources are severely constrained, and the MMFQ-ACC algorithm achieves a normalized reward of 0.78, outperforming mean-field Q-learning and independent Q-learning by 14.5% and 51%, respectively. As channels increase to 80, the performance gap narrows, but MMFQ-ACC still leads by 2.5% over mean-field Q-learning. With 100 channels, the system saturates, and all algorithms reach near-optimal performance. The results highlight the advantage of our algorithm in resource-constrained scenarios.
| Available Channels | MMFQ-ACC | Mean-field Q | Independent Q | Probabilistic Q |
|---|---|---|---|---|
| 40 | 0.78 | 0.68 | 0.52 | 0.60 |
| 60 | 0.87 | 0.79 | 0.63 | 0.72 |
| 80 | 0.92 | 0.88 | 0.73 | 0.79 |
| 100 | 0.96 | 0.94 | 0.81 | 0.86 |
5.4. Covertness Performance
We evaluate the covertness performance by monitoring Willie’s SINR during the learning process. As training progresses, the average Willie SINR decreases from approximately 4 dB to below -2 dB, indicating that the MMFQ-ACC algorithm effectively suppresses the listener’s detection capability. The normalized detection risk probability for Willie decreases by 79.73% compared to the initial training stage, while the normalized throughput for China UAV drones improves by 25.09%. This demonstrates the successful joint optimization of anti-jamming and covertness. Table 4 summarizes the covertness metrics before and after training.
| Metric | Initial Stage | After Convergence | Improvement |
|---|---|---|---|
| Willie average SINR (dB) | 4.0 | -2.1 | −6.1 dB |
| Willie detection risk (normalized) | 0.82 | 0.17 | -79.73% |
| UAV normalized throughput | 0.68 | 0.91 | +25.09% |
5.5. Convergence Performance
Table 5 compares the convergence performance of the algorithms, including the number of episodes required for convergence and the final normalized reward. The MMFQ-ACC algorithm converges in approximately 50 episodes, achieving a final reward of 0.92. This represents a 44.56% improvement in convergence speed compared to mean-field Q-learning, which requires about 90 episodes, and a 16.46% improvement over probabilistic Q-learning. The final reward of MMFQ-ACC is also higher and more stable, with lower variance in the steady-state regime. The learning efficiency of MMFQ-ACC is further validated by the slope of the regression line over time slots within a single episode. The MMFQ-ACC achieves a slope of $$1.6 \times 10^{-3}$$, which is eight times higher than that of mean-field Q-learning ($$2 \times 10^{-4}$$), indicating faster within-episode improvement.
| Algorithm | Convergence Episodes | Final Normalized Reward | Reward Variance (Steady State) |
|---|---|---|---|
| MMFQ-ACC | 50 | 0.92 | 0.008 |
| Mean-field Q | 90 | 0.88 | 0.015 |
| Probabilistic Q | 120 | 0.79 | 0.022 |
| Independent Q | 160 | 0.73 | 0.028 |
5.6. Generalization Performance
Table 6 evaluates the generalization capability of MMFQ-ACC across three task categories with different training and adaptation configurations. In all task categories, MMFQ-ACC achieves significantly higher normalized rewards compared to the baselines. For Category C, which has the largest task diversity, the MMFQ-ACC algorithm reaches a final reward of 0.93, compared to 0.82 for mean-field Q-learning and 0.71 for probabilistic Q-learning. The performance improvement is particularly pronounced in Categories A and B, where adaptation samples are limited, confirming that the meta-initialization enables rapid adaptation to new jamming scenarios. The standard deviation of MMFQ-ACC is also lower, indicating greater stability across different task instances.
| Task Category (Training/Adaptation) | MMFQ-ACC | Mean-field Q | Probabilistic Q | Independent Q |
|---|---|---|---|---|
| A (30 straight / 2 circular) | 0.85 ± 0.02 | 0.68 ± 0.05 | 0.62 ± 0.06 | 0.55 ± 0.07 |
| B (50 straight / 2 circular) | 0.89 ± 0.02 | 0.73 ± 0.04 | 0.66 ± 0.05 | 0.59 ± 0.06 |
| C (50 straight / 50 circular) | 0.93 ± 0.01 | 0.82 ± 0.03 | 0.75 ± 0.04 | 0.68 ± 0.05 |
6. Conclusion
We have presented a comprehensive framework for anti-jamming and covert communication in large-scale China UAV drones networks. The proposed meta mean-field Q-learning algorithm effectively addresses the challenges of high-dimensional coupling, dynamic interference, and detection risk. By combining mean-field approximation with meta-reinforcement learning, our approach enables distributed channel access decisions that jointly optimize throughput and covertness. Extensive simulations demonstrate that the MMFQ-ACC algorithm achieves a 15.41% improvement in normalized reward and a 44.56% improvement in convergence speed compared to traditional methods. The algorithm also exhibits excellent generalization across different jamming and eavesdropping scenarios, making it suitable for practical China UAV drones deployments in complex electromagnetic environments. Future work will extend the framework to incorporate multiple listeners and heterogeneous drone types, as well as explore cross-layer optimization for enhanced security in China UAV drones communication systems.