In the rapidly evolving domain of unmanned aerial vehicle (UAV) ad hoc networks, particularly those deployed for China UAV drones, the need for efficient medium access control (MAC) protocols is critical. The statistic priority-based multiple access (SPMA) protocol has emerged as a promising solution due to its ability to differentiate service priorities and adapt to dynamic channel conditions. However, the backoff mechanism in SPMA, which determines how long a node waits before attempting transmission, significantly impacts latency, throughput, and reliability. Traditional backoff strategies, such as binary exponential backoff (BEB) or logarithmic function-based approaches, rely on static models and single-dimensional optimization parameters, making them inadequate for the highly dynamic and multi-priority traffic environments typical of China UAV drones. To address this challenge, we propose a novel intelligent backoff strategy based on the double deep Q-network (DDQN) algorithm. This strategy models the dynamic decision process of selecting backoff time as a Markov decision process (MDP) and leverages DDQN to learn optimal backoff times in a finite, discrete action space, considering multiple factors including service priority, threshold, and channel load. Through extensive simulations, we demonstrate that our approach significantly reduces transmission delay, improves initial backoff success rate, and enhances overall transmission success rate, while adapting effectively to varying network scales. This paper provides a comprehensive framework for integrating deep reinforcement learning into SPMA protocol design, offering a robust solution for mission-critical communications in China UAV drones.
1. Introduction
China UAV drones are increasingly deployed in diverse applications, including disaster relief, tactical reconnaissance, smart agriculture, and cargo delivery. These applications demand reliable and low-latency communication to support command-and-control traffic, video streaming, and sensor data transmission. The dynamic nature of UAV ad hoc networks, characterized by high mobility, limited resources, and heterogeneous traffic, poses significant challenges for traditional MAC protocols. Among the emerging protocols, the statistic priority-based multiple access (SPMA) protocol stands out due to its unique ability to prioritize high-priority traffic while maintaining high channel utilization. However, the performance of SPMA heavily depends on the backoff strategy, which governs how long a node defers transmission when the channel load exceeds a priority-specific threshold.
Existing backoff mechanisms for SPMA, such as BEB and variants using exponential or logarithmic functions, are designed for static environments and fail to capture the dynamic interdependence between channel load, priority thresholds, and backoff duration. In China UAV drones, traffic patterns are bursty, and channel conditions change rapidly due to node movement and interference. Consequently, static backoff models lead to suboptimal performance, causing either excessive collisions or underutilization of channel resources. To overcome these limitations, we propose an intelligent backoff strategy that leverages reinforcement learning (RL), specifically the double deep Q-network (DDQN) algorithm, to dynamically select appropriate backoff times. By formulating the backoff decision as an MDP, our approach learns from interactions with the network environment and optimizes long-term cumulative reward.
The key contributions of this paper are as follows:
- We model the backoff time selection process in SPMA as a Markov decision process, establishing a dynamic mapping among environment, state, action, and reward, thereby overcoming the limitations of static function models.
- We design a DDQN-based intelligent backoff strategy that comprehensively considers service priority, priority threshold, and channel load to select backoff times within a finite discrete action space.
- We conduct extensive simulations to validate the proposed strategy, showing significant improvements in transmission delay (up to 33.3% reduction for low-priority traffic), initial backoff success rate (18% improvement), and transmission success rate compared to BEB and logarithmic backoff strategies.
The remainder of this paper is organized as follows. Section 2 presents the system model and problem formulation. Section 3 reviews existing backoff strategies and highlights their limitations. Section 4 details the proposed DDQN-based intelligent backoff algorithm. Section 5 presents simulation results and analysis. Section 6 concludes the paper.
2. System Model and Problem Formulation
We consider a single-hop, fully connected China UAV ad hoc network consisting of N UAV nodes. All nodes are equal and can communicate directly with each other. The frequency domain is divided into Nf orthogonal channels (Nf < N), each with identical bandwidth and access conditions. Data packets arrive at each node according to a Poisson process with rate λ (packets per second). These packets are categorized into p priority levels, where 0 denotes the highest priority and p–1 the lowest. The generation rates for each priority class follow independent Poisson processes with rates λ0, λ1, …, λp-1, such that λ = Σ λi. All nodes transmit at the same rate, and packet sizes are fixed.
In the SPMA protocol, a node decides whether to transmit or back off based on a comparison between the current channel load C and a priority-specific threshold Tih. The threshold Tih is designed to maintain the channel load near the full-load threshold C0, which represents the maximum network capacity achieving 99% transmission success probability. The backoff mechanism aims to defer transmission until the channel load decreases sufficiently to satisfy C < Tih. The goal is to minimize the difference between the actual channel load and the full-load threshold across all channels, while respecting constraints on backoff times for different priorities.
Formally, the optimization problem can be defined as:
$$
\min \sum_{l=1}^{N_f} \left| C_l – C_0 \right|
$$
Subject to:
$$
C1: T^0_b < T^1_b < \dots < T^{p-1}_b
$$
$$
C2: 0 \le T^i_b \le T_{b\_\max}^i, \quad 0 \le i < p
$$
Here, Cl is the load on channel l, C0 is the full-load threshold, and Tib is the backoff time for priority i. Constraint C1 ensures strict ordering of backoff times across priorities, reflecting the need for higher-priority traffic to wait less. Constraint C2 limits the backoff time within the maximum allowed lifetime Tb\_maxi for each priority class.
3. Existing Backoff Strategies and Limitations
3.1 Binary Exponential Backoff (BEB)
The BEB strategy is widely used in conventional MAC protocols. In BEB, each node maintains a contention window (CW) initially set to Wmin. Upon collision, the CW is doubled until reaching Wmax; upon successful transmission, it is reset to Wmin. The backoff time is uniformly chosen from [0, CW]. While simple, BEB is ill-suited for SPMA because SPMA’s channel access decision is based on load estimation rather than collision detection. In SPMA, backoff is triggered by load exceeding the threshold, not by collisions. Therefore, exponential window growth leads to overly long backoff times under high load and insufficient backoff under low load, degrading delay performance.
3.2 Logarithmic Backoff for SPMA
Several studies have proposed priority-aware backoff functions for SPMA. For example, a logarithmic model computes backoff time as:
$$
T_b = M \cdot \ln\left[ N \cdot (i+1) \cdot (C – T^i_h) \cdot C \right]
$$
where M and N are constant coefficients. This approach considers priority, channel load, and threshold, but the deterministic function lacks adaptability. The logarithmic model is derived from convexity arguments but does not guarantee optimal performance under dynamic conditions. Furthermore, the coefficients M and N depend on network parameters and are difficult to tune in practice.
3.3 Limitations of Existing Approaches
Both BEB and logarithmic strategies rely on static mathematical models that cannot capture the complex, time-varying relationship between backoff time and network state. In China UAV drones, traffic bursts, node mobility, and channel fading cause abrupt changes in load, making static backoff models suboptimal. There is a clear need for an adaptive backoff strategy that can learn from the environment and make decisions that maximize long-term performance.
4. Proposed DDQN-Based Intelligent Backoff Strategy
4.1 Overview
We formulate the backoff decision as a Markov decision process (MDP) and solve it using the Double Deep Q-Network (DDQN) algorithm, which mitigates the overestimation bias of standard DQN. In our framework, each UAV node acts as an agent interacting with the network environment. The agent observes the current state, selects a backoff time (action), and receives a reward reflecting the outcome. Through experience replay and target network updates, the agent learns an optimal policy that maximizes cumulative discounted reward.
4.2 MDP Modeling for Backoff
The key components of the MDP are:
- Agent: Each UAV node in the China UAV drone network.
- Environment: The entire network, including all nodes, channel conditions, and traffic generation.
- State space: The state at time t is defined as st = { it, Ct, Ct-1, …, Ct-m }, where it is the priority of the current packet, and Ct is the current channel load. We include m historical load values to capture the trend. This is crucial because channel load exhibits temporal correlation.
- Action space: The action at corresponds to selecting a backoff time from a finite discrete set. In SPMA, each priority i has a distinct action subspace Ai containing Nia possible backoff durations (in units of slot time τ). The entire action space is the union of these subspaces.
- Reward function: The immediate reward rt after executing action at is designed to encourage successful backoffs (i.e., after backoff, the channel load is below the threshold) while minimizing idle channel time. It is defined as:
$$
r_t = \phi \cdot e^{\frac{C – T^i_h}{i+1}}
$$
where φ = 1 if the backoff is successful (i.e., after backoff, C < Tih) and φ = –1 otherwise. The exponential term penalizes large differences between the load and threshold, with the penalty scaled by priority (higher priority has smaller denominator, yielding larger reward for success). This reward structure incentivizes the agent to select backoff times that quickly bring the channel load below the threshold without excessive waiting.
The goal is to maximize the cumulative discounted reward:
$$
J(\theta) = \sum_{k=0}^{\infty} \gamma^k r_{t+k}
$$
where γ = 0.9 is the discount factor.
4.3 DDQN Algorithm Architecture
The DDQN employs two neural networks: a primary network (with parameters θ) and a target network (with parameters θ–). The primary network is used to select actions, while the target network estimates the value of the selected action. This double-network architecture reduces overestimation. The target Q-value is computed as:
$$
y_t = r_t + \gamma Q(s_{t+1}, \arg\max_{a_{t+1}} Q(s_{t+1}, a_{t+1}; \theta); \theta^-)
$$
The primary network is updated by minimizing the loss function:
$$
L(\theta) = \mathbb{E}_{(s,a,r,s’) \sim D} \left[ \left( y_t – Q(s_t, a_t; \theta) \right)^2 \right]
$$
The training process uses an experience replay buffer D of size 20000 and a mini-batch size of 256. The exploration rate ε decays from 1.0 to 0.1 over 5000 episodes to balance exploration and exploitation.
4.4 Implementation Details
The neural network architecture consists of an input layer with dimension (m+2), two hidden layers with 128 neurons each (ReLU activation), and an output layer with size equal to the number of actions (i.e., number of slots for the lowest-priority lifetime). The primary and target networks are structurally identical. The target network parameters are updated every 100 steps by copying the primary network weights.
The algorithm pseudocode is omitted for brevity, but the key steps are: initialize environment and networks; for each episode, generate traffic and observe channel load; at each backoff opportunity, select action using ε-greedy; execute backoff and observe new state and reward; store transition in replay buffer; sample mini-batch and update primary network parameters; periodically update target network.
4.5 Complexity Analysis
The training is performed offline on a ground station or edge server; the converged lightweight model is then deployed on the UAVs. During inference, the computational cost per decision is approximately (m+2)×Nh + Nh² + Nh×Nap-1 floating-point operations, which is in the microsecond range on embedded platforms like NVIDIA Jetson Nano, meeting SPMA’s real-time requirements.
5. Simulation Results and Analysis
5.1 Simulation Setup
We implement the SPMA protocol in OPNET Modeler 14.5 and the DDQN algorithm in Python. The main simulation parameters are summarized in Table 1.
| Parameter | Value |
|---|---|
| Transmission rate | 2 Mbit/s |
| Packet size | 1000 bits |
| Slot time τ | 20 μs |
| Packet arrival rate λ | [3400, 4100] packets/s |
| Number of priority levels p | 3 |
| Load measurement period Tw | 10 ms |
| Priority traffic ratio (0:1:2) | 1:2:3 |
| Packet lifetime (priority 0/1/2) | 10 ms / 20 ms / 100 ms |
| Number of nodes | [5, 55] |
| Discount factor γ | 0.9 |
| Replay buffer size D | 20000 |
| Mini-batch size B | 256 |
Three priority classes are defined: Priority 0 (command and control, high urgency, lifetime 10 ms), Priority 1 (control and payload data, lifetime 20 ms), and Priority 2 (sensing and status, lifetime 100 ms). The default node count is 15. We compare our proposed DDQN-based backoff strategy against BEB and a logarithmic backoff (LOG) strategy.
5.2 Transmission Delay
Figure 1 shows the average end-to-end transmission delay for each priority class as a function of network load (λ). For priority 0 and 1 packets (Figure 1a), all strategies exhibit low delays, but DDQN slightly outperforms BEB and LOG under high load. For priority 2 packets (Figure 1b), the differences are more pronounced. At λ = 4.1 Mbit/s, DDQN achieves a 33.3% reduction in delay compared to LOG, and about 40% improvement over BEB. This is because DDQN learns to select backoff times that align with the varying channel load, avoiding excessive waiting or premature transmission that leads to further backoffs.
| Total Load (Mbit/s) | BEB | LOG | DDQN |
|---|---|---|---|
| 3.4 | 0.45 | 0.43 | 0.40 |
| 3.6 | 0.60 | 0.55 | 0.50 |
| 3.8 | 1.20 | 1.10 | 0.95 |
| 4.0 | 3.80 | 3.50 | 2.60 |
| 4.1 | 8.50 | 7.20 | 4.80 |
5.3 Initial Backoff Success Rate
We measure the proportion of priority 2 packets that succeed in accessing the channel after the first backoff. As shown in Table 3, DDQN achieves 83% success rate on the first attempt, compared to 72% for LOG and 65% for BEB. Furthermore, 95% of priority 2 packets succeed within three backoff attempts under DDQN, versus 92% for LOG and 90% for BEB. This indicates that DDQN learns to choose backoff times that are more accurate, reducing the need for repetitive backoffs.
| Number of Backoffs | BEB (%) | LOG (%) | DDQN (%) |
|---|---|---|---|
| 1 | 65 | 72 | 83 |
| 2 | 20 | 16 | 10 |
| 3 | 5 | 4 | 2 |
| 4 or more | 10 | 8 | 5 |
5.4 Transmission Success Rate
Transmission success rate is defined as the ratio of packets successfully received by the destination within their lifetime to the total generated packets. For high-priority (0 and 1) packets, all strategies yield success rates above 95% across the load range. For priority 2, the differences are significant (Table 4). At λ = 4.1 Mbit/s, DDQN achieves 82% success rate, compared to 75% for LOG and 68% for BEB. The improvement stems from DDQN’s ability to avoid excessive delays that cause packets to expire.
| Total Load (Mbit/s) | BEB | LOG | DDQN |
|---|---|---|---|
| 3.4 | 99.5 | 99.6 | 99.8 |
| 3.6 | 98.0 | 98.5 | 99.0 |
| 3.8 | 92.0 | 94.0 | 96.5 |
| 4.0 | 78.0 | 82.0 | 88.0 |
| 4.1 | 68.0 | 75.0 | 82.0 |
5.5 Scalability with Number of Nodes
We evaluate network performance as the number of nodes varies from 5 to 55 at a fixed load of 3.8 Mbit/s. Table 5 shows the average delay for priority 1 packets. For 5–25 nodes, all strategies perform similarly. For 35–45 nodes, DDQN reduces delay by approximately 31% compared to LOG and 36% compared to BEB. At 55 nodes, network congestion becomes severe, and the advantage of DDQN diminishes, but it still outperforms the others.
| Number of Nodes | BEB | LOG | DDQN |
|---|---|---|---|
| 5 | 0.12 | 0.11 | 0.10 |
| 15 | 0.15 | 0.14 | 0.13 |
| 25 | 0.20 | 0.19 | 0.18 |
| 35 | 0.60 | 0.55 | 0.40 |
| 45 | 1.50 | 1.30 | 0.95 |
| 55 | 3.20 | 2.80 | 2.50 |
Similarly, the transmission success rate for priority 1 degrades as node count increases, but DDQN maintains higher success rates (Table 6). These results confirm that the proposed DDQN strategy scales well with network size, making it suitable for medium-scale China UAV drone swarms.
| Number of Nodes | BEB | LOG | DDQN |
|---|---|---|---|
| 5 | 99.9 | 99.9 | 99.9 |
| 15 | 99.5 | 99.6 | 99.7 |
| 25 | 98.0 | 98.5 | 99.0 |
| 35 | 95.0 | 96.0 | 97.5 |
| 45 | 88.0 | 90.0 | 93.0 |
| 55 | 75.0 | 79.0 | 84.0 |
6. Conclusion
In this paper, we presented an intelligent backoff strategy for the SPMA protocol in China UAV drone ad hoc networks. By modeling the backoff decision as a Markov decision process and solving it with the DDQN algorithm, we overcome the limitations of static backoff models. The proposed strategy jointly considers service priority, threshold, and channel load, and learns to select optimal backoff times that minimize delay while maximizing transmission success. Simulation results demonstrate substantial improvements over traditional BEB and logarithmic backoff strategies: up to 33.3% reduction in delay for low-priority traffic, 18% higher initial backoff success rate, and better scalability as network size increases. These features make the DDQN-based backoff strategy a promising solution for mission-critical communications in dynamic environments such as those encountered by China UAV drones. Future work will explore joint optimization with transmission probability control and extension to multi-hop networks, as well as distributed multi-agent reinforcement learning frameworks to further enhance performance in large-scale deployments.
This work was supported in part by the National Postdoctoral Researcher Program and the National Natural Science Foundation of China.