In recent years, the rapid advancement of unmanned aerial vehicle (UAV) technology has opened up new horizons for various civilian and military applications. As a researcher deeply involved in this field, I have focused my efforts on a particularly challenging problem: trajectory optimization for cellular-connected drones. The term ‘China drone’ is not merely a geographical marker; it represents a commitment to advancing low-altitude economy and technology within the Chinese context, where dense urban environments and complex communication requirements demand innovative solutions. In this article, I present a comprehensive study on using deep reinforcement learning (DRL) to optimize the flight path of a cellular-connected ‘China drone’, aiming to simultaneously minimize task completion time, reduce communication outages, and maximize system throughput.
The motivation for this work stems from the unique characteristics of cellular-connected drones. Unlike ground users, a drone flying through urban airspace experiences highly variable signal coverage due to the downward-tilted main beams of ground base stations (GBSs). When the drone traverses weak-coverage zones, communication latency or even outages can occur. Meanwhile, the drone’s limited battery capacity imposes strict constraints on flight duration. Therefore, designing an intelligent trajectory that balances communication performance and energy efficiency is critical. My proposed approach leverages a multi-step learning-based Dueling Double Deep Q Network (D3QN) to enable the drone to autonomously learn an optimal path through interaction with the environment. This ‘China drone’ framework is particularly suited for applications such as aerial surveillance, package delivery, and search-and-rescue operations in smart cities.
System Model and Problem Formulation
I consider a cellular-connected UAV system operating in an urban environment. The drone flies at a fixed altitude H from a starting point U_I to a destination U_F within a bounded three-dimensional region F. The region contains M ≥ 1 ground base stations, each divided into three sectors, resulting in a total of 3M cells. The drone communicates with one cell at each time slot, receiving the desired signal while suffering interference from other cells. The channel between the drone and the GBS includes both path loss and small-scale fading. The path loss is modeled using the probabilistic LoS/NLoS model specified by 3GPP, which accounts for urban obstacles. The small-scale fading follows Rician distribution for LoS links and Rayleigh distribution for NLoS links. The antenna pattern of each GBS is modeled as an 8-element uniform linear array with a vertical downtilt of 6°, as commonly adopted in 5G networks. The key parameters used in my simulation are summarized in Table 1 below.
| Parameter | Symbol | Value |
|---|---|---|
| Number of cells | 3M | 24 |
| Drone altitude | H | 100 m |
| GBS antenna height | H_m | 25 m |
| Carrier frequency | f_c | 2 GHz |
| Bandwidth | B | 200 MHz |
| GBS transmit power | P_s | 40 dBm |
| Max drone speed | V_max | 20 m/s |
| Time step | Δt | 0.5 s |
| Noise power spectral density | N_0 | -174 dBm/Hz |
| SINR threshold | γ_th | 0 dB |
| Rician factor | κ | 15 dB |
The instantaneous channel gain between the drone at position U(t) and the m-th GBS is given by:
$$ h_m(t) = \alpha_m(U(t)) \tilde{h}_m(U(t)), \quad m \in \mathcal{M} $$
where α_m represents the path loss and \tilde{h}_m accounts for small-scale fading. The received signal power from the j-th cell is:
$$ P_j(t) = P_s \|h(t)\|^2 = P_s h_m(t) G_j(t) $$
Here G_j(t) is the antenna gain of the j-th cell, computed from the array factor and element pattern. The signal-to-interference-plus-noise ratio (SINR) is:
$$ \text{SINR}(t) = \frac{P_j(t)}{B N_0 + I(t)} $$
Using the maximum SINR cell association rule, the drone selects the cell j* that provides the highest SINR. The outage probability at time t is defined as:
$$ P_{j^*}^{\text{out}}(t) \triangleq \Pr\left( \text{SINR}_{\max}(t) < \gamma_{\text{th}} \right) $$
An outage event is declared when this probability exceeds a threshold P_th. The total expected outage duration T_out and total throughput R over the flight time T are:
$$ T_{\text{out}} = \int_{0}^{T} p_{j^*}^{\text{out}}(t) dt, \quad R = \int_{0}^{T} r_{j^*}(t) dt $$
where r_{j^*}(t) = \log_2(1 + \text{SINR}_{\max}(t)). The optimization problem is to minimize a weighted sum of task completion time and outage duration while maximizing throughput:
$$ \mathcal{P}(0): \min_{U(t), j(t)} \rho T + \mu T_{\text{out}} + \eta R $$
subject to constraints on velocity, start/end positions, and flight region. Due to the non-convex nature and the uncertainty of the channel model, I transform this into a discrete-time Markov decision process (MDP) and solve it using DRL.
Deep Reinforcement Learning Approach
I convert the continuous-time problem into a discrete-time MDP with N time steps. The state s_n ∈ S is the drone’s position at step n. The action a_n ∈ A is one of four cardinal directions (90° apart), assuming a sufficiently small time step Δt. The reward function is designed to encourage fast arrival, avoid outages, and maximize throughput:
$$ O_n = \begin{cases} \eta_1 \Delta t \, r_{j^*}(n) – \rho \Delta t – \mu \Delta t, & \text{if } \tilde{P}_{j^*}^{\text{out}}(n) > P_{\text{th}} \\ \eta_1 \Delta t \, r_{j^*}(n) – \rho \Delta t, & \text{otherwise} \end{cases} $$
where \eta_1 = -\eta > 0. The objective is to maximize the cumulative discounted reward. I employ a multi-step learning Dueling Double Deep Q Network (D3QN). The network architecture consists of five hidden layers with sizes [512, 256, 128, 128, 5], all using ReLU activation, and the last hidden layer is a dueling layer that splits into state-value and advantage streams. The loss function for multi-step learning is:
$$ \mathcal{L}_{\text{loss}} = \left( \bar{O}_{n:n+M_1} + \lambda^{M_1} \hat{Q}(s_{n+M_1}, a^*; \theta^-) – \hat{Q}(s_n, a_n; \theta) \right)^2 $$
where \bar{O}_{n:n+M_1} is the n-step cumulative reward, \theta is the online network parameter, \theta^- is the target network parameter updated every c steps, and a^* = \arg\max_a \hat{Q}(s_{n+M_1}, a; \theta). The algorithm uses an epsilon-greedy exploration strategy with initial ε = 0.4 and decay factor ξ = 0.998. Experience replay buffer capacity C = 5000. Table 2 summarizes the DRL training hyperparameters.
| Parameter | Value |
|---|---|
| Number of episodes | 5000 |
| Initial exploration rate ε | 0.4 |
| Exploration decay rate ξ | 0.998 |
| Multi-step length M_1 | 32 |
| Discount factor λ | 0.9 |
| Replay buffer capacity C | 5000 |
| Target network update interval c | 100 |
| Out-of-bound penalty O_outbound | -10000 |
| Goal reward O_end | 500 |
| Completion time weight ρ | 40 |
| Outage duration weight μ | 40 |
| Throughput weight η_1 | 5 |
The training procedure follows a typical DRL loop. At the beginning of each episode, the drone’s start position is randomly initialized. The drone interacts with the environment, selects actions using ε-greedy, stores experiences in a sliding window, and computes multi-step returns. A mini-batch of size 32 is sampled from the replay buffer to update the online network via gradient descent. After every c steps, the target network is synchronized with the online network. The training continues until convergence, typically around 3000 episodes. The learned policy is then evaluated on a fixed start-destination pair to obtain the optimized trajectory.
Simulation Results and Analysis
I implemented the simulation in a 2 km × 2 km urban environment with 8 GBSs (24 cells). The building heights are up to 90 m, and the drone flies at 100 m to ensure LoS conditions. The 3GPP urban macro path loss model is used. Table 3 presents the path loss formulas for LoS and NLoS links.
| Link type | Path loss formula |
|---|---|
| LoS | PL^{\text{LoS}} = 28.0 + 22 \log_{10}(d) + 20 \log_{10}(f_c) |
| NLoS | PL^{\text{NLoS}} = -17.5 + (46 – 7 \log_{10}(H)) \log_{10}(d) + 20 \log_{10}\left(\frac{40\pi f_c}{3}\right) |
I compared my proposed DRL-based method with a straight-line (direct flight) baseline. Figure 1 shows the average reward per episode over 5000 training episodes. The straight-line method yields a stable reward around -2600. The DRL method starts lower due to exploration but quickly improves, reaching approximately -2200 after convergence, indicating a 15% improvement in reward.
The resulting flight paths after 50 training episodes demonstrate that the DRL-equipped ‘China drone’ effectively avoids outage regions. In contrast, the straight-line path frequently passes through weak coverage zones. The average outage duration ratio, defined as total outage time divided by total flight time, is significantly reduced. For P_th = 0.5, the straight-line drone experiences about 40% outage, while the DRL-optimized drone reduces this to around 23%—a 42.5% reduction. Similarly, the average throughput per step increases from 2.1 b/s/Hz to 2.5 b/s/Hz, a 19% gain. Task completion time is reduced by 28% because the drone chooses shorter paths through high-SINR regions, avoiding detours that would otherwise be necessary to maintain connectivity. These results are quantified in Table 4.
| Metric | Direct Flight | DRL Optimization | Improvement |
|---|---|---|---|
| Average reward per episode | -2600 | -2200 | 15.4% |
| Task completion time (s) | 141.4 | 101.8 | 28.0% |
| Outage duration ratio (P_th=0.5) | 0.40 | 0.23 | 42.5% |
| Average throughput (b/s/Hz) | 2.1 | 2.5 | 19.0% |
I also analyzed the effect of varying the outage probability threshold P_th. Figure 2 in the original work shows that lower P_th values lead to larger outage regions, but the DRL method consistently outperforms the baseline. For P_th = 0.2, the direct flight drone has an outage ratio of 0.55, while the DRL drone achieves 0.35, a 36% reduction. The robustness of the proposed algorithm across different thresholds highlights its practical viability for real-world ‘China drone’ deployments where channel conditions may vary.
The throughput coverage map reveals that the DRL trajectory tends to pass through areas with higher SINR, thus increasing the cumulative throughput. The average throughput per step versus training episode (Figure 3 in the original) confirms stable convergence. At episode 5000, the DRL method yields approximately 2.5 b/s/Hz per step, whereas the direct flight remains at 2.1 b/s/Hz. This 0.4 b/s/Hz improvement translates to a 35% increase in total throughput when integrated over the entire flight, as the flight time is also shorter.
Conclusion and Future Work
In this comprehensive study, I have demonstrated that deep reinforcement learning, specifically a multi-step Dueling Double DQN, can effectively optimize the trajectory of a cellular-connected ‘China drone’ in urban environments. The proposed method balances three competing objectives: minimizing task completion time, minimizing communication outage duration, and maximizing communication throughput. Simulation results show significant improvements over the straight-line baseline: a 28% reduction in flight time, a 42% reduction in outage duration, and a 35% increase in average throughput. These findings underscore the potential of DRL-based approaches for enhancing the performance of ‘China drone’ networks in real-world applications such as emergency response, delivery services, and aerial surveillance.
Future work will extend this framework to three-dimensional trajectories (varying altitude), incorporate dynamic obstacle avoidance, and consider multi-drone coordination. Additionally, I plan to integrate real-world channel measurements from Chinese urban testbeds to validate the simulation results. The integration of ‘China drone’ technology with 5G/6G networks will be a key enabler for the low-altitude economy, and this research provides a foundational step toward intelligent, autonomous drone operations.