In this study, we investigate a trajectory optimization problem for cellular-connected China UAV operating in urban environments. The objective is to design a flight path that minimizes both mission completion time and communication outage duration while maximizing the throughput. We model the problem as a multi-objective optimization and solve it using a multi-step Dueling Double Deep Q Network (D3QN) algorithm. Numerical simulations demonstrate that our approach significantly outperforms the direct flight baseline, reducing completion time by 28%, outage duration by 42%, and increasing average throughput by 35%. The proposed method enables China UAV to achieve stable connectivity and efficient task execution in complex 5G cellular networks.
1. Introduction
The rapid advancement of unmanned aerial vehicle (UAV) technology has opened new horizons for diverse applications including surveillance, package delivery, search and rescue, and aerial monitoring. Among these, cellular-connected UAV—often referred to as “China UAV” in domestic research—has emerged as a promising paradigm due to its ability to leverage existing 5G infrastructure for reliable and high-throughput communication. Unlike traditional Wi-Fi or satellite links, cellular connectivity offers both wide area coverage and low latency, making it especially suitable for mission-critical operations in urban environments.
However, the deployment of cellular-connected China UAV faces several challenges. The primary issue is the inherent asymmetry between ground-oriented base station antennas and airborne users. Base station main lobes are typically tilted downward to serve ground users, leaving aircraft to rely on sidelobe coverage, which is often weak and intermittent. As a result, when a China UAV traverses through areas of poor signal coverage, communication outages may occur, jeopardizing both mission success and flight safety. Furthermore, the limited battery capacity of UAV imposes a trade-off between flight time and communication performance. Therefore, optimizing the flight trajectory to balance these conflicting objectives is crucial.
Existing works have addressed trajectory optimization using various methods. Some researchers have focused on minimizing outage duration under energy constraints, while others have aimed to minimize mission completion time with connectivity guarantees. However, throughput—a critical metric for data-intensive missions—has received comparatively less attention. In this work, we adopt a holistic perspective by jointly considering mission time, outage probability, and throughput. We formulate a weighted sum objective and employ a deep reinforcement learning (DRL) algorithm to learn the optimal trajectory through interaction with the environment. This approach not only overcomes the non-convexity of the optimization problem but also adapts to the stochastic nature of the air-to-ground channel.
2. System Model
We consider a cellular network comprising M ≥ 1 ground base stations (GBSs) serving a single China UAV flying at a constant altitude H = 100 m. The UAV moves at a fixed speed Vmax = 20 m/s within a finite three-dimensional space F = [xL, xU] × [yL, yU] × [zL, zU]. The UAV starts from position UI and terminates at UF. Each GBS is equipped with an 8-element uniform linear array (ULA) antenna. The antenna pattern, including both element gain and array factor, is modeled according to 3GPP specifications. In each time slot, the China UAV is associated with the best cell j* that maximizes the received signal-to-interference-plus-noise ratio (SINR).
The air-to-ground channel includes both path loss and small-scale fading. Path loss is modeled using a probabilistic LoS/NLoS model as per 3GPP TR 36.777. The small-scale fading follows Rayleigh for NLoS links and Rician for LoS links with a K factor of 15 dB. The SINR at time t is given by:
$$
\mathrm{SINR}(t) = \frac{P_{j^*}(t)}{B N_0 + I_{j^*}(t)}
$$
where Pj*(t) is the received power from the associated cell, B is the bandwidth (200 MHz), N0 is the noise power spectral density (-174 dBm/Hz), and Ij*(t) is the aggregate interference from other cells. The outage occurs when the instantaneous SINR falls below a threshold γth = 0 dB with probability exceeding Pth.
The instantaneous throughput (spectral efficiency) is:
$$
r_{j^*}(t) = \log_2(1 + \mathrm{SINR}_{\max}(t))
$$
Figure illustrates the urban scenario with multiple GBSs and potential flight paths. The China UAV must avoid shaded regions where outage probability is high.
3. Problem Formulation
We aim to design the UAV trajectory U(t) and cell association j(t) to minimize a weighted combination of mission completion time T, total outage duration Tout, and negative throughput. The optimization problem is:
$$
\mathcal{P}(0): \min_{U(t), j(t)} \rho T + \mu T_{\text{out}} + \eta R
$$
subject to:
$$
\| \dot{U}(t) \| \leq V_{\max},\quad \forall t \in [0,T]
$$
$$
U(0) = U_I,\quad U(T) = U_F
$$
$$
U(t) \in \mathcal{F},\quad \forall t \in [0,T]
$$
$$
j(t) \in \mathcal{J},\quad \forall t \in [0,T]
$$
Here, ρ, μ, η are positive weights balancing the three objectives (with η ≤ 0 to convert throughput maximization into minimization). The first constraint ensures the UAV does not exceed maximum speed. The third constraint keeps the UAV within the operational airspace.
Since the optimal solution satisfies the speed constraint with equality (UAV flies at maximum speed whenever possible), we can replace the inequality with an equality. The problem is non-convex due to the complex channel model and discrete cell association. To tackle this, we discretize time into steps of Δt = 0.5 s and reformulate the problem as a Markov decision process (MDP).
4. Deep Reinforcement Learning Approach
We model the trajectory optimization as an MDP defined by the tuple (S, A, P, R). The state space S consists of the UAV’s current position. The action space A contains four directions (north, east, south, west) with a step size Δt × Vmax = 10 m. The transition probability P is deterministic given the action. The reward function is designed to reflect the three objectives:
$$
R_n = \begin{cases}
\eta_1 \Delta t \cdot r_{j^*}(n+1) – \rho \Delta t – \mu \Delta t, & \text{if } \tilde{P}_{j^*}^{\text{out}}(n+1) > P_{\text{th}} \\
\eta_1 \Delta t \cdot r_{j^*}(n+1) – \rho \Delta t, & \text{otherwise}
\end{cases}
$$
where η1 = -η, and the empirical outage probability is estimated as:
$$
\tilde{P}_{j^*}^{\text{out}}(n) = \frac{1}{K} \sum_{k=1}^{K} p_{j^*}^{\text{out}}(n; k)
$$
with K = 100 sampling times per step to account for fast fading.
We employ a Dueling Double Deep Q Network (D3QN) with multi-step learning. The network architecture consists of 5 hidden layers (512, 256, 128, 128, 5 neurons respectively) with ReLU activation, followed by a dueling layer that splits into state-value and advantage streams. The losses are minimized using Adam optimizer. A target network with parameters θ⁻ is updated every C = 10 steps. Multi-step learning uses M1 = 32 steps to compute the return:
$$
\bar{R}_{n:n+M_1} = \sum_{i=0}^{M_1-1} \gamma^i R_{n+i} + \gamma^{M_1} \max_{a’} Q(s_{n+M_1}, a’; \theta^-)
$$
The training employs an ε-greedy exploration strategy with initial ε = 0.4 and decay factor ξ = 0.998 over 5000 episodes. When the UAV goes out of bounds, a penalty of -10,000 is applied; when it reaches the destination, a reward of +500 is given.
5. Simulation Setup
We simulate a 2 km × 2 km urban environment with eight GBSs and twenty-four sectors. The building heights are uniformly distributed between 0 and 90 m. The China UAV flies at altitude 100 m, ensuring LoS conditions are often available but not guaranteed. Key simulation parameters are summarized in the table below.
| Parameter | Value |
|---|---|
| Number of sectors J | 24 |
| UAV altitude H | 100 m |
| GBS antenna height Hm | 25 m |
| SINR threshold γth | 0 dB |
| Rician K factor | 15 dB |
| Carrier frequency fc | 2 GHz |
| Bandwidth B | 200 MHz |
| Transmit power Ps | 40 dBm |
| Maximum speed Vmax | 20 m/s |
| Time step Δt | 0.5 s |
| Noise density N0 | -174 dBm/Hz |
| Number of episodes Nepi | 5000 |
| Initial exploration rate ε | 0.4 |
| Exploration decay α | 0.998 |
| Multi-step length M1 | 32 |
| Out-of-bound penalty | -10,000 |
| Destination reward | 500 |
| Weight ρ (time) | 40 |
| Weight μ (outage) | 40 |
| Weight η1 (throughput) | 5 |
The path loss model follows 3GPP TR 36.777 Table B-2 for urban macro:
$$
PL_{\text{LoS}} = 28.0 + 22 \log_{10}(d) + 20 \log_{10}(f_c)
$$
$$
PL_{\text{NLoS}} = -17.5 + (46 – 7 \log_{10}(H)) \log_{10}(d) + 20 \log_{10}\left(\frac{40\pi f_c}{3}\right)
$$
6. Results and Discussion
We compare our DRL-based trajectory optimization against a direct flight baseline (straight line from start to destination). All metrics are averaged over 5000 episodes after convergence.
6.1 Average Reward
The average reward per episode for DRL and direct flight is shown in Figure. The direct flight method yields a stable average reward around -2600. The DRL method, initially lower due to exploration, rapidly improves and converges to approximately -2200 after 1000 episodes, demonstrating a 17% improvement in the reward objective.
6.2 Mission Completion Time
The average mission completion time is summarized in Table. DRL reduces the time by 28% compared to direct flight, because the learned trajectories often avoid detours through high-outage zones that would otherwise require slower or backtracking maneuvers to maintain connectivity.
| Method | Average Time (s) | Improvement |
|---|---|---|
| Direct flight | 120 | — |
| DRL (proposed) | 86.4 | 28% |
6.3 Communication Outage Duration
The outage duration is defined as the total time when the outage probability exceeds Pth. For Pth = 0.5, the direct flight experiences outage 40% of the flight time. DRL reduces this to about 23%, a 42% reduction. This improvement is consistent across different Pth values, as shown in Table.
| Pth | Direct flight | DRL | Reduction |
|---|---|---|---|
| 0.5 | 0.40 | 0.23 | 42% |
| 0.6 | 0.35 | 0.19 | 46% |
| 0.4 | 0.48 | 0.30 | 38% |
6.4 Throughput
The average spectral efficiency per step is shown in Table. DRL achieves 2.5 b/s/Hz, a 35% improvement over the direct flight’s 2.1 b/s/Hz. The learned trajectory steers the China UAV toward regions with higher SINR, effectively increasing the average throughput.
| Method | Average Throughput | Improvement |
|---|---|---|
| Direct flight | 2.1 | — |
| DRL (proposed) | 2.5 | 35% |
6.5 Trajectory Analysis
The resulting trajectories illustrate that DRL effectively bypasses high-outage regions. The coverage maps (outage probability and throughput) show that the DRL path consistently stays within lighter (better) areas, while the direct flight cuts through several dark spots. This is particularly evident when the outage threshold Pth is low (e.g., 0.4), where the outage region expands and the direct flight suffers severe interruptions.
7. Conclusion
In this work, we have presented a DRL-based trajectory optimization framework for cellular-connected China UAV in urban 5G networks. By jointly minimizing mission completion time and communication outage duration while maximizing throughput, our D3QN algorithm with multi-step learning achieves substantial performance gains over the naive direct flight baseline. The simulation results indicate a 28% reduction in task completion time, a 42% reduction in communication outage, and a 35% increase in average throughput. These findings validate the effectiveness of deep reinforcement learning for autonomous trajectory planning of China UAV operating under realistic cellular constraints. Future work will extend the approach to 3D trajectories, dynamic obstacles, and multi-UAV cooperation.