The rapid proliferation of Unmanned Aerial Vehicles (UAVs), or drones, across diverse sectors such as autonomous navigation, surveillance, and disaster response has necessitated the development of highly reliable and efficient path planning algorithms. The core objective of UAV path planning is to identify an optimal trajectory from a start point to a goal that satisfies various constraints, including obstacle avoidance, path smoothness, and operational efficiency. Traditional algorithms, including A*, Dijkstra, and sampling-based methods like Rapidly-exploring Random Trees (RRT), often face challenges in complex, dynamic environments due to computational intensity or lack of adaptability. Similarly, intelligent optimization algorithms like Genetic Algorithms or Particle Swarm Optimization can suffer from premature convergence or high computational cost.
Deep Reinforcement Learning (DRL) has emerged as a powerful paradigm for tackling sequential decision-making problems like UAV path planning. By enabling a UAV drone agent to learn optimal navigation policies through direct interaction with a simulated environment, DRL offers the potential for adaptive and robust planning. The Deep Q-Network (DQN) algorithm represents a foundational milestone in DRL, successfully combining Q-learning with deep neural networks to handle high-dimensional state spaces. The goal of the DQN agent is to learn an action-value function $Q^*(s,a)$ that estimates the maximum expected cumulative reward:
$$ Q^*(s, a) = \max_{\pi} \mathbb{E} \left[ \sum_{t=0}^{\infty} \gamma^t r_t | s_0 = s, a_0 = a, \pi \right] $$
Here, $s$ and $a$ represent the state and action, $r_t$ is the reward at time $t$, $\gamma$ is the discount factor, and $\pi$ is the policy. A neural network parameterized by $\theta$, denoted $Q(s,a;\theta)$, is used to approximate this function. The network is trained by minimizing the mean-squared Temporal Difference (TD) error loss:
$$ L(\theta) = \mathbb{E}_{(s,a,r,s’) \sim D} \left[ \left( y – Q(s, a; \theta) \right)^2 \right] $$
where the target $y$ is computed as $y = r + \gamma \max_{a’} Q(s’, a’; \theta^-)$ using a separate target network with parameters $\theta^-$.
However, applying standard DQN directly to UAV drone path planning presents several significant challenges:
- Sparse and Ineffective Rewards: Using only terminal rewards (e.g., large positive reward for reaching the goal, large negative reward for collision) leads to extremely slow learning as the UAV drone receives little guidance during the long exploration phase.
- Inefficient Exploration: The standard $\epsilon$-greedy exploration strategy performs many random, undirected actions, resulting in a vast number of ineffective trials, such as repeated collisions or movements away from the target.
- Poor Path Quality: Even when a path is found, it may be suboptimal, characterized by excessive length, numerous sharp turns (waypoints), and unsafe proximity to obstacles.
- Slow Convergence: Uniform sampling from the experience replay buffer does not prioritize more informative transitions, slowing down the learning process.
To overcome these limitations, this article presents a comprehensive improved DQN framework, termed RNDQN, specifically designed for efficient and high-quality UAV drone three-dimensional path planning.
1. The RNDQN Algorithmic Framework
The proposed RNDQN framework introduces two key modifications to the standard DQN architecture: Prioritized Experience Replay (PER) and a parameter-space Noise injection mechanism. Furthermore, it is driven by a meticulously designed dense reward function inspired by the Artificial Potential Field (APF) method.
1.1 Prioritized Experience Replay (PER)
Uniform sampling from the replay buffer treats all transitions equally, regardless of their potential learning value. Prioritized Experience Replay addresses this by sampling transitions with a probability relative to their TD error, a measure of how “surprising” or corrective the transition is for the current network. The TD-error for transition $i$ is:
$$ \delta_i = | r_i + \gamma \cdot \max_{a’} Q(s’_i, a’; \theta^-) – Q(s_i, a_i; \theta) | $$
The priority $p_i$ of a transition is then calculated as:
$$ p_i = (|\delta_i| + \epsilon)^\alpha $$
where $\epsilon$ is a small positive constant to ensure non-zero priority, and $\alpha$ determines the prioritization strength ($\alpha=0$ reverts to uniform sampling). The sampling probability $P(i)$ for transition $i$ in a buffer of size $N$ is:
$$ P(i) = \frac{p_i}{\sum_{k=1}^{N} p_k} $$
This mechanism ensures that transitions from which the UAV drone agent can learn more—such as those involving collisions, near-misses, or significant progress—are replayed more frequently, dramatically accelerating convergence.
1.2 Noisy Networks for Exploration
To replace the undirected exploration of $\epsilon$-greedy, Noisy Networks inject noise directly into the parameters of the neural network. This induces structured, state-dependent exploration where the amount of exploration is learned and adjusted. In a standard linear layer with input $x$, weight $W$, and bias $b$, the output is $y = Wx + b$. In a noisy layer, the weights and biases are perturbed:
$$ W = \mu^W + \sigma^W \odot \varepsilon^W, \quad b = \mu^b + \sigma^b \odot \varepsilon^b $$
Here, $\mu^W$ and $\mu^b$ are learnable mean parameters, $\sigma^W$ and $\sigma^b$ are learnable standard deviation parameters, $\varepsilon^W$ and $\varepsilon^b$ are noise variables sampled from a Gaussian distribution, and $\odot$ denotes element-wise multiplication. The output becomes:
$$ y = (\mu^W + \sigma^W \odot \varepsilon^W) x + (\mu^b + \sigma^b \odot \varepsilon^b) $$
This approach allows the UAV drone agent to explore action values in a more consistent and efficient manner across similar states, significantly reducing the number of purely random, counterproductive actions.
1.3 Action and State Space Design
The environment is discretized into a 3D grid. The UAV drone’s action space $A$ is defined by 26 discrete movements, allowing motion to any adjacent grid cell, including diagonals in three dimensions. This provides fine-grained control for the UAV drone.
| Dimension | Possible Values | Description |
|---|---|---|
| $dx, dy, dz$ | -1, 0, 1 | Movement component in x, y, z direction |
| Excluding (0,0,0) | Null movement is not allowed |
The state $S_t$ provided to the UAV drone agent at time $t$ is designed to encapsulate all necessary information for decision-making:
$$ S_t = (P^{drone}_t, P^{target}_t, O_t) $$
where $P^{drone}_t$ is the UAV drone’s current 3D coordinate $(x_t, y_t, z_t)$, $P^{target}_t$ is the goal’s 3D coordinate $(x_g, y_g, z_g)$, and $O_t$ is a perceptual vector encoding the presence of obstacles within the drone’s sensor range.
2. Composite Dense Reward Function
The reward function is the critical guide for the UAV drone’s learning process. A purely sparse reward ($R_{goal}$ on success, $R_{fail}$ on collision) is insufficient. The proposed composite function provides dense, step-by-step guidance by combining APF-inspired forces with penalties for inefficiency.
| Component | Symbol | Purpose | Key Parameters |
|---|---|---|---|
| Attractive Reward | $R_{att}$ | Guide drone towards target | Distance $\rho$, distance change $\Delta d$ |
| Repulsive Penalty | $R_{rep}$ | Penalize proximity to obstacles | Obstacle distance $d_i$, repulsive coefficient |
| Time Penalty | $R_{time}$ | Encourage shorter paths | Current step $t$, max steps $T$ |
| Direction Penalty | $R_{dir}$ | Encourage stable heading & reduce turns | Angle $\theta$ between consecutive moves |
| Goal Reward | $R_{goal}$ | Signal task completion | Constant large value |
| Failure Penalty | $R_{fail}$ | Signal collision/boundary violation | Constant large negative value |
The total reward for the UAV drone at each step is the sum: $R_{total} = R_{goal} + R_{fail} + R_{att} + R_{rep} + R_{time} + R_{dir}$.
2.1 Attractive Reward ($R_{att}$)
This component creates a virtual “attractive force” pulling the UAV drone toward the goal. The reward is proportional to the reduction in distance to the target $\Delta d = \rho_{t-1} – \rho_t$, where $\rho$ is the Euclidean distance. A dynamic scaling factor makes the guidance stronger when the UAV drone is closer to the goal, addressing reward sparsity at long ranges.
$$
R_{att} =
\begin{cases}
15 \cdot \Delta d & \text{if } \rho(q, q_{goal}) < 5 \\
10 \cdot \Delta d & \text{if } 5 \leq \rho(q, q_{goal}) < 20 \\
5 \cdot \Delta d & \text{otherwise}
\end{cases}
$$
2.2 Repulsive Penalty ($R_{rep}$)
Inspired by APF’s repulsive force, this penalty discourages the UAV drone from flying too close to obstacles, ensuring a safety margin. It is computed for all obstacles $i$ within a perception range $d_{rep}$.
$$
R_{rep} = \sum_{i=1}^{N} \begin{cases}
-\frac{10}{d_i^2} & \text{if } d_i \leq d_{rep} \\
0 & \text{otherwise}
\end{cases}
$$
Here, $d_i$ is the shortest distance from the UAV drone to the $i$-th obstacle. The inverse-square relationship creates a steep penalty field near obstacles, effectively teaching the drone to avoid them proactively.
2.3 Time and Direction Penalties ($R_{time}$, $R_{dir}$)
To promote efficiency and smoothness, two additional penalties are introduced. The time penalty encourages the UAV drone to find shorter paths by applying a cost for every step taken. It increases progressively to prevent excessively long episodes:
$$ R_{time} = – \left( \beta_1 + \frac{\beta_2 \cdot t}{T} \right) $$
where $\beta_1$ and $\beta_2$ are constants, $t$ is the current step, and $T$ is the maximum allowed steps per episode.
The direction penalty discourages sharp turns, which are inefficient for a real UAV drone and increase wear. It penalizes turns where the angle $\theta$ between the current and previous movement direction exceeds 90 degrees:
$$
R_{dir} = \begin{cases}
-5 \cdot |\cos \theta| & \text{if } \cos \theta < 0 \\
0 & \text{otherwise}
\end{cases}
$$
3. Experimental Analysis and Results
The proposed RNDQN algorithm was implemented and evaluated in a custom 3D grid environment built using the Gym toolkit. The environment contained multiple cubic obstacles of varying sizes and heights to simulate a complex urban or industrial scenario. The UAV drone’s mission was to navigate from a fixed start point to a fixed goal.
3.1 Hyperparameter and Environment Configuration
| Hyperparameter | Symbol | Value |
|---|---|---|
| Learning Rate | $\alpha$ | 0.001 |
| Discount Factor | $\gamma$ | 0.995 |
| PER Exponent | $\alpha_{PER}$ | 0.6 |
| Noise Standard Deviation | $\sigma$ | 0.7 |
| Max Steps per Episode | $T$ | 500 |
| Time Penalty Constants | $\beta_1, \beta_2$ | 0.3, 10 |
The neural network for the UAV drone agent consisted of three fully connected layers (128, 256, 256 neurons) with ReLU activation, followed by a noisy linear output layer.
3.2 Training Performance: RNDQN vs. Baseline DQN
The training curves clearly demonstrate the superiority of the RNDQN framework. The baseline DQN (using the same composite reward but with uniform replay and $\epsilon$-greedy) exhibited high volatility and slower convergence, only stabilizing after approximately 400 training episodes. In contrast, the RNDQN algorithm converged significantly faster and more smoothly, reaching a stable, high-reward policy around 200 episodes. The reduced volatility is attributed to the more efficient, directed exploration from noisy networks, while the faster convergence stems from the prioritized replay focusing learning on the most critical experiences for the UAV drone.
3.3 Path Planning Performance Comparison
The trained models were compared against a standard sampling-based planner, the RRT algorithm, in a fixed scenario. Quantitative metrics were gathered over 30 independent runs with randomized obstacle seeds and start positions within a region.
| Algorithm | Average Path Length | Average Number of Waypoints | Average Planning Time (s) | Success Rate |
|---|---|---|---|---|
| RRT | 44.16 ± 1.58 | 24 ± 1.3 | 0.42 | 100% |
| DQN (with composite reward) | 42.48 ± 1.46 | 22 ± 1.2 | 0.25 | 100% |
| RNDQN (Proposed) | 37.72 ± 1.32 | 17 ± 1.2 | 0.22 | 100% |
The results are conclusive. The proposed RNDQN algorithm outperforms both the baseline DQN and the RRT algorithm across all key metrics for UAV drone path quality. Specifically:
- Path Length: RNDQN shortened the path by 11.2% compared to DQN and by 14.6% compared to RRT.
- Path Smoothness (Waypoints): The number of turns/waypoints was reduced by 22.7% vs. DQN and 29.1% vs. RRT, indicating significantly smoother trajectories more suitable for real UAV drone flight.
- Computational Efficiency: Once trained, the RNDQN policy generates paths nearly instantaneously (0.22s), faster than both the baseline DQN inference and the online computation of RRT.
Visually, the paths generated by RNDQN are more direct, maintain a safer distance from obstacles due to the repulsive penalty, and exhibit fewer unnecessary zigzags compared to the paths from DQN and RRT.
4. Conclusion and Future Work
This work presents a robust and effective framework for UAV drone three-dimensional path planning based on enhanced deep reinforcement learning. By integrating Prioritized Experience Replay and Noisy Networks into the DQN architecture, the RNDQN algorithm achieves markedly improved learning efficiency and stability. The composite reward function, fusing concepts from Artificial Potential Fields with penalties for time and erratic heading, provides the UAV drone agent with dense, actionable guidance throughout its learning and operational phases. Empirical results validate that the proposed method generates shorter, smoother, and safer flight paths compared to a standard DQN baseline and a classical RRT planner, fulfilling core requirements for autonomous UAV drone navigation.
Future research directions include extending the framework to dynamic environments where obstacles move, integrating wind and other atmospheric disturbances into the model, and scaling the approach to multi-UAV drone cooperative path planning scenarios. Furthermore, investigating more advanced actor-critic DRL algorithms within this improved framework could yield additional performance gains for complex UAV drone missions.