The rapid pace of urbanization has led to the typical modern cityscape characterized by dense high-rise buildings and frequent traffic congestion. This environment presents significant and escalating challenges for traditional firefighting efforts, particularly in densely populated districts, older urban areas, and commercial zones where narrow streets and haphazardly parked vehicles often block access for fire trucks. These delays can be catastrophic, especially in high-rise building fires where every second counts. The core challenges of urban fire rescue can be summarized as: exceeding the operational height limits of ladder trucks for super-tall buildings, physical access restrictions for firefighting vehicles in confined spaces, and compromised response times due to complex traffic environments.
In this context, fire UAVs (Unmanned Aerial Vehicles) have emerged as a transformative tool for aerial firefighting support. Their inherent flexibility, rapid response capability, and ability to navigate over ground-based obstacles make them ideally suited to bypass traffic jams and reach fire scenes quickly, potentially carrying extinguishing agents like dry powder or water-based firefighting capsules. Trajectory planning is the cornerstone technology enabling the autonomous and efficient operation of a fire UAV in complex urban environments. It is the core algorithm that determines a safe, feasible, and optimal flight path from a launch point (e.g., a fire truck staging area) to the target fire location. The choice of planning algorithm directly impacts flight efficiency, safety, and energy consumption, which are critical for successful rescue missions.
Numerous algorithms have been applied to UAV path planning, including Ant Colony Optimization (ACO), Rapidly-exploring Random Trees (RRT), Particle Swarm Optimization (PSO), Genetic Algorithms (GA), and Sparrow Search Algorithm (SSA). While effective in various scenarios, these methods exhibit limitations when applied to the specific, urgent, and dynamically constrained context of urban fire rescue. Traditional optimization algorithms like ACO and GA often rely on global environmental knowledge and suffer from high computational complexity, making them less suitable for real-time response requirements. Although RRT can adapt to complex obstacle fields, the generated paths are often non-optimal and jagged, compromising flight stability and safety. Algorithms like PSO and SSA typically require complete re-planning when encountering unexpected dynamic obstacles, lacking the necessary timeliness.
In contrast, Deep Reinforcement Learning (DRL) algorithms demonstrate superior suitability due to their powerful online learning and adaptive decision-making capabilities. These algorithms can continuously optimize their policy through interaction with a simulated or real environment, without relying on a complete prior map, and can effectively handle unexpected obstacles and situational changes during a mission. Recent advancements have seen DRL successfully applied to UAV path planning. However, existing research often focuses on general scenarios and lacks optimizations specifically designed for the critical demands of urban fire rescue. Furthermore, the design of reward functions in many studies fails to effectively balance the multi-objective optimization needs of minimizing flight time, ensuring obstacle avoidance, and reliably reaching the target.
This paper addresses the specific challenges of urban fire rescue by proposing a fire UAV trajectory planning algorithm based on an Improved Deep Q-Network (IDQN). The fire UAV is tasked with traversing dense urban building clusters to deliver extinguishing payloads to the fire location. Tailored to the environmental characteristics, mission requirements, and performance parameters of fire UAVs in high-rise rescue scenarios, the algorithm incorporates several key enhancements: the integration of a Prioritized Experience Replay mechanism to improve training efficiency; a redesigned state and action space appropriate for the 3D urban environment; and the construction of a novel composite reward function. This reward function is centered on an exponentially increasing time penalty, strategically combined with obstacle collision penalties and goal-reaching rewards to prioritize mission speed while ensuring safety.
1. Modeling the Fire UAV Trajectory Planning Problem
1.1 Problem Description
Fire UAVs can assist in high-rise firefighting primarily in two modes: vertical take-off from below the fire point, or launching from a fire truck staging area and navigating through urban building clusters. This study focuses on the latter. The core objective is to plan an optimal flight trajectory from the staging point \( \mathbf{r}_{\text{base}} = (x_0, y_0, z_0) \) to the fire point \( \mathbf{r}_{\text{goal}} = (x_g, y_g, z_g) \), passing through a sequence of waypoints \( \{\mathbf{r}_1, \mathbf{r}_2, …, \mathbf{r}_n\} \). This trajectory must strictly adhere to the UAV’s kinematic and operational constraints while absolutely guaranteeing safety and minimizing the total mission time \( T \). The following figure conceptually illustrates this planning scenario.
1.2 Constraint Conditions
The flight trajectory for a fire UAV in a rescue mission must satisfy a comprehensive set of constraints related to performance, safety, and operational limits:
$$
\begin{aligned}
& \|\dot{\mathbf{r}}(t)\| = \sqrt{\dot{x}(t)^2 + \dot{y}(t)^2} \leq V_{\text{max}} \\
& 0 \leq z(t) \leq h_{\text{max}} \\
& T \leq T_{\text{endurance}} \\
& \|\ddot{\mathbf{r}}(t)\| \leq a_{\text{max}} \\
& \|\mathbf{r}(t) – \mathbf{r}_{\text{base}}\| \leq R_{\text{operation}} \\
& \|\mathbf{r}(t) – \mathbf{o}_j\| \geq d_{\text{safe}}, \quad \forall j
\end{aligned}
$$
where \( V_{\text{max}} \) is the maximum cruise speed, \( h_{\text{max}} \) is the maximum cruise altitude, \( T_{\text{endurance}} \) is the maximum endurance time, \( a_{\text{max}} \) is the maximum acceleration, \( R_{\text{operation}} \) is the maximum operational radius, \( d_{\text{safe}} \) is the minimum safe distance from any obstacle \( \mathbf{o}_j \), \( t \) is the time variable, and \( \mathbf{r}(t) \) is the UAV’s position vector at time \( t \).
1.3 Objective Function
The primary objective is to minimize the total flight path length \( L \), which directly correlates with minimizing the total mission time \( T \), thereby maximizing response speed. The objective function is formulated as:
$$
\min L = \sum_{k=1}^{n} \|\mathbf{r}_{k} – \mathbf{r}_{k-1}\|
$$
where \( \mathbf{r}_0 = \mathbf{r}_{\text{start}} \) and \( \mathbf{r}_n = \mathbf{r}_{\text{goal}} \), and \( \|\mathbf{r}_{k} – \mathbf{r}_{k-1}\| \) represents the Euclidean distance between consecutive waypoints.
2. Algorithm Design
2.1 Foundation: Deep Reinforcement Learning and DQN
Reinforcement Learning (RL) is centered on an agent learning to make decisions by interacting with an environment. The agent executes actions, receives rewards or penalties, and observes new states, aiming to learn a policy that maximizes cumulative reward. Q-learning is a classic model-free RL algorithm that learns an action-value function \( Q(s, a) \), representing the expected future reward for taking action \( a \) in state \( s \). Its update rule is:
$$
Q_{\text{new}}(s, a) \leftarrow Q(s, a) + \alpha \left[ r + \gamma \max_{a’} Q(s’, a’) – Q(s, a) \right]
$$
where \( \alpha \) is the learning rate, \( \gamma \) is the discount factor, \( r \) is the immediate reward, and \( s’ \) is the next state.
Deep Q-Networks (DQN) combine Q-learning with deep neural networks to handle high-dimensional state spaces. It stabilizes training through two key techniques: Experience Replay, which stores and randomly samples past transitions to break correlations, and a Target Network, a separate network used to generate the Q-targets, reducing oscillations during learning.
2.2 The Proposed Improved DQN (IDQN) Algorithm
The proposed IDQN algorithm introduces specific enhancements to the standard DQN framework to better suit the fire UAV urban rescue problem.
2.2.1 Prioritized Experience Replay
Standard DQN samples experiences uniformly from its replay buffer. Prioritized Experience Replay (PER) improves data efficiency by assigning a sampling probability proportional to the Temporal Difference (TD) error of each experience. Experiences with larger TD errors, which signify a greater potential for learning, are sampled more frequently. The TD error \( \delta_t \) for a transition \( (s_t, a_t, r_t, s_{t+1}) \) is:
$$
\delta_t = r_t + \gamma \max_{a’} Q(s_{t+1}, a’; \boldsymbol{\theta}^-) – Q(s_t, a_t; \boldsymbol{\theta})
$$
where \( \boldsymbol{\theta} \) are the parameters of the online network and \( \boldsymbol{\theta}^- \) are the parameters of the target network. The sampling probability \( P(i) \) for the \( i \)-th transition is:
$$
P(i) = \frac{p_i^{\sigma}}{\sum_k p_k^{\sigma}}, \quad \text{where } p_i = |\delta_i| + \epsilon
$$
Here, \( \sigma \) controls the priority intensity (with \( \sigma=0 \) reverting to uniform sampling), and \( \epsilon \) is a small constant to ensure non-zero probability. To correct the bias introduced by non-uniform sampling, importance-sampling weights \( w_i \) are applied during the network update:
$$
w_i = \left( \frac{1}{N \cdot P(i)} \right)^{\beta}
$$
where \( N \) is the replay buffer size and \( \beta \) is a parameter that anneals from an initial value to 1.
2.2.2 State and Action Space Design
The state space \( \mathcal{S}_t \) is designed to provide the fire UAV agent with all necessary information for decision-making in the 3D urban environment:
$$
\mathcal{S}_t = \{ \mathbf{P}_t, \mathbf{G}_t, \mathbf{d}_t^{o,0}, \mathbf{d}_t^{o,1}, …, \mathbf{d}_t^{o,k-1} \}
$$
- \( \mathbf{P}_t = (x_t, y_t, z_t) \): The UAV’s current 3D position.
- \( \mathbf{G}_t = (x_g, y_g, z_g) \): The 3D coordinates of the target fire point.
- \( \mathbf{d}_t^{o,j} = (d_x^{o,j}, d_y^{o,j}, d_z^{o,j}) \): The relative distance vector from the UAV to the \( j \)-th obstacle (building), for \( j = 0, 1, …, k-1 \), where \( k \) is the total number of considered obstacles.
The action space is defined to allow fine-grained 3D movement. The fire UAV can move one discrete step in any of 26 possible directions: 9 directions in the upper layer (increasing altitude), 8 directions in the middle layer (constant altitude), and 9 directions in the lower layer (decreasing altitude). This comprehensive action set enables the UAV to navigate efficiently around complex urban structures.
2.2.3 Composite Reward Function Design
A meticulously designed reward function is crucial for guiding the fire UAV agent toward optimal behavior. To meet the extreme timeliness requirement of fire rescue, we propose a composite reward function \( R_{\text{total}} \) with the following components:
- Exponential Time Penalty (\( R_{\text{time}} \)): This is the dominant term, imposing a cost that grows exponentially with each time step. It creates a strong incentive to find the shortest path, aligning with the mission’s primary objective. The specific form \( R_{\text{time}} = -\delta \cdot t^{\mu} \) was tuned through experimentation. Comparative tests, as shown in Table 1, determined that \( \delta = 0.01 \) and \( \mu = 2 \) yielded the shortest task duration while maintaining high success rates.
- Collision Penalty (\( R_{\text{collision}} \)): A large negative reward (e.g., -50) is given if the UAV’s chosen action would result in a collision with an obstacle, and the move is invalidated. This ensures flight safety. Tests confirmed that a penalty of -50 effectively minimized collisions without unduly compromising efficiency (see Table 2).
- Directional Guidance Reward (\( R_{\text{direction}} \)): This dense reward provides immediate feedback based on proximity change: \( R_{\text{direction}} = d_{\text{prev}} – d_{\text{next}} \), where \( d_{\text{prev}} \) and \( d_{\text{next}} \) are the distances to the goal before and after taking the action. A positive reward encourages movement toward the target.
- Goal Achievement Reward (\( R_{\text{goal}} \)): A large positive reward (e.g., +1000) is granted upon successfully reaching the target point, signifying mission completion. This value was also optimized via testing (see Table 3).
The total reward for a time step is the sum of the applicable components:
$$
R_{\text{total}} = R_{\text{time}} + R_{\text{collision}} + R_{\text{direction}} + R_{\text{goal}}
$$
| Parameter Set (\( \delta, \mu \)) | Average Task Duration (s) | Success Rate (%) |
|---|---|---|
| (0.100, 1) | 5.7 | 83 |
| (0.010, 2) | 4.9 | 95 |
| (0.001, 3) | 5.5 | 91 |
| Collision Penalty | Collision Rate (%) | Average Task Duration (s) |
|---|---|---|
| -10 | 18.2 | 4.5 |
| -50 | 5.3 | 4.6 |
| -100 | 2.1 | 4.9 |
| Goal Reward | Success Rate (%) | Average Task Duration (s) |
|---|---|---|
| +500 | 92 | 5.2 |
| +1000 | 95 | 4.9 |
| +2000 | 94 | 4.8 |
2.3 IDQN Algorithm Workflow
The workflow of the proposed IDQN-based fire UAV planner is as follows:
- Initialization: Initialize the online Q-network \( Q(\boldsymbol{\theta}) \) and the target network \( Q(\boldsymbol{\theta}^-) \) with random weights. Create an empty prioritized replay buffer \( \mathcal{D} \).
- Episode Loop: For each training episode:
- Reset the environment to the starting state \( s_0 \).
- Step Loop: For each time step \( t \) within the episode:
- Select an action \( a_t \) using an \( \epsilon \)-greedy policy based on \( Q(s_t, \cdot; \boldsymbol{\theta}) \).
- Execute \( a_t \) in the environment, observe reward \( r_t \) and next state \( s_{t+1} \).
- Store the transition \( (s_t, a_t, r_t, s_{t+1}) \) in \( \mathcal{D} \) with its initial priority (maximum priority if buffer is not full, else calculated TD-error).
- Prioritized Sampling & Learning: Sample a mini-batch of transitions from \( \mathcal{D} \) according to priorities \( P(i) \). For each sampled transition \( j \):
- Compute the importance-sampling weight \( w_j \).
- Calculate the Q-target: \( y_j = r_j + \gamma \max_{a’} Q(s’_{j}, a’; \boldsymbol{\theta}^-) \).
- Compute the TD-error: \( \delta_j = y_j – Q(s_j, a_j; \boldsymbol{\theta}) \).
- Update the transition’s priority in \( \mathcal{D} \) with \( |\delta_j| \).
- Accumulate the weighted loss: \( \mathcal{L} = \frac{1}{N} \sum_j w_j \cdot \delta_j^2 \).
- Perform a gradient descent step on \( \mathcal{L} \) to update \( \boldsymbol{\theta} \).
- Periodically update the target network: \( \boldsymbol{\theta}^- \leftarrow \boldsymbol{\theta} \).
- Set \( s_t \leftarrow s_{t+1} \).
- Terminate the episode if the fire UAV reaches the goal or violates a constraint.
- Deployment: After training, use the greedy policy based on the trained \( Q(\boldsymbol{\theta}) \) network to generate optimal paths for the fire UAV in new, unseen urban environments.
3. Simulation Experiments and Analysis
3.1 Experimental Setup
The simulation platform utilized hardware with an R7-6800H 8-core processor, 16GB RAM, and an RTX 3050 GPU. The TH-MH-30 multi-rotor fire UAV was selected as the reference platform for its stability, payload capacity (30 kg), endurance (30 min), and operational specifications (max speed: 30 km/h, max altitude: 200 m, operational radius: 5 km). Algorithm hyperparameters were set as: learning rate \( \alpha = 10^{-3} \), discount factor \( \gamma = 0.99 \), initial exploration \( \epsilon_0 = 0.9 \) with a decay rate, PER exponent \( \sigma = 0.8 \), replay buffer size \( 10^6 \), and neural network hidden layers with 256 and 128 nodes.
3.2 Simulation Environment Modeling
Two distinct 3D simulation environments were constructed based on real urban fire incidents to test the algorithm under different complexity levels. Buildings were modeled as axis-aligned bounding boxes (AABBs) defined by a base vertex \( (x, y, z) \) and dimensions (length, width, height). The vertices \( \mathbf{V}_i \) of any building are given by:
$$
\mathbf{V}_i = (x + l_{\text{ength}}, y + w_{\text{idth}}, z + h_{\text{eight}})
$$
Case 1 (Mingshang Xiyuan-Inspired): Simulates a 180m x 180m x 150m area containing 8 buildings. The tallest building is 150m high, interspersed with lower structures like an 18m kindergarten, representing a moderately dense urban block.
Case 2 (Jiaozhou Road Apartment-Inspired): Simulates a more complex 150m x 180m x 200m area with 12 buildings. The primary structure reaches 200m in height, creating a denser and more challenging navigation environment for the fire UAV.
3.3 Simulation Results and Analysis
3.3.1 Algorithm Training Performance
Both the baseline DQN and the proposed IDQN algorithms were trained for 1000 episodes in each environment. The training curves (episode reward and average Q-value) reveal significant advantages for IDQN:
- Faster Convergence: The IDQN agent’s reward stabilized in approximately 200 episodes, whereas the DQN agent required around 400 episodes.
- Enhanced Stability: The DQN training showed severe reward collapses and high variance between episodes 400-600, indicating instability and potential convergence to poor local optima. The IDQN training was markedly smoother and more stable.
- Higher Performance: The final average reward achieved by IDQN was consistently higher than that of DQN, closely approaching the maximum possible reward. The Q-value curve for IDQN also showed a steadier, more monotonic increase, reflecting more accurate value estimation.
These improvements are directly attributable to the PER mechanism, which focuses learning on more informative experiences, and the finely tuned composite reward function, which provides clearer guidance to the fire UAV agent.
3.3.2 Trajectory Planning Results
The trained IDQN policy was compared against the baseline DQN and a classic sampling-based planner, the RRT algorithm, in both test environments. The RRT algorithm, while robust for finding feasible paths, is not optimized for path length or smoothness.
Case 1 Results: Start: (24, 20, 1)m, Goal: (140, 140, 90)m.
| Algorithm | Path Length (m) | Number of Turns | Planning/Execution Time (s) |
|---|---|---|---|
| RRT | 321.34 | 18 | 10.041 |
| DQN | 259.30 | 10 | 4.803 |
| IDQN (Proposed) | 248.10 | 6 | 4.356 |
The IDQN-generated path for the fire UAV was 22.78% shorter than RRT and 4.32% shorter than DQN. More importantly, the number of sharp turns (inflection points) was reduced by 66.67% compared to RRT and 40% compared to DQN, resulting in a smoother, more energy-efficient, and faster flight path.
Case 2 Results: Start: (30, 10, 1)m, Goal: (75, 150, 60)m.
| Algorithm | Path Length (m) | Number of Turns | Planning/Execution Time (s) |
|---|---|---|---|
| RRT | 242.33 | 13 | 6.195 |
| DQN | 189.35 | 11 | 3.928 |
| IDQN (Proposed) | 180.79 | 4 | 3.756 |
In the more complex Case 2 environment, the advantages of the IDQN fire UAV planner were even more pronounced. It achieved a 25.40% shorter path than RRT and a 4.52% improvement over DQN. The reduction in the number of turns was substantial: 69.23% fewer than RRT and 63.64% fewer than DQN, demonstrating superior path optimality and smoothness.
Comprehensive Analysis: The RRT algorithm, while capable of finding a feasible path, produces convoluted, sub-optimal trajectories with many turns, leading to longer flight times—a critical drawback in emergency response. The standard DQN algorithm generates better paths but still suffers from inefficiencies in training and final path quality. The proposed IDQN algorithm, with its prioritized learning and mission-specific reward shaping, consistently produces the shortest and smoothest paths. This directly translates to minimized mission time for the fire UAV, which is the paramount objective in urban fire rescue operations within dense, access-restricted environments.
4. Conclusion
This paper presented an Improved Deep Q-Network (IDQN) algorithm for the critical task of fire UAV trajectory planning in urban fire rescue scenarios. To address the unique challenges of this domain—specifically the imperative for minimal response time within complex 3D obstacle fields—the algorithm incorporated several key innovations over standard DQN. The integration of Prioritized Experience Replay accelerated and stabilized the learning process. A state space incorporating positional and obstacle proximity information, coupled with a comprehensive 26-direction action space, enabled precise 3D navigation. Most importantly, a novel composite reward function was designed, dominated by an exponentially increasing time penalty and augmented with obstacle collision penalties and goal achievement rewards. This function effectively encodes the mission priorities: speed and safety.
Simulation experiments based on two realistic urban fire scenarios demonstrated the superiority of the proposed IDQN fire UAV planner. Compared to both the sampling-based RRT algorithm and the baseline DQN algorithm, IDQN achieved significantly shorter flight paths and drastically reduced the number of sharp turns. This optimization directly leads to faster mission completion times, enhancing the overall effectiveness and responsiveness of fire UAVs in life-saving operations. Future work will focus on extending this framework to dynamic environments with moving obstacles and investigating its application in coordinated multi-fire UAV rescue missions.