The acceleration of urbanization has led to densely packed high-rise buildings and frequent traffic congestion, forming defining characteristics of modern cities that significantly increase the difficulty of urban fire rescue. In particular, within old urban districts and commercial areas characterized by dense construction, narrow roads, and haphazardly parked vehicles, traditional firefighting trucks are often unable to arrive promptly due to traffic jams and blocked fire lanes. This delay in rescue response is especially critical in high-rise building fires. Urban firefighting primarily faces three core challenges: firstly, fires in super high-rise buildings exceed the operational height limits of ladder trucks; secondly, confined operational spaces restrict the entry of firefighting vehicles; and thirdly, complex traffic environments make it difficult to guarantee rescue timeliness.
Firefighting Unmanned Aerial Vehicles (UAVs), or fire drones, have emerged as crucial tools for high-rise fire rescue due to their flexibility, rapid response capability, and ability to circumvent obstacles. These fire drones can avoid traffic congestion and narrow streets, quickly reaching the fire scene to execute firefighting tasks, such as carrying dry powder extinguishers or deploying water-based fire suppression capsules. Trajectory planning, a key technology for the autonomous firefighting operations of fire drones in urban rescue scenarios, is the core capability that enables UAVs to find safe and feasible flight paths quickly and accurately. The choice of planning algorithm directly determines flight efficiency, safety, and energy consumption. Traditional optimization algorithms like Ant Colony, Genetic Algorithm, or sampling-based methods like Rapidly-exploring Random Tree (RRT) often face limitations in this specific, urgent, and dynamically changing urban fire rescue context. They may rely on global environmental information, have high computational complexity leading to slow responses, or generate non-optimal, jerky paths unsuitable for stable and safe UAV flight.
In contrast, deep reinforcement learning (DRL) algorithms are particularly suitable due to their powerful online learning and adaptive decision-making capabilities. These algorithms can optimize decision-making policies in real-time through continuous interaction with the environment without relying on a complete prior map, effectively handling obstacles and unexpected situations during rescue. Among DRL methods, the Deep Q-Network (DQN) has shown significant promise. However, standard DQN applied to generic scenarios may not be fully optimized for the stringent demands of urban firefighting, such as extreme time sensitivity and the need for efficient navigation in dense clutter.
This paper focuses on the scenario where a fire drone takes off from a fire truck’s parking location and must navigate through a cluster of buildings to reach the fire point on a high-rise structure. We propose a firefighting UAV trajectory planning algorithm based on an Improved Deep Q-Network (IDQN) tailored for urban fire rescue missions. The primary objective is to plan an optimal flight trajectory from the launch point to the fire point that guarantees absolute safety while minimizing the total mission time, which is directly related to the trajectory length.
1. Modeling the Firefighting UAV Trajectory Planning Problem
1.1 Problem Description
Ignoring external factors like low-altitude wind fields and electromagnetic interference, and under the premise of satisfying the UAV’s 3D mobility performance constraints, the goal is to plan an optimal flight trajectory from the parking point \( \mathbf{r}_{\text{base}} = (x_0, y_0, z_0) \) to the fire point \( \mathbf{r}_{\text{goal}} = (x_g, y_g, z_g) \), potentially via a set of predefined waypoints \( \{ \mathbf{r}_1, \mathbf{r}_2, …, \mathbf{r}_n \} \). The core objective is to minimize the total flight path length \( L \), thereby reducing the total mission time \( T \).
1.2 Constraints
The fire drone must operate within several physical and operational limits during the rescue mission:
$$
\begin{aligned}
& \sqrt{\dot{x}(t)^2 + \dot{y}(t)^2} \leq V_{\text{max}} \\
& 0 \leq z(t) \leq h_{\text{max}} \\
& T \leq T_{\text{endurance}} \\
& \|\ddot{\mathbf{r}}(t)\| \leq a_{\text{max}} \\
& \|\mathbf{r}(t) – \mathbf{r}_{\text{base}}\| \leq R_{\text{operation}} \\
& \|\mathbf{r}(t) – \mathbf{o}_j\| \geq d_{\text{safe}}
\end{aligned}
$$
Where \( V_{\text{max}} \) is the maximum cruise speed, \( h_{\text{max}} \) is the maximum cruise altitude, \( T_{\text{endurance}} \) is the maximum endurance, \( a_{\text{max}} \) is the maximum acceleration, \( R_{\text{operation}} \) is the maximum operational radius, \( d_{\text{safe}} \) is the minimum safe distance from obstacles, \( t \) is time, \( T \) is total mission time, \( \mathbf{r}(t) \) is the UAV’s position vector at time \( t \), and \( \mathbf{o}_j \) is the position vector of the \( j \)-th obstacle.
1.3 Objective Function
The objective is to minimize the total Euclidean flight path length \( L \):
$$
\min L = \sum_{k=1}^{n} \| \mathbf{r}_{k} – \mathbf{r}_{k-1} \|
$$
where \( \mathbf{r}_0 = \mathbf{r}_{\text{start}} \) and \( \mathbf{r}_{n+1} = \mathbf{r}_{\text{goal}} \). Minimizing \( L \) directly contributes to minimizing the mission time \( T \) for the fire drone.
2. Improved Deep Q-Network (IDQN) Algorithm Design
2.1 Foundation: Deep Q-Network (DQN)
Reinforcement learning involves an agent interacting with an environment, learning a policy through rewards. Q-learning is a model-free method that learns an action-value function \( Q(s, a) \). DQN combines Q-learning with deep neural networks to handle high-dimensional state spaces, utilizing experience replay for sample efficiency and a target network for training stability.
2.2 Proposed Improvements for Firefighting UAV
2.2.1 Prioritized Experience Replay
We replace the uniform sampling of standard experience replay with Prioritized Experience Replay. This mechanism assigns a higher sampling probability to experiences with a larger Temporal Difference (TD) error, which measures how “surprising” or informative the transition is for learning. The TD-error \( \delta_t \) for an experience \( (s_t, a_t, r_t, s_{t+1}) \) is:
$$
\delta_t = r_t + \gamma \max_{a’} Q(s_{t+1}, a’; \theta_t^-) – Q(s_t, a_t; \theta)
$$
where \( \theta \) and \( \theta_t^- \) are the parameters of the main and target networks, respectively, and \( \gamma \) is the discount factor. The sampling probability \( P(i) \) for the \( i \)-th experience is proportional to \( |\delta_i|^\sigma \), where \( \sigma \) controls the priority degree. To correct the bias introduced by non-uniform sampling, importance-sampling weights \( w_i = (N \cdot P(i))^{-\beta} / \max_j w_j \) are applied during the Q-update, where \( \beta \) anneals from an initial value to 1.
2.2.2 State and Action Space Design
The state space \( S_t \) is designed to provide the fire drone with sufficient information about its environment and mission goal:
$$
S_t = \{ \mathbf{P}_t, \mathbf{G}_t, \mathbf{d}_{t}^{o,0}, \mathbf{d}_{t}^{o,1}, …, \mathbf{d}_{t}^{o,k-1} \}
$$
where \( \mathbf{P}_t = (x_t, y_t, z_t) \) is the UAV’s current 3D position, \( \mathbf{G}_t = (x_g, y_g, z_g) \) is the target fire point position, and \( \mathbf{d}_t^{o,j} = (x^{o,j}, y^{o,j}, z^{o,j}) \) is the relative distance vector from the UAV to the \( j \)-th obstacle (for \( j = 0, …, k-1 \), with \( k \) being the total number of considered obstacles).
The action space is expanded to allow fine-grained 3D movement. The fire drone can choose from 26 discrete movement directions: 9 directions for upward movement (inclining up), 8 directions for level movement, and 9 directions for downward movement (inclining down). This provides the agility needed to navigate complex urban canyons.
2.2.3 Composite Reward Function for Time-Critical Rescue
A novel composite reward function is designed to meet the extreme timeliness requirement of fire drone rescue missions, balancing time penalty, safety, and goal orientation.
1) Exponential Time Penalty: A negative reward that increases exponentially with time encourages the fire drone to find the fastest path. The function is \( R_t(t) = -\delta \cdot t^{\mu} \). Comparative tests determined optimal parameters.
| Parameter Combination (\( \delta, \mu \)) | Mission Duration (s) | Success Rate (%) |
|---|---|---|
| (0.100, 1) | 5.7 | 83 |
| (0.010, 2) | 4.9 | 95 |
| (0.001, 3) | 5.5 | 91 |
Based on these results, \( \delta = 0.01 \) and \( \mu = 2 \) were selected.
2) Collision Penalty: A large negative reward \( R_c = -50 \) is given if the UAV’s next action would result in a collision, forcing it to prioritize safety.
| Collision Penalty Value | Collision Rate (%) | Mission Duration (s) |
|---|---|---|
| -10 | 18.2 | 4.5 |
| -50 | 5.3 | 4.6 |
| -100 | 2.1 | 4.9 |
3) Goal-Oriented Rewards:
Directional Reward: \( R_d = d_{\text{prev}} – d_{\text{new}} \), where \( d \) is the distance to the goal. This provides immediate feedback for moving closer (positive) or farther (negative) from the target.
Precise Arrival Reward: A large positive reward \( R_a = +1000 \) is granted upon successfully reaching the goal point, strongly reinforcing the final objective.
| Arrival Reward Value | Success Rate (%) | Mission Duration (s) |
|---|---|---|
| 500 | 92 | 5.2 |
| 1000 | 95 | 4.9 |
| 2000 | 94 | 4.8 |
4) Total Reward: The composite reward at each step is the sum: \( R_{\text{total}} = R_t + R_c + R_d + R_a \).
3. Simulation Experiments and Analysis
3.1 Experimental Setup
Simulations were conducted on a platform with an R7-6800H processor, 16GB RAM, and an RTX 3050 GPU. Algorithm parameters: learning rate \( \gamma = 10^{-3} \), discount factor \( \alpha = 0.99 \), initial exploration rate \( \epsilon_0 = 0.9 \), exploration decay rate, replay buffer size \( 10^6 \), priority exponent \( \sigma = 0.8 \). A TH-MH-30 multi-rotor fire drone model was used with key specs: max load 30 kg, endurance 30 min, max speed 30 km/h, max altitude 200 m, operational radius 5 km.
3.2 Simulation Environment Modeling
Two 3D simulation environments were built based on real urban fire scenarios of different complexity, modeling buildings as axis-aligned bounding boxes (AABB). The vertices \( \mathbf{V}_i \) of a building with start point \( (x, y, z) \), length \( l \), width \( w \), and height \( h \) are given by: \( \mathbf{V}_i = (x + l, y + w, z + h) \).
Case 1 (Dense Residential Area): Simulates a 180m x 180m x 150m area with 8 buildings, including a 150m high-rise and an 18m kindergarten.
Case 2 (High-Density Apartment Zone): Simulates a 150m x 180m x 200m area with 12 buildings, including a 200m tower.
3.3 Results and Analysis
3.3.1 Algorithm Training Performance
Both DQN and IDQN were trained for 1000 episodes. The IDQN algorithm demonstrated superior training characteristics: faster convergence (stable at ~200 episodes vs. ~400 for DQN), a more stable training process without the reward collapse observed in DQN, higher average reward, and a smoothly increasing Q-value curve indicating accurate value estimation.
3.3.2 Trajectory Planning Results
The proposed IDQN was compared against the standard RRT algorithm and the baseline DQN. For Case 1 (start: (24,20,1)m, goal: (140,140,90)m), the planned trajectories are shown, and quantitative results are as follows:
| Algorithm | Trajectory Length (m) | Number of Inflection Points | Planning Time (s) |
|---|---|---|---|
| RRT | 321.34 | 18 | 10.041 |
| DQN | 259.30 | 10 | 4.803 |
| IDQN (Proposed) | 248.10 | 6 | 4.356 |
For Case 2 (start: (30,10,1)m, goal: (75,150,60)m):
| Algorithm | Trajectory Length (m) | Number of Inflection Points | Planning Time (s) |
|---|---|---|---|
| RRT | 242.33 | 13 | 6.195 |
| DQN | 189.35 | 11 | 3.928 |
| IDQN (Proposed) | 180.79 | 4 | 3.756 |
In both cases, the IDQN algorithm significantly outperformed the others. Compared to RRT, it reduced path length by 22.78% and 25.40%, and inflection points by 66.67% and 69.23% in Case 1 and 2, respectively. Compared to DQN, it achieved reductions of 4.32% and 4.52% in path length, and 40.00% and 63.64% in inflection points. This demonstrates that the IDQN algorithm not only finds shorter paths but also smoother ones with fewer sharp turns, leading to more efficient and stable flight for the fire drone. The RRT algorithm, while adaptable, produces long, winding paths unsuitable for time-critical rescue. The baseline DQN shows improvement but lacks the efficiency gains from the prioritized experience replay and the mission-optimized reward function of the IDQN.
4. Conclusion
This paper addressed the critical challenge of trajectory planning for fire drones in complex urban fire rescue scenarios. An Improved Deep Q-Network (IDQN) algorithm was proposed, featuring prioritized experience replay for efficient learning, a comprehensive 26-direction action space for agile 3D navigation, and a novel composite reward function centered on an exponential time penalty to enforce urgency. Simulation experiments based on two realistic urban fire environments demonstrated that the IDQN algorithm significantly outperforms both the sampling-based RRT algorithm and the standard DQN algorithm. It generates shorter, smoother flight trajectories with fewer inflection points, effectively optimizing the flight path of the fire drone. This leads to reduced mission completion time, thereby enhancing the response speed and task execution efficiency of fire drone units in urban firefighting, where every second counts. Future work will focus on extending this approach to dynamic environments and multi-agent cooperative fire drone rescue missions.