With the rapid advancement of drone technology, Unmanned Aerial Vehicles (UAVs) have become integral in various low-altitude applications, including urban surveillance, military reconnaissance, and logistics delivery. The proliferation of Unmanned Aerial Vehicle systems in shared airspace introduces significant challenges, such as dynamic environmental constraints, spatial limitations, and incomplete information. These factors elevate the risk of mid-air collisions among multiple UAVs and static obstacles, necessitating robust and real-time collision avoidance strategies. Traditional methods, including rule-based approaches, optimization techniques, and control theories, often fall short in addressing the complexities of multi-UAV interactions due to their lack of adaptability, high computational demands, and reliance on precise environmental models. Reinforcement learning (RL) has emerged as a promising solution, enabling autonomous policy learning through environmental interactions without explicit modeling. However, existing RL-based methods struggle with non-stationarity in multi-agent settings and partial observability in real-world scenarios. To overcome these limitations, this paper proposes a novel Multi-Agent Dueling Double Deep Q-Network (MAD3QN) framework for cooperative collision avoidance in low-altitude airspace. The approach integrates a sector-based local observation model, a multi-factor reward function, and a shared policy architecture to enhance scalability, robustness, and training efficiency. Extensive simulations demonstrate that the proposed method achieves a collision avoidance success rate exceeding 98% across diverse scenarios, outperforming state-of-the-art techniques. This research provides a foundational framework for the safe and efficient operation of multi-UAV systems, advancing the practical deployment of drone technology in complex environments.
The integration of Unmanned Aerial Vehicle systems into low-altitude airspace requires addressing dynamic coupling issues, where individual UAV decisions are influenced by static obstacles and other UAVs’ state changes. This problem is formulated as a Multi-Agent Markov Decision Process (MMDP), focusing on optimizing the cumulative expected reward while adhering to safety and task constraints. The key contributions include a unified observation model that encodes dynamic and static obstacles into fixed-dimensional vectors, a composite reward function that balances goal-directed behavior and collision penalties, and a distributed training framework that mitigates non-stationarity through parameter sharing. The MAD3QN algorithm leverages Dueling Double Deep Q-Network (D3QN) principles to reduce overestimation bias and improve value function approximation. Experimental results confirm the method’s superiority in terms of success rate, path length, and completion time, highlighting its potential for real-world applications in drone technology. Future work will extend the model to three-dimensional airspace and validate it through hardware-in-the-loop testing, further enhancing the practicality of Unmanned Aerial Vehicle operations.
Problem Formulation
In low-altitude environments, multiple Unmanned Aerial Vehicles operate amidst static obstacles, creating a complex, spatially constrained, and partially observable setting. The collision avoidance problem is defined as a multi-constraint optimization task, where each UAV acts as an independent agent. For UAV $i$ at time $t$, its position is denoted as $\mathbf{P}_i(t)$, velocity as $v_i(t)$, and heading angle as $\psi_i(t)$. The primary objective is to ensure all UAVs reach their target positions $\mathbf{P}_i^g$ within a finite time horizon $T$ while avoiding collisions with other UAVs and static obstacles. The safety constraints are mathematically expressed as:
$$ \begin{cases}
\min \max \quad & \text{Objective: Maximize cumulative reward} \\
\text{s.t.} \quad & \|\mathbf{P}_i(t) – \mathbf{P}_j(t)\| \geq R_c, \quad \forall i \neq j, \forall t \\
& \|\mathbf{P}_i(t) – \mathbf{P}_o\| \geq R_c, \quad \forall i, \forall t \\
& \|\mathbf{P}_i(t) – \mathbf{P}_i^g\| \leq R_g, \quad \forall t \in [0, T] \\
& |\Delta \psi_i(t)| \leq \Delta \psi_{\max}, \quad |v_i(t)| \leq v_{\max}, \quad \forall t
\end{cases} $$
Here, $R_c$ represents the collision safety threshold, $R_g$ is the goal proximity radius, $\Delta \psi_{\max}$ is the maximum heading change rate, and $v_{\max}$ is the maximum velocity. The problem is transformed into an MMDP, where the global objective is to maximize the average cumulative discounted reward across all agents:
$$ \max_{\pi_\theta} \mathbb{E}_{\pi_\theta} \left[ \frac{1}{N} \sum_{i=1}^N \sum_{t=0}^T \gamma^t r_i(t) \right] $$
where $\pi_\theta$ is the parameterized policy shared by all agents, $\gamma$ is the discount factor, and $r_i(t)$ is the immediate reward for UAV $i$ at time $t$. This formulation underscores the need for cooperative strategies in drone technology, ensuring that Unmanned Aerial Vehicle systems can navigate densely populated airspace safely and efficiently.
Reinforcement Learning Environment Model
The reinforcement learning environment for multi-UAV collision avoidance is designed to provide each agent with localized observations, process actions, and compute rewards based on task performance. The environment model comprises three core components: a partial observation state, a discrete action space, and a multi-factor reward function. This structure ensures that the Unmanned Aerial Vehicle can operate under realistic constraints, mimicking the limitations of onboard sensors in drone technology.
Partial Observation State
Each UAV’s observation is derived from a sector-based detection model, where the sensing range is divided into $K$ sectors. The observation vector $\mathbf{O}$ for a UAV includes its own state and the states of the most threatening intruder and static obstacle in each sector. This design ensures a fixed-dimensional input, facilitating neural network processing and scalability. The observation is defined as:
$$ \mathbf{O} = \left\{ \mathbf{O}_{\text{own}}, \mathbf{O}_{1d}, \mathbf{O}_{2d}, \dots, \mathbf{O}_{Kd} \right\} $$
The own-state vector $\mathbf{O}_{\text{own}}$ includes velocity $v_{\text{own}}$, heading angle $\psi_{\text{own}}$, distance to goal $D_g$, and relative heading to goal $\phi_g$:
$$ \mathbf{O}_{\text{own}} = \left\{ v_{\text{own}}, \psi_{\text{own}}, D_g, \phi_g \right\} $$
For each sector $i$, the observation $\mathbf{O}_{id}$ captures the intruder UAV and static obstacle information:
$$ \mathbf{O}_{id} = \left\{ v_{\text{int}}, \psi_{\text{int}}, D_{\text{int}}, \phi_{\text{int}}, D_{\text{obs}}, \phi_{\text{obs}} \right\} $$
Here, $v_{\text{int}}$ and $\psi_{\text{int}}$ denote the intruder’s velocity and heading, $D_{\text{int}}$ and $D_{\text{obs}}$ represent distances to the intruder and obstacle, and $\phi_{\text{int}}$ and $\phi_{\text{obs}}$ are the relative headings. If no intruder or obstacle is present in a sector, the corresponding values are zero-padded. This model not only standardizes the input space but also allows for future extension to 3D navigation in Unmanned Aerial Vehicle systems, incorporating altitude and vertical maneuvers.
Action Space
The action space for each UAV is discrete, enabling real-time decision-making by selecting from a finite set of maneuvers. Actions involve adjustments in heading angle and velocity, defined as:
$$ \mathcal{A} = \left\{ (\Delta \psi, \Delta v) \mid \Delta \psi \in \{ -\delta_\psi, 0, +\delta_\psi \}, \Delta v \in \{ -\delta_v, 0, +\delta_v \} \right\} $$
where $\delta_\psi$ and $\delta_v$ are the step changes for heading and velocity, respectively. This discrete formulation balances computational efficiency with sufficient maneuverability, critical for responsive collision avoidance in dynamic drone technology applications.
Reward Function
A composite reward function is designed to guide UAV behavior toward goal achievement while penalizing collisions and encouraging smooth trajectories. The reward $r_i(t)$ for UAV $i$ at time $t$ is a sum of five components:
$$ r_i(t) = r_{g,i}(t) + r_{d,i}(t) + r_{h,i}(t) + r_{c,i}(t) + r_{s,i}(t) $$
Goal Reward $r_{g,i}(t)$: Provides a positive reward $C_g$ when the UAV reaches within a threshold $R_g$ of its goal:
$$ r_{g,i}(t) = \begin{cases}
C_g & \text{if } D_g \leq R_g \\
0 & \text{otherwise}
\end{cases} $$
Distance Guidance Reward $r_{d,i}(t)$: Encourages reduction in distance to the goal, normalized by the maximum velocity $V_{\max,i}$:
$$ r_{d,i}(t) = \frac{D_g(t-1) – D_g(t)}{V_{\max,i}} $$
Heading Guidance Reward $r_{h,i}(t)$: Promotes alignment with the goal direction using a cosine similarity weighted by $C_h$:
$$ r_{h,i}(t) = C_h \cos(\phi_g) $$
Collision Penalty $r_{c,i}(t)$: Imposes a large negative reward for collisions with other UAVs ($C_{\text{uav}}$) or obstacles ($C_{\text{obs}}$):
$$ r_{c,i}(t) = \begin{cases}
C_{\text{uav}} & \text{if } D_{\text{uav}} \leq R_c \\
C_{\text{obs}} & \text{if } D_{\text{obs}} \leq R_c \\
0 & \text{otherwise}
\end{cases} $$
Smoothing Reward $r_{s,i}(t)$: Penalizes abrupt changes in heading and velocity to ensure trajectory smoothness, weighted by $C_s$:
$$ r_{s,i}(t) = C_s \left( |\Delta \psi(t) – \Delta \psi(t-1)| + |\Delta v(t) – \Delta v(t-1)| \right) $$
This multi-factor reward system ensures that Unmanned Aerial Vehicle agents learn policies that are not only safe but also efficient and natural in their movements, reflecting the operational requirements of advanced drone technology.
MAD3QN Algorithm for Multi-UAV Collision Avoidance
The Multi-Agent Dueling Double Deep Q-Network (MAD3QN) algorithm is developed to address the challenges of non-stationarity and partial observability in multi-UAV environments. By combining the strengths of D3QN with a distributed training framework, MAD3QN enables efficient and scalable learning for Unmanned Aerial Vehicle systems.
D3QN Algorithm Foundations
D3QN integrates Double DQN (DDQN) and Dueling DQN to mitigate overestimation bias and enhance value function approximation. In traditional DQN, the Q-value is updated as:
$$ Q(s_t, a_t) \leftarrow Q(s_t, a_t) + \alpha \left[ r_t + \gamma \max_{a’} Q(s_{t+1}, a’) – Q(s_t, a_t) \right] $$
where $\alpha$ is the learning rate and $\gamma$ is the discount factor. DDQN decouples action selection from valuation using separate online and target networks:
$$ y^{\text{DDQN}}_t = r_t + \gamma Q_{\text{target}} \left( s_{t+1}, \arg\max_{a’} Q_{\text{online}}(s_{t+1}, a’) \right) $$
Dueling DQN decomposes the Q-function into a state value function $V(s)$ and an advantage function $A(s, a)$:
$$ Q(s, a) = V(s) + \left( A(s, a) – \frac{1}{|\mathcal{A}|} \sum_{a’} A(s, a’) \right) $$
Combining these, D3QN computes the target value as:
$$ y^{\text{D3QN}}_t = r_t + \gamma Q_{\text{target}} \left( s_{t+1}, \arg\max_{a’} Q_{\text{online}}(s_{t+1}, a’) \right) $$
This approach reduces variance and improves policy stability, which is crucial for managing the interconnected dynamics of multiple Unmanned Aerial Vehicles.
Network Architecture Design
The neural network for MAD3QN features an input layer that accepts the fixed-dimensional observation vector $\mathbf{O}$, with a size of $(6K + 4)$ nodes for $K$ sectors. This is followed by two fully connected layers with 128 and 64 neurons, respectively, using ReLU activation functions for nonlinear feature extraction. The network branches into two streams: one for the value function $V(s)$ and another for the advantage function $A(s, a)$. The outputs are combined to produce Q-values for each discrete action, as defined in the action space. This architecture enables robust function approximation in high-dimensional state spaces, a key requirement for sophisticated drone technology applications.
Distributed Training Framework
The training process employs a distributed framework where all UAV agents share the same policy network parameters. Each agent interacts with the environment based on its local observations, storing experiences in a shared replay buffer. During training, mini-batches are sampled to update the network using the D3QN loss function. The $\epsilon$-greedy strategy is used for exploration, with $\epsilon$ decaying from 1 to 0.05 over time. The algorithm proceeds as follows:
- Initialize the environment, shared policy network parameters $\theta$, target network parameters $\theta^-$, and replay buffer of capacity $M$.
- For each episode up to a maximum number:
- Reset the environment and obtain initial observations.
- For each time step until the maximum step count:
- Each UAV selects an action based on its local observation and $\epsilon$-greedy policy.
- Execute actions collectively, observe rewards and next states.
- Store transitions $(\mathbf{O}_t, a_t, r_t, \mathbf{O}_{t+1})$ in the replay buffer.
- If the buffer is full, sample a mini-batch and compute target values using D3QN.
- Update the online network parameters $\theta$ via gradient descent.
- Periodically update the target network $\theta^- \leftarrow \theta$.
This framework enhances sample efficiency and policy consistency, addressing the non-stationarity inherent in multi-agent reinforcement learning for Unmanned Aerial Vehicle systems.
Simulation and Performance Analysis
To validate the proposed MAD3QN method, comprehensive simulations were conducted under various scenarios, comparing its performance against baseline algorithms such as MADQN, D3QN, IDQN, and random action selection. The experiments were performed on a high-performance workstation with an i9-10900 CPU, TITAN RTX GPU, and 128GB RAM, using Python 3.11 and PyTorch. Key hyperparameters are summarized in Table 1.
| Parameter | Value |
|---|---|
| Learning Rate | 0.001 |
| Discount Factor ($\gamma$) | 0.995 |
| Batch Size | 256 |
| Replay Buffer Capacity | 100,000 |
| Target Network Update Frequency | 50 steps |
| Exploration Rate ($\epsilon$) | 1 → 0.05 |
| Maximum Steps per Episode | 500 |
Training Process Evaluation
The training performance of MAD3QN was compared with single-agent D3QN over 5000 episodes, with the average cumulative reward per agent as the metric. As shown in Figure 1, MAD3QN achieved rapid reward improvement after an initial slow phase, stabilizing around 43.5 after 1500 episodes. In contrast, D3QN exhibited slower learning and greater volatility, plateauing near 20.5. This demonstrates MAD3QN’s superior efficiency and stability in multi-agent settings, essential for scalable drone technology solutions.
Effectiveness in Multi-UAV Scenarios
Tests were conducted with 2, 3, and 5 UAVs in environments containing static obstacles. The results confirmed that all UAVs successfully avoided collisions and reached their goals with smooth trajectories. Key metrics, including minimum inter-UAV distances and distances to obstacles, remained above the safety threshold $R_c$. Trajectory plots illustrated in Figures 2-4 highlight the method’s ability to handle increasing UAV densities without performance degradation. For example, in the 5-UAV case, the average path length was 815.79 meters, with a completion time of 57.9 seconds, underscoring the scalability of the approach for Unmanned Aerial Vehicle deployments.
Performance Metrics and Comparative Analysis
Monte Carlo simulations with 200 independent runs were used to evaluate success rate, average flight distance, and average flight time across different UAV counts (2 to 6). The results, summarized in Table 2, show that MAD3QN maintained a success rate of at least 98% in all cases, with flight times ranging from 52.7 to 58.6 seconds. As UAV density increased, path lengths slightly grew due to more frequent heading adjustments, but the success rate remained high, demonstrating robustness.
| Number of UAVs | Success Rate (%) | Flight Distance (m) | Flight Time (s) |
|---|---|---|---|
| 2 | 99.5 | 804.20 | 52.7 |
| 3 | 99.5 | 806.52 | 55.9 |
| 4 | 99.0 | 811.53 | 56.3 |
| 5 | 98.5 | 815.79 | 57.9 |
| 6 | 98.0 | 819.83 | 58.6 |
Comparative analysis with baseline methods in a 6-UAV scenario (Table 3) revealed that MAD3QN outperformed others, achieving a 98% success rate compared to 83.5% for D3QN, 86.5% for MADQN, and 79.5% for IDQN. Additionally, MAD3QN yielded shorter flight distances (819.83 m) and times (58.6 s), highlighting its efficiency in complex airspace management for Unmanned Aerial Vehicle systems.
| Method | Success Rate (%) | Flight Distance (m) | Flight Time (s) |
|---|---|---|---|
| MAD3QN | 98.0 | 819.83 | 58.6 |
| D3QN | 83.5 | 832.63 | 62.3 |
| MADQN | 86.5 | 839.42 | 63.1 |
| IDQN | 79.5 | 847.56 | 69.2 |
| Random Action | 3.0 | 4932.8 | 436.2 |
High-Fidelity Simulation Platform Testing
The trained MAD3QN model was further validated in a high-fidelity Unity3D-based simulator, incorporating real-world terrain data from Chengdu, China. In a scenario with 5 UAVs at the same altitude, the model successfully guided all agents to their goals while avoiding collisions with buildings and other UAVs. Trajectory visualizations from top-down and oblique perspectives (Figures 5-6) confirm the method’s practicality and transferability to real-world drone technology applications, emphasizing its readiness for deployment in Unmanned Aerial Vehicle operations.
Conclusion
This paper presents a novel MAD3QN-based framework for multi-UAV cooperative collision avoidance in low-altitude airspace. The approach addresses dynamic coupling challenges through a sector-based observation model, a composite reward function, and a distributed training architecture with shared policies. Extensive simulations demonstrate that MAD3QN achieves over 98% success rates across varying UAV densities, outperforming existing methods in terms of safety, efficiency, and scalability. The algorithm’s ability to maintain smooth trajectories and adhere to constraints underscores its potential for real-world drone technology implementations. Future work will focus on extending the model to 3D navigation, incorporating altitude control, and conducting hardware-in-the-loop tests to bridge the gap between simulation and actual Unmanned Aerial Vehicle deployments. By advancing the state of multi-agent reinforcement learning, this research contributes to the safe and efficient integration of UAVs into increasingly crowded airspace, paving the way for broader adoption of drone technology in commercial and industrial applications.