We address the critical challenge of spatio-temporal cooperative trajectory planning for multiple UAV drone systems operating within a shared low-altitude airspace under strict flight schedule constraints. In the burgeoning low-altitude economy, UAV drone swarms are increasingly deployed for logistics, urban air mobility, and emergency response. These UAV drones must navigate from designated vertiports to destinations, adhering to a pre-defined timetable while avoiding static obstacles (e.g., buildings, no-fly zones) and dynamic conflicts with other airborne UAV drones. The asynchronous takeoff nature of these missions introduces a hard temporal coupling that conventional path planning methods struggle to handle efficiently. To tackle this, we propose an improved Multi-Agent Deep Deterministic Policy Gradient algorithm guided by an Artificial Potential Field (APF-MADDPG). Our approach transforms the sparse terminal rewards typical in deep reinforcement learning into a dense, physically motivated feedback signal. This study demonstrates that the proposed method significantly accelerates convergence, enhances safety, and improves overall mission success rates in highly dynamic and stochastic environments.
1. Introduction
The rapid advancement of the low-altitude economy has propelled the large-scale deployment of UAV drone swarms for commercial and public services. Unlike traditional structured airspace management that relies on fixed airways, the free-flight concept offers greater airspace utilization and mission flexibility. However, enabling safe and efficient autonomous navigation for a fleet of UAV drones under tight temporal and spatial constraints remains a significant technical barrier. The core problem involves planning collision-free trajectories for each UAV drone from its start point to its target point, optimizing for flight efficiency and energy consumption while strictly adhering to a flight schedule schedule.
Classical path planning algorithms, such as A*, Artificial Potential Field (APF), Rapidly-exploring Random Trees (RRT), and Velocity Obstacle (VO), have laid solid foundations for static and dynamic obstacle avoidance. For multi-UAV drone coordination, approaches based on Distributed Model Predictive Control (DMPC) and Reciprocal Velocity Obstacles (ORCA) have been explored. However, these methods often struggle with scalability, real-time performance, and handling the non-stationary environment inherent in multi-agent systems where the behavior of one UAV drone directly affects the state of others.
Deep Reinforcement Learning (DRL) has emerged as a powerful paradigm for autonomous navigation due to its ability to learn end-to-end policies from raw sensor data. Single-agent algorithms like DQN and DDPG have been successfully applied to individual UAV drone obstacle avoidance. For multi-UAV drone collaboration, Lowe et al. introduced MADDPG, which utilizes a “centralized training with decentralized execution” (CTDE) framework to mitigate the non-stationarity problem. While effective, standard MADDPG suffers from several fundamental issues in complex urban low-altitude scenarios:
- Sparse Rewards: The agent only receives meaningful feedback upon reaching the goal or crashing, leading to extremely low exploration efficiency in high-dimensional state spaces.
- Instability in Training: Multi-agent training is prone to policy collapse and performance fluctuations, especially when the number of UAV drones or environmental complexity increases.
- Temporal Coupling: The asynchronous takeoff of UAV drones creates a time-varying state distribution that pure data-driven networks fail to model effectively, often leading to prediction lag and unsafe maneuvers.
To overcome these limitations, we propose the APF-MADDPG algorithm, which integrates physical gradient information from an Artificial Potential Field into the learning process. The main contributions of this work are threefold:
- Dense Reward Shaping via APF: We design a hybrid reward function that converts sparse terminal rewards into continuous, distance-sensitive dense rewards. The repulsive field provides a heuristic hazard warning, while the attractive field guides the UAV drone towards its goal, dramatically improving early exploration efficiency.
- Training Stability Mechanism: We introduce a Priority Experience Replay (PER) mechanism and a novel policy rollback protection mechanism. By monitoring the moving average success rate, the rollback mechanism can detect and revert catastrophic performance drops, significantly enhancing the robustness of the learning process.
- Spatio-Temporal Constraint Handling: We formulate a hard constraint model that couples the 2D spatial domain with the takeoff timing. The APF provides a deterministic physical prior that helps the policy network handle the sudden appearance of a new UAV drone entering the airspace, mitigating the “data distribution drift” problem.
2. Problem Formulation
We consider a class of multi-UAV drone cooperative trajectory planning problems within a shared low-altitude airspace. The task environment is defined as a 2D Euclidean workspace W ∈ R², containing a set of M vertiports V = {v₁, v₂, …, v_M} and K static obstacles O = {o₁, o₂, …, o_K}. A fleet of N UAV drones U = {u₁, u₂, …, u_N} operate according to a pre-defined flight schedule. Each UAV drone u_i is assigned a mission tuple M_i:
$$
M_i = (p_{start}^i, p_{goal}^i, t_{dep}^i, v_{max}^i)
$$
Where p_start^i and p_goal^i ∈ W are the start and goal positions, t_dep^i is the scheduled departure time, and v_max^i is the maximum speed. The core of the problem is to find a set of collision-free trajectories for all UAV drones that satisfy the following coupled constraints:
- Temporal Constraint: Each UAV drone must strictly adhere to its scheduled departure time t_dep^i.
- Static Obstacle Constraint: The trajectory of each UAV drone must avoid all static obstacles.
- Dynamic Collision Avoidance: The trajectories of all UAV drones must be mutually collision-free.
The asynchronous departure creates a dynamic conflict zone that changes over time, making the problem a classic spatio-temporal coupling challenge. The goal is to minimize the total mission cost, typically defined as the sum of flight time or path length, while guaranteeing safety.
3. APF-Guided Multi-Agent Deep Reinforcement Learning
3.1 Markov Decision Process Formulation
The multi-UAV drone cooperative planning problem is modeled as a multi-agent Markov Decision Process (MDP). The MDP is defined by the tuple (S, A, P, R, γ), where S is the joint state space, A = A₁ × A₂ × … × A_N is the joint action space, P is the state transition probability, R is the reward function, and γ is the discount factor. The objective for each UAV drone i is to learn a policy π_i that maximizes the expected cumulative discounted reward:
$$
\pi_i^* = \arg \max_{\pi_i} \mathbb{E}_{\pi_i} \left[ \sum_{t=0}^{T} \gamma^t r_i(s_t, a_t) \right]
$$
To address the non-stationarity inherent in multi-agent environments, we adopt the centralized training with decentralized execution (CTDE) framework of MADDPG. In this framework, each agent i maintains a policy network (Actor) μ_i and a value network (Critic) Q_i. The Critic network is trained using the joint observations and actions of all agents, providing a stable gradient for policy improvement.
3.2 State and Action Space Design
We design a comprehensive state space for each UAV drone i. The local observation vector O_i includes four key components: own state (O_UAV), teammate information (O_team), goal information (O_target), and obstacle information (O_obs). All features are normalized to stabilize the training process.
| Component | Description | Formula |
|---|---|---|
| Own State (OUAV) | Normalized position and velocity | OUAV = [x/lw, y/ll, vx/vm, vy/vm] |
| Teammate Info (Oteam) | Relative position to other UAV drones | Oteam = [(xj-xi)/lw, (yj-yi)/ll] |
| Goal Info (Otarget) | Distance and bearing to the target | Otarget = [||pgoal-p||/√(lw²+ll²), atan2(Δy, Δx)] |
| Obstacle Info (Oobs) | Distance to the nearest obstacle | Oobs = [||pobs-p||/√(lw²+ll²)] |
lw and ll represent the width and length of the workspace, and vm is the maximum velocity. The action space for each UAV drone is a 2D continuous acceleration vector:
$$
a_i(t) = [u_{x}^i, u_{y}^i]^T
$$
The state of each UAV drone is updated using a simple kinematic model:
$$
v_i(t+1) = v_i(t) + a_i(t) \cdot \Delta t
$$
$$
p_i(t+1) = p_i(t) + v_i(t+1) \cdot \Delta t
$$
3.3 Hybrid Reward Function with Artificial Potential Field
The key innovation of our approach is the construction of a dense hybrid reward function that integrates the principles of an Artificial Potential Field (APF). The APF decomposes the reward into an attractive component, guiding the UAV drone towards its goal, and a repulsive component, pushing it away from obstacles and other UAV drones. This transforms the sparse terminal reward problem into a continuous, dense signal, dramatically improving early-stage exploration efficiency. The total reward for UAV drone i at time t is:
$$
R_i(t) = r_{att} + r_{rep} + r_{task}
$$
Attractive Reward (ratt): This reward encourages the UAV drone to move towards its goal. It is a piecewise function based on the change in distance to the goal Δd, providing stronger guidance when the agent is far away.
$$
r_{att} =
\begin{cases}
\lambda_1 \cdot \Delta d & \text{if } \Delta d \leq d_1 \\
\lambda_2 \cdot \Delta d & \text{if } d_1 < \Delta d \leq d_2 \\
\lambda_3 \cdot \Delta d & \text{if } \Delta d > d_2
\end{cases}
$$
where λ₁ > λ₂ > λ₃ are monotonically decreasing gain coefficients, and d₁, d₂ are distance thresholds.
Repulsive Penalty (rrep): This penalty provides a continuous, distance-sensitive negative signal when the UAV drone approaches an obstacle or another UAV drone. The closer the agent gets to a hazard, the stronger the penalty, providing an early warning before a collision occurs.
$$
r_{rep} =
\begin{cases}
-\eta \left( \frac{1}{d_{min}} – \frac{1}{d_0} \right)^2 & \text{if } d_{min} \leq d_0 \\
0 & \text{if } d_{min} > d_0
\end{cases}
$$
where η is the repulsive gain coefficient, d_min is the distance to the nearest hazard (obstacle or other UAV drone), and d₀ is the radius of influence.
Task Reward (rtask): This component provides sparse terminal rewards for reaching the goal and penalties for collisions and boundary violations.
$$
r_{task} =
\begin{cases}
R_{goal} & \text{if } ||p_i – p_{goal}|| < \zeta_{range} \\
R_{crash} & \text{if collision occurs} \\
R_{boundary} & \text{if out of bounds} \\
0 & \text{otherwise}
\end{cases}
$$
where ζrange is the arrival radius, and Rgoal, Rcrash, and Rboundary are constant reward/penalty values.
| Parameter | Symbol | Value |
|---|---|---|
| Goal Reward | Rgoal | +100 |
| Collision Penalty | Rcrash | -200 |
| Boundary Penalty | Rboundary | -200 |
| APF Repulsive Gain | η | -6 |
| APF Influence Radius | d₀ | 8 m |
| Attraction Gain 1 | λ₁ | 15 |
| Attraction Gain 2 | λ₂ | 10 |
| Attraction Gain 3 | λ₃ | 8 |
3.4 Training Stability Mechanisms
To further enhance the robustness and convergence speed of the training process, we integrate two key mechanisms into the MADDPG framework: Priority Experience Replay (PER) and a policy rollback protection mechanism.
Priority Experience Replay (PER): Standard uniform experience replay treats all samples with equal importance. PER prioritizes samples with a larger Temporal Difference (TD) error, which are considered more informative for learning. The probability of sampling a transition j is:
$$
P(j) = \frac{p_j^\alpha}{\sum_{k} p_k^\alpha}
$$
where pj = |δj| + ε is the priority of the j-th transition, δj is the TD error, and α ∈ [0,1] controls the degree of prioritization. This mechanism ensures that the UAV drone learns more from critical, unexpected situations.
Policy Rollback Protection: In multi-agent training, the policy of one agent can suddenly collapse as other agents’ policies evolve. To mitigate this, we introduce a rollback mechanism based on a sliding window average of the mission success rate St. If the current success rate drops significantly below the historical best, the policy network is rolled back to a previously saved stable state. The rollback condition is:
$$
R_t =
\begin{cases}
True & \text{if } S_t < S_{best} \cdot k \\
False & \text{otherwise}
\end{cases}
$$
where Sbest is the historical best average success rate, and k ∈ (0,1) is a collapse detection threshold. This mechanism effectively prevents catastrophic forgetting and performance fluctuation.
4. Spatio-Temporal Constraint Model
The asynchronous takeoff of UAV drones creates a unique challenge: the state distribution of the environment is non-stationary even from the perspective of a single agent, as new UAV drones suddenly appear at their scheduled times. We model this as a hard constraint:
$$
\begin{aligned}
&p_i(t) = 0 \quad \text{if } t < t_{dep}^i \\
&\text{Collision Detection Active Only if } t \ge t_{dep}^i \land t \ge t_{dep}^j \\
&\text{Mission Complete if } ||p_i(T) – p_{goal}^i|| < \zeta_{range}
\end{aligned}
$$
Before their scheduled departure, UAV drones are treated as static objects, not as dynamic agents. This prevents false collision detections. When a UAV drone takes off, it transitions from a static obstacle to a dynamic agent, causing a sudden shift in the environmental state. Our APF-MADDPG algorithm handles this well because the APF provides a deterministic, continuous physical gradient that immediately guides the newly appeared UAV drone away from hazards, unlike pure data-driven methods that suffer from prediction lag.
5. Experiments and Results
5.1 Experimental Setup
We conducted extensive experiments in a simulated 100×100 m² 2D environment. The fleets consisted of 5 to 10 UAV drones with different start and goal pairs. We tested the algorithm against several baseline methods:
- IDDPG: Independent DDPG, treating other UAV drones as environmental obstacles.
- APF-IDDPG: Independent DDPG with our APF reward shaping, used to isolate the effect of the multi-agent framework.
- MADDPG: The classic multi-agent algorithm without our enhancements.
- MAPPO & MATD3: State-of-the-art (SOTA) multi-agent algorithms.
- CBF-MADDPG: A safe reinforcement learning baseline using a Control Barrier Function to filter actions.
All algorithms were trained using the same hyperparameters, and we performed multiple random-seed experiments to ensure statistical validity.
| Parameter | Symbol | Value |
|---|---|---|
| Workspace Size | lw × ll | 100 × 100 m² |
| Max Speed | vmax | 5 m/s |
| Time Step | Δt | 0.5 s |
| Discount Factor | γ | 0.95 |
| Buffer Size | B | 50000 |
| Batch Size | N | 1024 |
| Actor Learning Rate | αactor | 1×10⁻⁴ |
| Critic Learning Rate | αcritic | 5×10⁻⁴ |
5.2 Performance Evaluation
We evaluated the algorithms using a multi-dimensional performance index system comprising effectiveness, collaboration efficiency, safety, and timeliness. The key metrics are defined in the table below.
| Dimension | Metric | Definition |
|---|---|---|
| Effectiveness | Task Completion Rate | SR = (1/N) Σ I(UAV drone i succeeds) |
| Effectiveness | Global Success Rate | GSR = (1/W) Σ SR over a window |
| Collaboration Efficiency | Trajectory Elongation Rate | E = (1/|Ω|) Σ (Lact – Lstr) / Lstr |
| Safety | Collision Rate | Rcol = (1/N) Σ I(collision) |
| Safety | Threat Frequency | F = (1/N) Σ Σ I(dmin(t) < dsafe) |
| Timeliness | Convergence Episodes | Minimum episode where GSR ≥ 0.95 over M consecutive episodes |
5.3 Comparison with Baseline Algorithms
We first compared APF-MADDPG with MADDPG, APF-IDDPG, and IDDPG. The results from multiple seeds are summarized below.
| Metric | APF-MADDPG (Ours) | MADDPG | APF-IDDPG | IDDPG |
|---|---|---|---|---|
| Average Reward | 4556.2 | 3198.3 | 3351.9 | 2863.5 |
| Task Completion Rate | 0.986 | 0.661 | 0.564 | 0.347 |
| Global Success Rate | 0.978 | 0.085 | 0.060 | 0.005 |
| Trajectory Elongation Rate | 0.041 | 0.074 | 0.118 | 0.085 |
| Average Flight Steps | 43.8 | 36.1 | 47.3 | 47.6 |
| Collision Rate | 0.007 | 0.348 | 0.459 | 0.632 |
The APF-MADDPG algorithm achieved significantly higher global success rates (97.8%) and lower collision rates (0.007) compared to other baselines. The MADDPG and IDDPG algorithms, lacking the APF guidance, failed to achieve consistent multi-UAV drone coordination, resulting in very low global success rates. The convergence speed analysis showed that our algorithm required only 4643 episodes to reach a stable 95% success rate, which is a 59.3% improvement over the standard MADDPG’s 11396 episodes.
We also analyzed the stability of the algorithms using the Interquartile Range (IQR) and Coefficient of Variation (CV). The results, based on 100 independent experiments, are shown below.
| Metric | Statistic | APF-MADDPG | MADDPG | APF-IDDPG | IDDPG |
|---|---|---|---|---|---|
| Average Reward | IQR | 72.1 | 294.5 | 297.3 | 327.2 |
| Average Reward | CV (%) | 1.18 | 6.51 | 6.96 | 9.79 |
| Task Completion Rate | IQR | 0.006 | 0.161 | 0.121 | 0.123 |
| Task Completion Rate | CV (%) | 1.46 | 17.02 | 18.31 | 28.58 |
| Average Flight Steps | IQR | 2.53 | 2.91 | 6.62 | 7.86 |
| Average Flight Steps | CV (%) | 4.46 | 6.17 | 11.12 | 12.76 |
The extremely low IQR and CV values for the APF-MADDPG algorithm demonstrate its remarkable robustness to initial conditions and random noise, confirming that the performance gains are statistically significant and reproducible.
5.4 Comparison with SOTA and Safe RL Baselines
We further compared our algorithm against the SOTA MAPPO and MATD3 algorithms, and the CBF-MADDPG safe RL baseline. The results, shown in the table below, highlight the superiority of our approach in handling complex spatio-temporal constraints.
| Metric | APF-MADDPG (Ours) | MATD3 | MAPPO | CBF-MADDPG |
|---|---|---|---|---|
| Task Completion Rate | 0.986 | 0.542 | 0.410 | 0.247 |
| Global Success Rate | 0.978 | 0.040 | 0.064 | 0 |
| Trajectory Elongation Rate | 0.04 | 0.20 | 0.22 | 0.10 |
| Average Flight Steps | 43.8 | 45.1 | 40.6 | 33.0 |
| Collision Rate | 0.007 | 0.453 | 0.551 | 0.740 |
The pure data-driven SOTA algorithms (MATD3, MAPPO) and the engineer-safe CBF-MADDPG all exhibited poor performance. The CBF-MADDPG, while theoretically safer, failed to find any feasible global solution in these dense, asynchronous scenarios, often getting stuck in local deadlocks. This is because the hard constraint from the CBF is too restrictive in a complex multi-UAV drone environment, while our APF provides a softer, more flexible guidance signal that is learned by the policy.
| Metric | Statistic | APF-MADDPG | MATD3 | MAPPO | CBF-MADDPG |
|---|---|---|---|---|---|
| Task Completion Rate | IQR | 0.006 | 0.214 | 0.221 | 0.153 |
| Task Completion Rate | CV (%) | 1.46 | 18.62 | 23.70 | 15.68 |
| Global Success Rate | IQR | 0.032 | 0.275 | 0.247 | 0.208 |
| Global Success Rate | CV (%) | 1.89 | 37.17 | 27.52 | 15.32 |
| Average Flight Steps | IQR | 2.53 | 3.93 | 2.77 | 4.88 |
| Average Flight Steps | CV (%) | 4.46 | 15.57 | 12.42 | 8.10 |
The high CV values for the baseline algorithms indicate extreme sensitivity to environmental and initial conditions, making them unreliable for real-world deployment. Our APF-MADDPG shows a clear advantage in both average performance and robustness.
5.5 Scalability and Generalization
To test the scalability and generalization of our algorithm, we conducted experiments with larger swarms (8 and 10 UAV drones) and in environments with completely random static obstacle layouts. The results, shown in the table below, demonstrate that our algorithm maintains high performance even as complexity increases.
| Fleet Size | Obstacle Layout | Global Success Rate (%) | Single-Step Inference Time (ms) |
|---|---|---|---|
| 5 | Random | 97.4 | 1.62 |
| 8 | Random | 94.8 | 2.07 |
| 10 | Random | 92.1 | 4.31 |
The global success rate remains above 92% even with 10 UAV drones in a completely randomized environment, showcasing the algorithm’s strong generalization capability and its potential for large-scale deployment.
6. Conclusion
This paper addressed the challenging problem of spatio-temporal cooperative trajectory planning for multi-UAV drone systems operating under strict flight schedules. We proposed the APF-MADDPG algorithm, which integrates an Artificial Potential Field to create a dense, physically meaningful reward signal, significantly improving the efficiency and safety of exploration in complex environments. Our key findings are:
- The APF-guided reward shaping reduced the convergence time by 59.3% and increased the steady-state average reward by 42.5% compared to the standard MADDPG algorithm, effectively solving the sparse reward problem.
- The policy rollback and PER mechanisms provided exceptional robustness, with the algorithm demonstrating high stability (CV as low as 1.18% for average reward) against random initialization and environmental noise.
- Our algorithm significantly outperformed both pure data-driven SOTA algorithms (MAPPO, MATD3) and a safe RL baseline (CBF-MADDPG), achieving a 97.8% global success rate and a 0.7% collision rate in challenging asynchronous scenarios. The baselines frequently failed, getting stuck in deadlocks or suffering from high collision rates.
- The algorithm showed strong scalability, maintaining a 92.1% global success rate even with a 10-UAV drone fleet in a completely random obstacle layout, confirming its potential for practical deployment in dense urban airspace.
Future work will focus on extending the framework to full 3D environments, handling non-cooperative targets, and studying the impact of communication delays on the collaborative decision-making process.