The coordinated operation of multi-agent systems, particularly unmanned aerial vehicle (UAV) swarms or formations, represents a frontier in autonomous systems with transformative potential for applications ranging from precision agriculture and infrastructure inspection to coordinated search & rescue and aerial logistics. At the heart of enabling these complex, collaborative missions lies the critical challenge of drone formation path planning. This task extends far beyond simple point-to-point navigation for a single agent; it demands the simultaneous generation of safe, efficient, and spatially coordinated trajectories for multiple UAVs operating in potentially cluttered and dynamic three-dimensional environments. The core objectives are multifaceted: each UAV must avoid static and dynamic obstacles, maintain a safe separation from its peers within the drone formation to prevent collisions, and converge upon a mission objective, all while optimizing for performance metrics such as total path length, energy expenditure, and trajectory smoothness.
Traditional approaches to path planning, while foundational, often exhibit significant limitations when applied to the drone formation context. Sampling-based methods like Rapidly-exploring Random Trees (RRT) can struggle with computational efficiency and optimality guarantees in dense environments. Classical optimization techniques, including variants of A* and evolutionary algorithms, may face scalability issues as the number of agents increases. The Artificial Potential Field (APF) method has been widely adopted for its conceptual elegance and computational simplicity, modeling the environment with attractive forces toward the goal and repulsive forces from obstacles. However, its application to drone formation planning is hampered by well-documented pitfalls: the frequent occurrence of local minima where the net force on a UAV sums to zero, trapping it before reaching the goal; oscillatory or jittery paths near obstacles or formation mates due to abrupt force changes; and a general lack of global path optimality, as the method is inherently reactive and myopic.
To overcome these challenges, this work presents a novel, hierarchical framework that synergistically combines the global, learning-based optimization prowess of Deep Reinforcement Learning (DRL) with the local, reactive refinement of a significantly enhanced APF method. Our core philosophy is a divide-and-conquer strategy tailored for drone formation operations. We designate one agent, the leader, whose primary responsibility is to discover a globally efficient, obstacle-free path from start to goal. This task is delegated to a sophisticated DRL agent, specifically a Double Deep Q-Network (DDQN) augmented with prioritized experience replay, which learns an optimal policy through interaction with a simulated 3D environment. Subsequently, the remaining UAVs—the followers—execute the local coordination and fine-grained obstacle avoidance. Their motion is governed by an improved APF algorithm that has been fundamentally redesigned to eliminate oscillations and improve convergence. Crucially, the followers use the leader’s DRL-generated trajectory as a dynamic “virtual target” or guiding path, ensuring the entire drone formation moves in a cohesive manner. This paper details the mathematical formulation, algorithmic design, and comprehensive simulation-based validation of this hybrid approach, demonstrating its superiority in generating smooth, safe, and efficient paths for collaborative UAV teams.
DDQN-Based Leader Path Planning for Global Trajectory Generation
The first layer of our hierarchical framework addresses the global path planning problem for the leader UAV. We formulate this as a Markov Decision Process (MDP) solved by a model-free, value-based Deep Reinforcement Learning agent. The objective is to learn a policy $$π^*(s)$$ that maximizes the expected cumulative discounted reward, guiding the leader from its start position $$P_0^L$$ to the goal $$P_G$$ in a 3D workspace cluttered with cylindrical obstacles.
The state space $$S$$ for the leader is discretized using a 3D grid. A state $$s_t \in S$$ at time $$t$$ is defined as:
$$ s_t = \{ P_t^L, P_G, \mathcal{O} \} $$
where $$P_t^L = (x_t^L, y_t^L, z_t^L)$$ is the leader’s current grid cell position, $$P_G$$ is the goal cell, and $$\mathcal{O}$$ is a perceptual representation of nearby obstacle cells within a sensor range. The action space $$A$$ is discrete, comprising 26 possible movements to adjacent and diagonally adjacent cells in the 3D grid, enabling omnidirectional exploration. The reward function $$r_t$$ is meticulously designed to incentivize goal-directed behavior and penalize failures:
$$ r_t = \underbrace{\kappa (\| P_t^L – P_G \|_2 – \| P_{t+1}^L – P_G \|_2 ) – \delta}_{r_{\text{act}}} + r_{\text{env}}(s_{t+1}) $$
where $$r_{\text{act}}$$ is the action reward with distance scaling factor $$\kappa$$ and a small step penalty $$\delta$$. The environment reward $$r_{\text{env}}$$ is defined as:
$$
r_{\text{env}}(s_{t+1}) =
\begin{cases}
+200, & \text{if } P_{t+1}^L = P_G \quad \text{(Goal reached)} \\
-50, & \text{if } P_{t+1}^L \in \mathcal{O} \text{ or out of bounds} \\
0, & \text{otherwise}
\end{cases}
$$
To learn the optimal action-value function $$Q^*(s,a)$$, we employ a Double Deep Q-Network (DDQN). This architecture mitigates the overestimation bias prevalent in standard DQN by decoupling action selection from value estimation. The DDQN uses two neural networks: an online network with parameters $$\theta$$ and a target network with parameters $$\theta^-$$. The update target for the online network is:
$$ y_t^{\text{DDQN}} = r_t + \gamma Q_{\theta^-}(s_{t+1}, \arg\max_{a’} Q_{\theta}(s_{t+1}, a’)) $$
where $$\gamma$$ is the discount factor. The network is trained to minimize the loss:
$$ \mathcal{L}(\theta) = \mathbb{E}_{(s,a,r,s’) \sim D} \left[ ( y_t^{\text{DDQN}} – Q_{\theta}(s, a) )^2 \right] $$
where $$D$$ is the experience replay buffer.
We integrate a Prioritized Experience Replay (PER) mechanism to accelerate learning. Transitions are stored with a priority $$p_i = |\delta_i| + \epsilon$$, where $$\delta_i$$ is the Temporal-Difference (TD) error. During training, transitions are sampled with probability $$P(i) = p_i^\alpha / \sum_k p_k^\alpha$$, where $$\alpha$$ controls the prioritization strength. Importance-sampling weights $$w_i = (N \cdot P(i))^{-\beta}$$ are applied to correct the bias introduced by non-uniform sampling. The synergy of DDQN and PER enables the leader agent to efficiently learn a robust policy that yields a globally optimal (or near-optimal) collision-free path, denoted as the waypoint sequence $$\mathcal{P}^L = \{P_0^L, P_1^L, …, P_G\}$$. This path serves as the foundational trajectory for the entire drone formation.
| Parameter Category | Parameter | Value / Setting |
|---|---|---|
| DDQN & Training | State Space Dimension | Grid-based (e.g., 80x80x30) |
| Action Space Size | 26 discrete movements | |
| Reward Discount Factor (γ) | 0.99 | |
| Experience Replay Buffer Size | 100,000 | |
| Batch Size | 256 | |
| Target Network Update Frequency | Every 50 episodes | |
| Leader Path Metrics | Start Point | (0, 0, 3) m |
| Goal Point | (72, 73, 23) m | |
| Workspace Bounds | x,y ∈ [-10,80] m; z ∈ [0,30] m |
Enhanced Artificial Potential Field for Follower Coordination and Local Refinement
While the leader’s path provides a global guide, the follower UAVs within the drone formation must solve a distinct local planning problem. They need to: 1) track the leader’s path while maintaining a desired formation geometry (e.g., a V-shape or diamond), 2) avoid static obstacles independently, as their positions differ from the leader’s, and 3) maintain safe separation from all other drones in the formation to prevent intra-swarm collisions. The classic APF is a natural candidate for this local, reactive control due to its low computational overhead. However, its standard formulation suffers from path oscillations and inefficiency near force field boundaries. We propose a comprehensive enhancement through the adaptive tuning of its core parameters.
The total artificial force acting on follower $$j$$ at position $$P^j$$ in a formation of $$n$$ drones is a superposition of several forces:
$$ \vec{F}_{\text{total}}^j = \vec{F}_{\text{att}}^j + \sum_{i \in \text{Obstacles}} \vec{F}_{\text{rep,obs}}^{i,j} + \sum_{\substack{m=1 \\ m \neq j}}^{n} \vec{F}_{\text{rep,uav}}^{j,m} $$
The adaptive attractive force towards the follower’s assigned sub-goal (a point on the leader’s path, offset for formation shape) is given by:
$$ \vec{F}_{\text{att}}^j = -k_{\text{att}}(d_g^j) \cdot d_g^j \cdot \frac{\partial d_g^j}{\partial P^j} $$
where $$d_g^j = \| P^j – P_{\text{subgoal}}^j \|_2$$. Crucially, the attractive gain $$k_{\text{att}}$$ is no longer constant but adapts based on the distance to the goal:
$$ k_{\text{att}}(d_g^j) = k_{\text{att}_0} \cdot \left( 1 + \rho \cdot \frac{d_g^j}{d_g^j + \eta} \right) $$
Here, $$k_{\text{att}_0}$$ is a base gain, $$\rho$$ controls the strength of distance-based augmentation, and $$\eta$$ is a smoothing factor. This design ensures a stronger pull when the drone is far from its target, accelerating convergence, and a weaker, finer pull when nearby, reducing overshoot and oscillation.
The adaptive obstacle repulsive force from obstacle $$i$$ is:
$$
\vec{F}_{\text{rep,obs}}^{i,j} =
\begin{cases}
k_{\text{rep,obs}}(d_{\text{obs}}^{i,j}) \cdot \left( \frac{1}{d_{\text{obs}}^{i,j}} – \frac{1}{d_{\text{obs}}^0} \right) \cdot \frac{1}{(d_{\text{obs}}^{i,j})^2} \cdot \frac{\partial d_{\text{obs}}^{i,j}}{\partial P^j}, & \text{if } d_{\text{obs}}^{i,j} \le d_{\text{obs}}^0 \\
0, & \text{if } d_{\text{obs}}^{i,j} > d_{\text{obs}}^0
\end{cases}
$$
where $$d_{\text{obs}}^{i,j}$$ is the distance to the obstacle, and $$d_{\text{obs}}^0$$ is its influence radius. The adaptive repulsive gain is:
$$ k_{\text{rep,obs}}(d_{\text{obs}}^{i,j}) = k_{\text{rep}_0} \cdot \left( \frac{d_{\text{obs}}^0}{d_{\text{obs}}^{i,j} + \tau} \right)^\xi $$
where $$k_{\text{rep}_0}$$ is the base repulsive gain, $$\tau$$ is a small constant preventing division by zero, and $$\xi$$ is an exponent that sharpens the repulsive field’s distance dependency. This makes the repulsion intense when very close to an obstacle for safety and decays rapidly to avoid undue influence at a distance.
The inter-agent repulsive force for collision avoidance within the drone formation follows a similar adaptive structure, ensuring drones maintain a safe distance $$d_{\text{uav}}^0$$ from each other.
Finally, we introduce an adaptive step size for integration. The movement step $$\Delta s^j$$ for follower $$j$$ in the direction of the total force is dynamically adjusted:
$$ \Delta s^j = \Delta s_0 \cdot \frac{d_g^j}{d_g^j + \mu \cdot d_{\text{min,obs}}^j} $$
where $$\Delta s_0$$ is the nominal step size, $$\mu$$ is a weighting factor, and $$d_{\text{min,obs}}^j$$ is the distance to the nearest obstacle. This logic reduces the step size when navigating close to hazards for precise, smooth maneuvering and increases it in open spaces for faster progress, directly combating the oscillatory behavior typical of fixed-step APF.
| Parameter | Symbol | Description | Typical Value / Role |
|---|---|---|---|
| Base Attractive Gain | $$k_{\text{att}_0}$$ | Strength of goal attraction | 2.0 |
| Distance Augmentation Factor | $$\rho$$ | Controls distance-based gain increase | 4.5 |
| Attractive Smoothing Factor | $$\eta$$ | Prevents excessive gain near goal | 3.0 |
| Base Repulsive Gain | $$k_{\text{rep}_0}$$ | Strength of obstacle/agent repulsion | 20.0 |
| Repulsive Exponent | $$\xi$$ | Sharpens repulsive field distance decay | 1.5 |
| Obstacle Influence Radius | $$d_{\text{obs}}^0$$ | Range of obstacle repulsive field | 5.0 m |
| Agent Safety Radius | $$d_{\text{uav}}^0$$ | Minimum allowed inter-agent distance | 3.0 m |
| Nominal Step Size | $$\Delta s_0$$ | Base integration step | 0.8 m |
| Obstacle Weight for Step Size | $$\mu$$ | Controls step reduction near obstacles | 28.0 |
Integrated Framework for Multi-UAV Formation Path Planning
The full power of the proposed methodology is realized in the integrated framework that orchestrates the leader and followers into a cohesive drone formation. We adopt a virtual leader-follower structure. The DDQN-trained policy generates the path for the virtual leader. This path is not necessarily physically occupied by a drone but serves as a spatial reference trajectory $$\mathcal{P}^L$$.
Each physical follower $$j$$ is assigned a formation offset vector $$\vec{\delta}^j$$ relative to the virtual leader’s current position. The follower’s instantaneous sub-goal $$P_{\text{subgoal}}^j$$ is calculated as the sum of the virtual leader’s current waypoint and its formation offset. The enhanced APF controller for follower $$j$$ then uses this $$P_{\text{subgoal}}^j$$ as the target for its adaptive attractive force. The repulsive components of its force field handle both static obstacles and other physical followers. The position update for a follower in discrete time is:
$$ P^j(t+1) = P^j(t) + \Delta s^j(t) \cdot \frac{\vec{F}_{\text{total}}^j(t)}{\|\vec{F}_{\text{total}}^j(t)\|_2} $$
This process ensures that all followers are simultaneously attracted to their formation-assigned positions relative to the moving virtual leader and repelled from obstacles and each other. The adaptive mechanisms within the APF guarantee that this tracking and local avoidance occur smoothly and without oscillations. The algorithm proceeds until all followers, guided by the virtual leader’s path, have converged to their final offsets at the goal location, completing the drone formation mission. The pseudocode below summarizes the integrated planning process.
Simulation Results and Performance Analysis
The proposed hybrid framework was rigorously evaluated in a comprehensive 3D simulation environment featuring multiple cylindrical obstacles. We compared the performance of three distinct strategies for a drone formation consisting of one (virtual) leader and five followers arranged in a triangular pattern.
- Baseline APF: Both leader and followers use the standard, non-adaptive APF.
- Hybrid (DDQN + Standard APF): Leader path from DDQN, followers use standard APF.
- Proposed Method (DDQN + Enhanced APF): Leader path from DDQN, followers use our enhanced APF with adaptive parameters.
The primary performance metrics were Path Length ($$L = \sum \|P_{v+1} – P_v\|_2$$) and Path Smoothness ($$S = \frac{1}{M-2}\sum_{v=1}^{M-2} \text{angle}(\vec{P_vP_{v+1}}, \vec{P_{v+1}P_{v+2}})$$), where lower values for both are desirable. A smoothness value below 10°/m is typically considered acceptable for stable UAV flight.
| Method | Agent | Average Path Length (m) | Average Path Smoothness (°/m) | Notes |
|---|---|---|---|---|
| Baseline APF | Leader | 142.19 | 44.26 | Long, oscillatory path; prone to local minima. |
| Followers (Avg.) | ~155-170 | 38.5 – 52.1 | Very high smoothness indicates severe oscillation. | |
| Hybrid (DDQN + Std. APF) | Leader (DDQN) | 111.45 | 3.91 | DDQN provides a globally efficient, smooth leader path. |
| Followers (Avg.) | ~125-140 | 8.7 – 16.4 | Improved over baseline but still above ideal smoothness threshold. | |
| Proposed (DDQN + Enh. APF) | Leader (DDQN) | 111.45 | 3.91 | Retains the optimal leader trajectory. |
| Followers (Avg.) | ~114 | ~2.3 | Path length converges nearly to leader’s; smoothness is excellent and superior to leader’s. |
The results are conclusive. The Baseline APF method performs poorly, generating long, highly oscillatory paths for all agents, rendering it unsuitable for practical drone formation deployment. The Hybrid method shows a major improvement in the leader’s path quality due to DDQN, and follower paths improve as a consequence. However, the followers’ use of standard APF still results in unnecessary detours and sub-optimal smoothness.
Our Proposed Method delivers the best overall performance. It retains the globally optimal, smooth leader path generated by DDQN. Most significantly, the followers utilizing the enhanced APF exhibit dramatic improvement. Their average path length (≈114 m) nearly matches the leader’s, indicating highly efficient trajectory tracking without wasteful deviations. The average path smoothness of approximately 2.3 °/m is not only well within the acceptable range but is also lower than the leader’s smoothness, indicating that the local adaptive controller successfully refines and smoothens the tracking path. This demonstrates the framework’s ability to achieve synergistic optimization: global optimality from DRL and local smoothness/coordination from the enhanced APF, fulfilling the core requirements for effective drone formation navigation.
Conclusion and Future Work
This paper presented a novel, hierarchical framework for solving the complex problem of 3D path planning for a collaborative drone formation. By strategically decomposing the task, we combined the strengths of two complementary paradigms. A DDQN agent with prioritized experience replay was employed to generate a globally optimal, obstacle-free trajectory for a virtual leader, effectively solving the long-horizon planning problem and overcoming the local minima issue. For the follower agents responsible for formation-keeping and fine-grained obstacle avoidance, we developed a significantly enhanced Artificial Potential Field algorithm. Through the introduction of adaptive mechanisms for tuning the attractive gain, repulsive gain, and motion step size based on real-time environmental cues, we successfully eliminated the pathological oscillations and inefficiencies inherent in the classical APF method.
The integrated system, operating on a virtual leader-follower principle, demonstrated superior performance in simulation. The resulting paths for the entire drone formation were not only collision-free but also exhibited near-optimal length and exceptional smoothness, key factors for the practical deployment of UAV swarms in real-world scenarios. The framework provides a robust solution that balances global planning intelligence with local reactive agility.
Future research directions are multifaceted. First, extending the DDQN policy to handle dynamic obstacles would significantly enhance the framework’s applicability to uncertain environments. Second, investigating multi-agent reinforcement learning (MARL) approaches could allow for more emergent and flexible cooperative behaviors beyond the rigid leader-follower hierarchy, potentially leading to more resilient drone formation strategies. Finally, formal robustness and safety guarantees, perhaps through integration with control barrier functions, would be a critical step toward certifying such systems for use in safety-critical airspace.