The coordinated operation of multiple unmanned aerial vehicles (UAVs), commonly referred to as a drone formation, presents a paradigm shift from single-agent systems, offering enhanced mission capabilities, robustness, and payload capacity. A critical enabler for such collaborative autonomy is intelligent path planning, which must synthesize safe, collision-free trajectories for all agents while maintaining cohesive group behavior amidst complex, often unknown, environments. Traditional and heuristic planning methods frequently rely on precise environmental models, limiting their applicability in dynamic scenarios with unforeseen obstacles. Reinforcement Learning (RL), with its model-free learning paradigm and strong generalization capabilities, has emerged as a powerful alternative. However, translating the high-level objective of drone formation flight—stable structure maintenance coupled with on-the-fly, environmentally-aware deformation—into effective RL rewards remains a significant challenge. This work addresses this gap by introducing a novel dynamic formation reward strategy integrated into a state-of-the-art multi-agent RL framework.
The core problem of drone formation path planning is decomposed into two interrelated sub-problems: multi-agent navigation and dynamic formation control. The former ensures each UAV reaches its destination while avoiding static obstacles, moving threats, and inter-agent collisions. The latter governs the spatial relationship between UAVs, aiming to preserve a predefined geometric structure (e.g., wedge, line) as a default but allowing autonomous, minimal adjustments to facilitate navigation through cluttered spaces. The essence of dynamic formation is not rigid adherence but intelligent elasticity—the spacing between any two UAVs should remain near an optimal value yet be adaptable based on external constraints.
Problem Formulation and System Modeling
Consider a formation of $N$ UAVs operating in a bounded 2D environment. Each UAV $i$ is modeled with a kinematic state $s_i$ and controlled through angular velocity and acceleration actions.
State Space: The state for UAV $i$ encompasses its private navigation data and relative information concerning the formation:
$$ s_i = [x_i, y_i, \psi_i, v_i, \mathbf{d}_i] $$
where $(x_i, y_i)$ is the position, $\psi_i$ is the heading angle, $v_i$ is the speed, and $\mathbf{d}_i$ is an $(N-1)$-dimensional vector containing the Euclidean distances to every other UAV in the drone formation.
Action Space: The control inputs for each UAV are:
$$ A_i = [\omega_i, a_i] $$
where $\omega_i$ is the angular velocity and $a_i$ is the acceleration. These are bounded by $|\omega_i| \leq \omega_{i}^{max}$ and $|a_i| \leq a_{i}^{max}$.
Kinematics: The discrete-time motion model for a time step $\Delta T$ is:
$$
\begin{aligned}
x_i^{t+1} &= x_i^t + v_i^t \cdot \Delta T \cdot \cos(\psi_i^t) \\
y_i^{t+1} &= y_i^t + v_i^t \cdot \Delta T \cdot \sin(\psi_i^t) \\
\psi_i^{t+1} &= \psi_i^t + \omega_i \cdot \Delta T \\
v_i^{t+1} &= v_i^t + a_i \cdot \Delta T
\end{aligned}
$$
The global objective for the drone formation is to guide all UAVs from their initial positions $\mathbf{p}_{init}$ to designated goal positions $\mathbf{p}_{goal}$, minimizing path length and formation distortion while avoiding collisions.
Algorithmic Foundation: MATD3 and Reward Design Philosophy
Our approach is built upon the Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (MATD3) algorithm. MATD3 extends the single-agent TD3 algorithm to multi-agent settings using a centralized-training-with-decentralized-execution (CTDE) paradigm. Each agent (UAV) employs an Actor network that outputs actions based on its local observation. Critic networks, however, are trained with global state-action information during training to properly assess the value of joint actions, mitigating the non-stationarity inherent in multi-agent learning. The twin Q-learning and delayed policy update mechanisms of TD3 help combat overestimation bias, leading to more stable and reliable policy learning—a crucial feature for the complex, cooperative task of drone formation control.
The efficacy of RL hinges on a well-designed reward function $R_i$ that accurately shapes the desired behavior for each agent $i$. Our reward design strategy is hierarchical, starting with foundational navigation rewards and culminating in the proposed dynamic formation reward.
Foundational Navigation Rewards
Before introducing formation constraints, we define rewards essential for basic point-to-point navigation in both obstacle-free and cluttered environments. These sparse and shaping rewards guide the UAV toward its goal efficiently.
| Reward Component | Symbol | Expression / Condition | Purpose |
|---|---|---|---|
| Goal Reached | $r_1$ | $+100$ if $\Delta d \leq d_1$ and heading aligned; scaled lower for misalignment. | Sparse positive reward for mission success. |
| Collision Penalty | $r_2$ | $-10$ upon collision with any obstacle or other UAV ($r_8$). | Strong negative incentive for safety. |
| Distance Progress | $r_3, r_7$ | $r_3 = k(d^{t-1} – d^t)$; $r_7$ applied only if progress is monotonic over a window. | Shaping reward for moving closer to the goal; $r_7$ adds robustness in clutter. |
| Heading Alignment | $r_4, r_6$ | Based on $|\psi_{act} – \psi_{des}|$. $\psi_{des}$ is either direct goal angle ($r_4$) or an obstacle-avoidance optimal angle ($r_6$). | Guides UAV orientation. $r_6$ is critical for obstacle-rich environments. |
| Step Penalty | $r_5$ | $-1$ per step. | Encourages time/distance efficiency. |
The total baseline navigation reward for UAV $i$ in a complex obstacle environment is:
$$ R^{nav}_i = \sum_{k \in \{1,2,5,6,7,8\}} \alpha_k r_k $$
where $\alpha_k$ are weighting coefficients, typically set to 1.
Core Contribution: The Dynamic Formation Reward Function
The defining feature of our drone formation path planning approach is the dynamic formation reward. Its purpose is to instill a dual behavioral objective: maintain a stable formation geometry as a default state, but permit and intelligently control deformation when necessary for navigation.
Let $d_{opt, ij}$ be the desired, optimal distance between UAV $i$ and UAV $j$ defined by the target formation geometry (e.g., in a wedge). Let $d_{ij}$ be their current Euclidean distance. The core pairwise distance reward is designed as an inverted bell curve:
$$
r_{d,ij} = -100 \left( \frac{d_{ij}}{d_{opt, ij}} – 1.1 \right) \left( \frac{d_{ij}}{d_{opt, ij}} – 0.9 \right)
$$
This quadratic function achieves its maximum value of $+1$ when the ratio $d_{ij}/d_{opt, ij} = 1$. The reward decreases symmetrically as the actual distance deviates from the optimal band (between 0.9 and 1.1 times $d_{opt}$), becoming negative for larger deviations. This shape creates a strong, smooth gradient for keeping inter-agent distances on target.
The total dynamic formation reward for UAV $i$ is the sum of its pairwise rewards with all other members of the drone formation:
$$
r_9 = \sum_{j=1, j\neq i}^{N} r_{d,ij}
$$
The complete reward for UAV $i$ in our proposed MATD3-IDFRF (MATD3 with Incorporated Dynamic Formation Reward Function) algorithm becomes:
$$
R_i = R^{nav}_i + \alpha_9 r_9
$$
where $\alpha_9$ is the formation reward weight. This composite reward seamlessly integrates the objectives of navigation ($R^{nav}_i$) and formation cohesion ($r_9$). The drone formation learns to navigate as a cohesive unit because maintaining $r_9$ is most easily achieved when the group moves together. When an obstacle forces agents apart, the algorithm performs a cost-benefit analysis: it accepts a temporary reduction in $r_9$ (formation deformation) to avoid a large penalty from $r_2$ or $r_8$ (collision), guided by the shaping rewards $r_6$ and $r_7$ towards efficient avoidance maneuvers. Once past the obstacle, the reward gradient from $r_9$ naturally pulls the formation back to its optimal geometry.
Experimental Results and Analysis
We evaluate our MATD3-IDFRF algorithm in a complex simulated environment containing static and dynamic obstacles. A formation of $N=5$ UAVs in a wedge configuration must navigate from scattered start points to designated goals. We compare against two strong baselines: the foundational MADDPG algorithm and the standard MATD3 algorithm, both trained only with the navigation reward $R^{nav}_i$ (no explicit formation reward $r_9$).
Qualitative Path Analysis
Visual inspection of the planned paths reveals distinct behaviors. MADDPG often results in meandering, uncoordinated paths. MATD3 produces more direct, efficient paths for individual UAVs but shows no collective formation behavior; each UAV solves its path planning problem largely in isolation. In stark contrast, MATD3-IDFRF orchestrates a cohesive drone formation strategy: UAVs quickly converge into the prescribed wedge shape, traverse the environment as a single entity with minor formation adjustments to avoid obstacles, and finally break formation to approach their individual goals. This demonstrates the successful internalization of the dynamic formation objective.
Quantitative Performance Metrics
To quantify formation keeping, we define a Formation Deformation Rate at time $t$:
$$
\text{DeformationRate}(t) = \sum_{i=1}^{N} \sum_{j\neq i}^{N} \frac{|d_{ij}(t) – d_{opt, ij}|}{d_{opt, ij}}
$$
A lower rate indicates better adherence to the desired formation geometry. The following table summarizes key comparative results, and the figure below shows the deformation rate over a typical episode.
| Metric | MADDPG | MATD3 | MATD3-IDFRF (Ours) |
|---|---|---|---|
| Average Path Length | 10,763 m | 10,438 m | 10,536 m |
| Average Formation Deformation Rate | 55.50% | 22.60% | 0.68% |
| Success Rate (Final) | ~88% | ~92% | ~98.8% |
| Converged Reward (Normalized Avg.) | ~211 | ~220 | ~575* |
*Note: The absolute reward value for MATD3-IDFRF is higher due to the added $r_9$ term. The key observation is its 2.3% improvement over MATD3 when compensating for this baseline shift, alongside significantly lower variance, indicating more stable policy performance.
The results are conclusive: MATD3-IDFRF reduces the formation deformation rate by over 97% compared to MATD3, while simultaneously improving the task success rate by 6.8% and achieving a higher, more stable converged reward. The learning curve of MATD3-IDFRF also shows faster initial convergence and greater stability post-convergence compared to the baselines, indicating that the dynamic formation reward provides a clear and effective learning gradient.
Generalization Test
To evaluate robustness and generalization, we tested the trained policies in a modified environment with different dynamic obstacle trajectories and speeds. All algorithms successfully completed the mission. However, only MATD3-IDFRF consistently maintained a tight, well-formed drone formation throughout the journey, dynamically adjusting its shape when confronted with new obstacle configurations before recovering its structure. This underscores the generalizability of the learned formation-keeping and deformation policies beyond the specific training scenarios.
Conclusion
This work presents a novel and effective solution for intelligent drone formation path planning in unknown, dynamic environments. By formulating the core challenge of dynamic formation—stable yet elastic spatial coordination—as a reward shaping problem within the RL framework, we develop a dynamic formation reward function. Integrating this function with the robust MATD3 algorithm yields the MATD3-IDFRF method. The proposed reward function $r_9$ successfully encodes the high-level objective: it maximizes when the formation is geometrically perfect and provides a smooth gradient for controlled deformation. Extensive simulations demonstrate that MATD3-IDFRF significantly outperforms state-of-the-art multi-agent RL baselines in formation keeping (97% lower deformation) and overall mission performance (higher success rate and stability). The algorithm enables a drone formation to autonomously navigate complex terrains as a cohesive, adaptable unit, balancing the demands of collective motion with the necessities of individual obstacle avoidance. Future work will explore extending this framework to 3D environments, integrating communication constraints, and investigating hierarchical reward structures for more complex mission profiles.