In modern autonomous systems, the deployment of drone formations to collaboratively execute complex tasks in large-scale, dynamic, and robust environments has emerged as a new paradigm. For a drone formation to accomplish given missions, it often needs to navigate between multiple designated points within obstacle-rich regions. Therefore, planning flight trajectories that minimize collision risks, enhance training stability, and ensure smooth paths for drone formations is a critical focus in the field. In leader-follower based drone formations, the leader is responsible for global path planning, while followers track the leader’s path and actions while maintaining relative distances, significantly reducing the complexity of formation management. In this context, various trajectory planning methods based on artificial potential fields, swarm intelligence, and deep reinforcement learning have been developed. However, existing deep reinforcement learning approaches for drone formation trajectory planning often suffer from issues such as prolonged training times, unstable training processes, and even non-convergence. To address these challenges, we propose a real-time 3D trajectory planning method for drone formations based on the SPER-TD3 algorithm. This method integrates a prioritized experience replay mechanism using SumTree into the TD3 algorithm, designs a network model combining LSTM, self-attention mechanisms, and multi-layer perceptrons, and employs an angle-based formation control method along with a dynamic trajectory smoothing algorithm to optimize follower paths. Our contributions include enhancing training stability and convergence speed, improving formation robustness and trajectory smoothness, and demonstrating superior performance through comprehensive simulations.
The trajectory planning problem for drone formations is modeled in a 3D task area with unknown terrain, where the goal is to find the shortest path from a start point to a target point while maintaining formation and avoiding collisions. The environment includes various obstacles simplified into geometric shapes like spheres, cylinders, and cones. The collision detection for a drone at position $(x, y, z)$ with obstacles is defined as follows. For a spherical obstacle with center $(x_{sp}, y_{sp}, z_{sp})$ and radius $r$, the condition is:
$$
\Phi(Q) = \left( \frac{x – x_{sp}}{r} \right)^2 + \left( \frac{y – y_{sp}}{r} \right)^2 + \left( \frac{z – z_{sp}}{r} \right)^2
$$
For a cylindrical obstacle with base center $(x_{cy}, y_{cy}, z_{cy})$, radius $r$, and height $H$ (where $0 \leq z \leq H$), the condition is:
$$
\Phi(Q) = \left( \frac{x – x_{cy}}{r} \right)^2 + \left( \frac{y – y_{cy}}{r} \right)^2
$$
For a conical obstacle with base center $(x_{co}, y_{co}, z_{co})$, radius $r$, and height $H$, the condition is:
$$
\Phi(Q) = \frac{1}{1 – \frac{z}{H}} \left( \left( \frac{x – x_{co}}{r} \right)^2 + \left( \frac{y – y_{co}}{r} \right)^2 \right)
$$
Here, $\Phi(Q) < 1$ indicates the drone is inside the obstacle, $\Phi(Q) = 1$ on the surface, and $\Phi(Q) > 1$ outside. The motion model of a drone is described by its 3D coordinates, heading angle $\psi$, and climb angle $\gamma$. At time $t$, the position $q_t = (x_t, y_t, z_t)$ is updated from $q_{t-1}$ with changes $(\Delta x, \Delta y, \Delta z)$:
$$
q_t = q_{t-1} + \begin{bmatrix} \Delta x \\ \Delta y \\ \Delta z \end{bmatrix}, \quad \psi = \arcsin\left( \frac{|\Delta y|}{\sqrt{\Delta x^2 + \Delta y^2}} \right), \quad \gamma = \arcsin\left( \frac{\Delta z}{\sqrt{\Delta x^2 + \Delta y^2 + \Delta z^2}} \right)
$$
Under speed and angle constraints, the constrained changes are computed as:
$$
\Delta x_{res} = R \cos(\psi_{res}) \cos(\gamma_{res}), \quad \Delta y_{res} = R \cos(\psi_{res}) \sin(\gamma_{res}), \quad \Delta z_{res} = R \sin(\gamma_{res})
$$
where $R = \sqrt{\Delta x^2 + \Delta y^2 + \Delta z^2}$ is the Euclidean distance, and $\psi_{res}$, $\gamma_{res}$ are constrained angles. The updated position is $q_{t+1} = q_t + [\Delta x_{res}, \Delta y_{res}, \Delta z_{res}]^T$.
For formation control, we design an angle-based virtual leader method. The leader’s direction vector $F$ is computed from its movement, and a rotation axis $V$ is determined perpendicular to $F$. Using Rodrigues’ rotation formula, a rotated vector $V_{rot}$ is obtained for a specified angle $\theta$ and distance $D$ to position the virtual leader $Q$ relative to the leader’s position $q_t$:
$$
F = \frac{q_{t+1} – q_t}{\|q_{t+1} – q_t\|}, \quad V = F \times K, \quad V’ = \frac{V}{\|V\|}
$$
$$
V_{rot} = V \cos \theta + (V’ \times V) \sin \theta + V’ (V’ \cdot V)(1 – \cos \theta), \quad Q = q_t + D \cdot V_{rot}
$$
where $K$ is a unit vector not parallel to $F$. Followers track their virtual leaders, but to smooth trajectories and reduce sharp turns, a dynamic trajectory smoothing algorithm based on curvature is applied. The curvature $\theta_{curvature}$ is computed from the angle between the average past direction $F_{ave}$ and the current resultant force $F’$:
$$
\theta_{curvature} = \arccos\left( \frac{F_{ave} \cdot F’}{\|F_{ave}\| \|F’\|} \right)
$$
A smoothing factor $\alpha$ is dynamically adjusted: $\alpha = 0.5$ if $\theta > \pi/4$, else $\alpha = 1$. The smoothed direction $F_{smooth}$ and updated follower position $q_{t+1}$ are:
$$
F_{smooth} = \alpha F’ + (1 – \alpha) F_{ave}, \quad q_{t+1} = q_t + v \cdot F_{smooth}
$$
where $v$ is the step size. The overall target position for a follower is $q_{i,follower} = q_{leader} + d_i + a_i$, with $d_i$ from virtual leader offset and $a_i$ from obstacle avoidance.
The drone formation trajectory planning problem is formulated as a weighted optimization to minimize path length, collision probability, and formation maintenance error. Let $L$ be the total path length for $N$ points, $P_{collision}(q)$ the collision probability at position $q$, and $\|q_{i,follower} – q_i\|$ the formation error. The optimization is:
$$
\min_{q_1, q_2, \dots, q_N} \left( \omega_1 L + \omega_2 \sum_{i=1}^N P_{collision}(q_i) + \omega_3 \sum_{i=1}^N \|q_{i,follower} – q_i\|^2 \right)
$$
subject to constraints on heading angle $\psi$ and climb angle $\gamma$: $-\psi_{max} \leq \psi \leq \psi_{max}$, $-\gamma_{max} \leq \gamma \leq \gamma_{max}$, $q_i \in \mathcal{A}$ (feasible area), and $\|q_i – o\| \geq d_{min}$ for all obstacles $o$. The path length is $L = \sum_{i=1}^{N-1} \|q_{i+1} – q_i\|$, and collision probability is estimated as $P_{collision}(q) = \exp(-\lambda \min_{o \in \text{obstacles}} \|q – o\|)$ with decay factor $\lambda$.
To solve this, we model the problem as a Markov Decision Process (MDP) $M = (\mathcal{S}, \mathcal{A}, P, R)$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ the action space, $P$ transition probabilities, and $R$ the reward function. The state for each obstacle $i$ includes relative positions to the obstacle and target: $s_i = [\Delta x_i, \Delta y_i, \Delta z_i, \Delta x_{goal}, \Delta y_{goal}, \Delta z_{goal}]$. The action $a_i$ is a steering angle generated by a TD3 network for each obstacle type. The reward function combines collision avoidance $R_{col}$, distance to target $R_{len}$, and angle smoothness $R_{ang}$:
$$
R_{col} =
\begin{cases}
\frac{d(q_t, obs) – R_{obs}}{R_{obs}}, & \text{if } d(q_t, obs) \leq R_{obs} \\
k \cdot [d(q_t, obs) – d_{safe}], & \text{if } d(q_t, obs) > d_{safe} \\
0, & \text{otherwise}
\end{cases}
$$
$$
R_{len} =
\begin{cases}
-\frac{d(q_t, q_{goal})}{d(q_{start}, q_{goal})} + 3, & \text{if } d(q_t, q_{goal}) < \text{threshold} \\
-\frac{d(q_t, q_{goal})}{d(q_{start}, q_{goal})}, & \text{otherwise}
\end{cases}
$$
$$
R_{ang} = -\left( \frac{|\Delta \psi|}{\psi_{max}} + \frac{|\Delta \gamma|}{\gamma_{max}} \right) \cdot 0.1
$$
The total reward is $R_{total} = R_{col} + R_{len} + R_{ang}$. We propose the SPER-TD3 algorithm, which integrates a prioritized experience replay mechanism based on SumTree into the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm. This prioritizes experiences with high Temporal Difference (TD) error to accelerate learning. The algorithm uses a fully decentralized framework where each drone has its own decision model. The network architecture combines LSTM layers to handle temporal dependencies, self-attention mechanisms to focus on important information, and multi-layer perceptrons (MLPs) for feature mapping. The Actor network takes state $s$ as input, passes through an LSTM layer (128 neurons), then MLP layers (128, 256, 256 neurons with ReLU), and outputs an action $a$ via a tanh output layer. The Critic network concatenates state $s$ and action $a$, processes through a self-attention layer (6 neurons), LSTM layer (128 neurons), MLP layers (similar to Actor), and outputs a Q-value.
The SPER-TD3 algorithm workflow is as follows. Initialize Actor and Critic networks with parameters $\theta$ and $\phi$, and target networks with $\theta’$, $\phi’$. Set prioritized replay buffer with priority exponent $\alpha$ and importance sampling exponent $\beta$. For each training step, observe state $s$, select action $a = \pi(\theta, s) + \epsilon$ with noise $\epsilon \sim \mathcal{N}(0, \sigma^2)$, execute $a$ to get reward $r$ and next state $s’$. Compute TD error $\delta = |r + \gamma \min(Q_1(\phi’, s’, \pi(\theta’, s’)), Q_2(\phi’, s’, \pi(\theta’, s’))) – \min(Q_1(\phi, s, a), Q_2(\phi, s, a))|$, and store transition $(s, a, r, s’, p)$ with priority $p = (|\delta| + \varepsilon)^\alpha$ in SumTree. When sampling, compute sample weight $\omega = (1/N \cdot 1/p)^\beta$. Update Critic networks by minimizing loss $Loss_Q = \omega \cdot (\text{MSE}(Q_1(\phi, s, a) – y) + \text{MSE}(Q_2(\phi, s, a) – y))$ where $y = r + \gamma \min(Q_1(\phi’, s’, a’), Q_2(\phi’, s’, a’))$ with target action $a’ = \pi'(s’) + \epsilon’$, $\epsilon’ \sim \text{clip}(\mathcal{N}(0, \sigma’^2), -\varepsilon, \varepsilon)$. Update Actor network every policy_delay steps to maximize $Q_1(\phi, s, a(\theta, s))$. Soft-update target networks: $\theta’ \leftarrow \tau \theta + (1-\tau) \theta’$, $\phi’ \leftarrow \tau \phi + (1-\tau) \phi’$. Repeat, updating priorities in SumTree based on new TD errors.
We conduct simulations in two 3D obstacle environments: a large-obstacle environment (10×10×6 km³) with spheres (radius 2 km), cones (base radius 2 km, height 2 km), and cylinders (base radius 1 km, height 6 km); and a small-obstacle environment (15×15×3 km³) with smaller versions. Parameters are set as in Table 1.
| Parameter | Meaning | Value |
|---|---|---|
| $\gamma$ | Discount factor | 0.99 |
| $I_a$ | Actor learning rate | 0.001 |
| $I_c$ | Critic learning rate | 0.001 |
| $\tau$ | Soft update coefficient | 0.005 |
| $\alpha$ | Priority exponent | 0.6 |
| $\beta$ | Importance sampling exponent | 0.4 |
| Max capacity | Replay buffer capacity | 10^6 |
| Batch size | Sampling size | 512 |
| Optimizer | Optimizer type | Adam |
| Max step | Maximum steps per episode | 500 |
| Max Episode | Maximum episodes | 500 |
We compare SPER-TD3 against mainstream deep reinforcement learning algorithms in both fully centralized and fully decentralized frameworks, including TD3, DDPG, SAC, PPO, and MADDPG. Evaluation metrics are path length (km), reward magnitude (absolute value), success rate (%), trajectory safety coverage rate (%), and an importance comprehensive evaluation value $I$. The safety coverage rate is the percentage of path within a safe distance from obstacles. The comprehensive evaluation $I$ normalizes each metric to [0,1] and computes a weighted sum:
$$
L’ = 1 – \frac{|L – L_{safe}|}{L_{max} – L_{min}}, \quad R’ = \frac{R – R_{min}}{R_{max} – R_{min}}, \quad S’ = S, \quad C’ = C
$$
$$
I = \omega_L \cdot L’ + \omega_R \cdot R’ + \omega_S \cdot S’ + \omega_C \cdot C’
$$
where $\omega_L = \omega_R = \omega_S = \omega_C = 0.25$ for balanced weighting. Results are summarized in Table 2.
| Algorithm | Path Length (km) | Reward Magnitude | Success Rate (%) | Trajectory Safety Coverage (%) | Importance Comprehensive Evaluation Value |
|---|---|---|---|---|---|
| SPER-TD3 | 13.26 | 29.96 | 98.80 | 33.33 | 0.650 |
| MADDPG | 13.17 | 29.92 | 99.00 | 19.40 | 0.596 |
| Fully Decentralized TD3 | 13.33 | 30.92 | 98.40 | 24.84 | 0.585 |
| Fully Centralized TD3 | 13.60 | 30.70 | 98.00 | 31.43 | 0.642 |
| Fully Decentralized DDPG | 12.84 | 28.54 | 97.40 | 22.72 | 0.599 |
| Fully Centralized DDPG | 13.78 | 32.01 | 94.00 | 31.22 | 0.477 |
| Fully Decentralized SAC | 12.74 | 29.86 | 92.20 | 30.43 | 0.478 |
| Fully Centralized SAC | 13.77 | 31.47 | 90.60 | 26.57 | 0.498 |
| Fully Centralized PPO | 13.92 | 31.91 | 72.20 | 26.57 | 0.376 |
The results show that SPER-TD3 achieves competitive path length, high reward magnitude, excellent success rate, and good trajectory safety coverage, leading to the highest importance comprehensive evaluation value among all algorithms. For formation trajectory smoothness, the dynamic trajectory smoothing algorithm significantly reduces sharp turning angles for followers. In a test with a leader and two followers, the average turning angles for followers decreased from 89.61° and 114.03° before smoothing to 51.67° and 54.68° after smoothing, enhancing safety and energy efficiency. The average episode reward curves in both large and small obstacle environments demonstrate that SPER-TD3 converges fastest and maintains the highest stability compared to other algorithms, with rewards approaching optimal values steadily.
In conclusion, we propose a novel real-time 3D trajectory planning method for drone formations based on the SPER-TD3 algorithm. By integrating prioritized experience replay with SumTree, designing an advanced network model with LSTM and self-attention, and employing angle-based formation control with dynamic trajectory smoothing, our approach addresses key challenges in training stability, convergence speed, and path smoothness for drone formations. Simulations in complex obstacle environments validate that SPER-TD3 outperforms existing deep reinforcement learning algorithms in terms of comprehensive performance metrics, with improvements in importance evaluation values ranging from 8.5% to 72.9%. Future work may focus on adapting the algorithm for larger-scale obstacle environments and further optimizing computational efficiency for real-time applications in dynamic drone formation scenarios. The ability to plan safe, smooth, and efficient trajectories is crucial for advancing autonomous drone formation systems in various fields such as surveillance, delivery, and disaster response.
The drone formation paradigm leverages multiple drones to achieve redundancy, robustness, and enhanced coverage, making it ideal for complex missions. Our method contributes to this by ensuring that each drone in the formation can navigate collaboratively while maintaining formation integrity and avoiding obstacles. The use of deep reinforcement learning allows the drone formation to adapt to unknown environments, and the prioritization mechanism accelerates learning from critical experiences. The trajectory smoothing algorithm specifically addresses the issue of sharp turns that can lead to collisions or excessive energy consumption in drone formations. Overall, the SPER-TD3 framework provides a scalable and effective solution for 3D trajectory planning in drone formations, paving the way for more advanced multi-drone applications.