In recent years, drone technology has advanced rapidly, making multi-UAV cooperative formation flight a key enabler for applications such as reconnaissance, surveillance, communication relay, and emergency rescue. However, trajectory planning for UAV formations in complex obstacle environments poses significant challenges, particularly in balancing formation maintenance and safe obstacle avoidance. In this work, we propose a cooperative trajectory planning method based on an improved Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm. Our approach addresses the sparse reward problem and overestimation issues inherent in traditional MADDPG by incorporating twin critic networks, curriculum learning, and a dynamic reward weighting strategy. Through extensive simulations in complex static obstacle scenarios, we demonstrate that our method achieves high task success rates, effective formation reconfiguration, and stable formation maintenance. The integration of curriculum learning significantly enhances training stability and convergence, proving essential for mastering the complex task.
Introduction
Drone technology has evolved from single-agent operations to multi-agent cooperative systems, enabling a wide range of complex missions. Formation flight allows multiple UAVs to accomplish tasks with improved coverage, robustness, and efficiency. However, in cluttered environments with obstacles or no-fly zones, maintaining a predefined formation while avoiding collisions becomes a critical issue. Traditional approaches such as leader-follower, artificial potential fields, or optimization-based methods often struggle to simultaneously satisfy formation constraints and obstacle avoidance requirements. Recent advances in reinforcement learning, particularly the multi-agent deep deterministic policy gradient (MADDPG) algorithm, offer a promising framework for learning cooperative policies under centralized training and decentralized execution (CTDE). Nevertheless, MADDPG suffers from Q-value overestimation, especially in high-dimensional continuous action spaces, leading to policy oscillation and convergence difficulties. Moreover, the multi-objective nature of formation-avoidance tasks leads to sparse reward signals, hindering effective learning. To overcome these challenges, we introduce an improved MADDPG algorithm that integrates twin critic networks with Clipped Double Q-learning, curriculum learning, and a dynamic reward weighting mechanism. This work contributes a comprehensive training environment for UAV formation trajectory planning, a curriculum strategy that progresses from simple navigation to full formation-avoidance tasks, and a three-phase dynamic weighting scheme that adapts the reward function to the current mission stage.
Problem Statement
Task Scenario
We consider a two-dimensional mission area containing static circular obstacles or no-fly zones. A diamond formation consisting of one leader and three followers is required to start from a designated area, safely navigate through the obstacle region, reconfigure to the predefined formation, and finally reach a target region while maintaining the formation. The task is divided into three phases: Phase I (obstacle traversal), where the formation is allowed to deform temporarily to avoid obstacles; Phase II (formation reconfiguration), where after passing the obstacles, the formation must converge to the desired shape within a limited number of steps; Phase III (formation maintenance), where the formation is maintained stably until the leader reaches the target. The mission is considered successful if the leader enters the target region without any collisions, boundary violations, or excessive flight time.
Failure Conditions
A trial terminates with failure under any of the following conditions: (1) any UAV collides with an obstacle or violates the mission boundary; (2) any two UAVs come within a minimum safe distance (collision); (3) the leader exceeds the maximum number of steps.
System Modeling
UAV Dynamics
Each UAV is modeled as a point mass with discrete dynamics:
$$
\begin{aligned}
\mathbf{p}_i(t+1) &= \mathbf{p}_i(t) + v_i(t) \Delta t [\cos\alpha_i(t), \sin\alpha_i(t)] ,\\
v_i(t+1) &= v_i(t) + a_i(t) \Delta t ,\\
\alpha_i(t+1) &= \alpha_i(t) + \omega_i(t) \Delta t ,
\end{aligned}
$$
where \(\mathbf{p}_i=(x_i,y_i)\) is the position, \(v_i\) is the speed, \(\alpha_i\) is the heading angle, and \(a_i, \omega_i\) are the linear acceleration and angular velocity, respectively. All variables are subject to physical constraints: speed bounded by \([0, v_{\max}]\), acceleration by \([-a_{\max}, a_{\max}]\), and angular velocity by \([-\omega_{\max}, \omega_{\max}]\). The safe inter-UAV distance is \(d_{\text{safe}}\), and the obstacle radius is \(R_o\).
Formation Design
The desired diamond formation is defined relative to the leader’s local coordinate frame. The leader’s heading direction defines the local x-axis, and the leftward perpendicular defines the local y-axis. The three follower positions relative to the leader are:
$$
\begin{aligned}
\Delta\mathbf{p}_1 &= [-d\cos\phi,\ d\sin\phi] ,\\
\Delta\mathbf{p}_2 &= [-d\cos\phi,\ -d\sin\phi] ,\\
\Delta\mathbf{p}_3 &= [-2d\cos\phi,\ 0] ,
\end{aligned}
$$
where \(d=10\) m is the formation layer spacing, and \(\phi=30^\circ\) is the half-angle. In the global frame, the desired position of follower \(i\) is \(\mathbf{p}_i^*(t)=\mathbf{p}_1(t) + \mathbf{R}(\alpha_1(t)) \Delta\mathbf{p}_i\), with \(\mathbf{R}(\alpha)\) the rotation matrix.
POMDP Formulation
We model the cooperative trajectory planning problem as a partially observable Markov decision process (POMDP) defined by the tuple \(\langle \mathcal{S},\mathcal{A},\mathcal{Z},\mathcal{R},\mathcal{P},\mathcal{O},\gamma \rangle\).
- State space \(\mathcal{S}\): includes the UAV states \(\mathbf{s}_i=[x_i,y_i,v_i,\alpha_i]\) and environment information (obstacle positions and target region).
- Action space \(\mathcal{A}\): each UAV’s action is \(\mathbf{a}_i=[a_i,\omega_i]\).
- Observation space \(\mathcal{Z}\): The leader observes its own state and the follower states? Actually, we define leader observation as \(\mathbf{z}_1 = [x_1,y_1,v_1,\alpha_1,a_1,\omega_1]\), and follower observations include their own state, the leader’s state, obstacle information, and the nearest neighbor follower’s relative position.
- Reward function \(\mathcal{R}\): decomposed into multiple sub-rewards with weights that vary by curriculum stage and mission phase.
- Transition probability \(\mathcal{P}\): deterministic from the dynamics, with absorbing states for success/failure.
- Observation function \(\mathcal{O}\): deterministic mapping from state to observation (no noise).
- Discount factor \(\gamma=0.95\).
Reward Components
We define several sub-rewards for each UAV:
- Goal progress: \(r_{\text{goal},i}(t) = d_i(t-1) – d_i(t)\), where \(d_i\) is distance to target. For the leader, an extra term encourages proximity.
- Time penalty: \(r_{\text{time},i}(t) = -1\) per step.
- Obstacle avoidance: piecewise penalty based on distance to obstacle boundary, with a buffer zone of width \(b=5\) m.
- Formation error: based on Euclidean distance to desired position, with a piecewise reward.
- Inter-UAV separation: penalizes distances below \(d_{\text{safe}}\).
- Control smoothness: penalizes large action changes.
- Terminal reward: \(+100\) for success, \(-500\) for failure (collision, boundary, timeout).
- Additional shaping for followers: includes distance to leader, moving consistency, and heading alignment.
The total reward for the leader and followers is a weighted sum of these components. We use a dynamic weighting strategy that changes according to the curriculum stage and the mission phase (Phase I/II/III). The tables below summarize the weights for different stages. (Tables omitted for brevity, but we present key tables.)
Curriculum Learning Stages
To mitigate the cold-start problem and sparse rewards, we design a four-stage curriculum:
| Stage | Environment | Training Objective |
|---|---|---|
| C1 | No obstacles, no formation constraint | Basic navigation to target |
| C2 | Static obstacles added | Obstacle avoidance |
| C3 | Formation required (collision not terminal but penalized) | Learn formation keeping while avoiding |
| C4 | Full task with terminal collision and desired formation | Achieve high success rate with low formation error |
Each stage is trained for a fixed number of episodes, and we only proceed to the next stage when the success rate stabilizes above predefined thresholds (95%, 90%, 85% for C1, C2, C3 respectively).
Improved MADDPG Algorithm
Centralized Training with Decentralized Execution (CTDE)
We adopt the CTDE framework where each UAV (agent) has an actor network \(\mu_i(\mathbf{o}_i;\theta_i)\) that outputs actions in the range \([-1,1]\), later scaled to actual control limits. During training, a centralized critic network (or twin critics) for each agent takes the joint observation \(\mathbf{s}_t\) (concatenation of all observations) and joint action \(\mathbf{a}_t\) as input to estimate Q-values.
Twin Critic Networks and Clipped Double Q
To reduce overestimation, we employ two critic networks per agent: \(Q_{\phi_{i,1}}(\mathbf{s},\mathbf{a})\) and \(Q_{\phi_{i,2}}(\mathbf{s},\mathbf{a})\), with corresponding target networks \(Q_{\phi’_{i,1}}, Q_{\phi’_{i,2}}\). For a sampled transition \((\mathbf{s}_t,\mathbf{a}_t,r_t,\mathbf{s}_{t+1},d_t)\) from the replay buffer, the target Q-value is:
$$
y_i = r_t + \gamma (1-d_t) \min_{k=1,2} Q_{\phi’_{i,k}}(\mathbf{s}’,\mathbf{a}’) ,
$$
where \(\mathbf{a}’\) is formed by the target actor outputs. The critics are updated by minimizing the mean squared error:
$$
\mathcal{L}_{Q_i} = \mathbb{E}\left[ \sum_{k=1}^2 \left( Q_{\phi_{i,k}}(\mathbf{s},\mathbf{a}) – y_i \right)^2 \right] .
$$
Actor Update
The actor is updated by maximizing the first critic’s output with respect to the action, while keeping other agents’ actions fixed from the sampled batch:
$$
\nabla_{\theta_i} J_i \approx \mathbb{E}\left[ \nabla_{\theta_i} \mu_i(\mathbf{o}_i) \nabla_{\mathbf{a}_i} Q_{\phi_{i,1}}(\mathbf{s},\mathbf{a}) \right] .
$$
Exploration and Prioritized Experience Replay
We add Ornstein-Uhlenbeck noise to the actions for exploration, with a noise schedule that resets to a high value at the beginning of each curriculum stage and decays linearly to a low value. We also implement prioritized experience replay to focus on transitions with high TD error, which is particularly helpful in sparse reward scenarios. Importance sampling weights are used to correct bias.
Three-Phase Dynamic Weighting in Full Task
During the full task (C4), we use a three-phase weight scheme that automatically transitions based on the leader’s position and formation error:
- Phase I (obstacle traversal): High weight on obstacle avoidance and separation; low weight on formation error.
- Phase II (reconfiguration): High weight on formation error; moderate obstacle avoidance.
- Phase III (maintenance): Balanced weights with emphasis on formation and goal progress.
The exact weight values are tuned heuristically and provided in the simulation setup.
Simulation Results and Analysis
Experiment Setup
We set up a square mission area of \(200 \times 200\) m. The start region is \([0,20]\times[0,20]\) m. The target is a circle of radius 15 m centered at (180,180). Obstacles are generated near the reconfiguration trigger zone (x between 75 and 90). We use four UAVs (one leader, three followers). The time step is 0.1 s, and the maximum steps per episode is 1000. Hyperparameters are summarized in the following table.
| Parameter | Value |
|---|---|
| Discount factor \(\gamma\) | 0.95 |
| Soft update coefficient \(\tau\) | 0.01 |
| Actor learning rate | 0.001 (halved in C4) |
| Critic learning rate | 0.003 |
| Replay buffer size | 50000 |
| Batch size | 512 |
| Noise schedule | Start: 0.20→0.02 (C1-2), 0.10→0.02 (C3), 0.05→0.01 (C4) |
| Curriculum episodes | C1:1200, C2:800, C3:800, C4:1600 |
Curriculum Learning Results
The training curves show that without curriculum learning, the model fails to converge (success rate zero). With the proposed curriculum, the average reward increases steadily across stages, and the final stage achieves high success rates. The twin critic structure combined with curriculum learning provides a stable value baseline, preventing overestimation errors from derailing the training.
Trajectory Analysis
We evaluate the trained policy in two-obstacle, three-obstacle, and four-obstacle scenarios (with random obstacle placements in the last). The formation successfully navigates through obstacles, reconfigures after passing, and maintains the diamond formation to the target. The overall success rate for the two-obstacle case exceeds 97%. For four obstacles trained with random placement, the success rate is 83%, indicating good generalization.
Comparative Analysis
We compare our full method (with curriculum, twin critics, PER, dynamic weighting) against three ablation variants: (1) without prioritized experience replay (PER), (2) with fixed reward weights instead of dynamic weighting, and (3) without curriculum learning (direct C4 training). The results are shown in the table below.
| Method | Success Rate | Collision Rate | Out-of-Bounds Rate | Avg Path Length (m) | Avg Steps | Avg Formation Error (m) |
|---|---|---|---|---|---|---|
| Full method | 1.00 | 0.00 | 0.00 | 225.84 | 256.6 | 11.02 |
| Without PER | 0.97 | 0.00 | 0.03 | 229.10 | 259.0 | 10.70 |
| Fixed weights | 0.99 | 0.01 | 0.00 | 226.75 | 253.1 | 9.02 |
| Without curriculum | 0.00 | 0.07 | 0.93 | 56.61 | 136.4 | 74.37 |
These results demonstrate that curriculum learning is essential for successful training. The other variants achieve similar performance, though PER shows a slight degradation in success rate, possibly due to unbalanced sampling. The fixed-weight variant performs comparably, indicating some robustness to weight tuning, but the dynamic weighting provides better adaptability during the mission.
Conclusion
In this work, we have presented an improved MADDPG algorithm for cooperative trajectory planning of UAV formations in complex obstacle environments. By incorporating twin critic networks, curriculum learning, and a dynamic reward weighting strategy, our method effectively addresses the challenges of sparse rewards and value overestimation. Simulation results confirm that the proposed approach achieves high success rates, efficient obstacle traversal, and stable formation reconfiguration. Future work will extend the method to three-dimensional dynamic environments, larger heterogeneous swarms, and scenarios with communication constraints and sensor noise, further advancing the capabilities of drone technology in autonomous operations.
“`