In recent years, unmanned aerial vehicles (UAVs), particularly multi-rotor drones, have seen significant advancements due to their low cost, high responsiveness, and agility. These platforms are increasingly used in applications such as fault detection, disaster rescue, and logistics, often replacing traditional manned aircraft. Compared to single UAV systems, multi-UAV systems offer enhanced robustness, adaptability, and resilience, making them crucial for tasks like target search and surveillance. Among these, drone formation control is a key research area in UAV swarm technology, focusing on coordinating multiple drones to maintain desired geometric configurations while performing complex missions in dynamic environments. The core challenge lies in ensuring that all drones maintain relative position and velocity consistency, avoid obstacles, and robustly sustain the formation structure. Therefore, multi-drone formation navigation in complex environments holds great significance, and this work addresses it through a novel reinforcement learning approach.
The control of drone formation involves various methods, such as leader-follower approaches, virtual center methods, and artificial potential field techniques. However, traditional methods like artificial potential fields may suffer from local optima, failing to guarantee global path solutions. Model-based approaches, such as consensus theory, often require precise system modeling, which is difficult in dynamic settings. Reinforcement learning (RL) has emerged as a powerful tool for solving optimal control problems in complex systems by learning from trajectory data. Specifically, deep deterministic policy gradient (DDPG) and its extension, twin delayed deep deterministic policy gradient (TD3), have shown promise in continuous control tasks. For multi-agent systems, multi-agent reinforcement learning (MARL) algorithms, like multi-agent TD3 (MATD3), enable collaborative decision-making through centralized training and decentralized execution. Yet, many existing studies neglect the multi-objective nature of drone formation control, where goals such as formation maintenance and obstacle avoidance often conflict. To tackle this, we propose a multi-objective multi-agent twin delayed deep deterministic policy gradient (MO-MATD3) algorithm, incorporating multi-objective planning for dynamic strategy switching. This work designs dense reward functions based on artificial potential field theory and introduces a mode-switching mechanism, validated through comparative, generalization, and robustness experiments.
We first describe the problem of multi-drone formation control in a dynamic obstacle environment. The goal is to coordinate a group of drones to navigate toward a target while preserving a predefined formation and avoiding collisions. We adopt a virtual center control architecture, where the formation’s geometric center serves as a dynamic reference point. For a system with N drones, the virtual center position \( p_c(x_c, y_c) \) is computed as:
$$ x_c = \frac{1}{N} \sum_{i=1}^{N} x_i, \quad y_c = \frac{1}{N} \sum_{i=1}^{N} y_i $$
Each drone \( i \) aims to maintain its position relative to the virtual center with a predefined offset \( \Delta_i \), such that:
$$ \| p_i – (p_c + \Delta_i) \| = 0, \quad i = 1, 2, \dots, N $$
where \( p_i(x_i, y_i) \) is the position of drone \( i \). The overall path planning problem can be formulated as minimizing the total Euclidean distance to the target while satisfying safety constraints:
$$ \min \sum_{i=1}^{N} D_i \quad \text{s.t.} \quad \begin{cases} \| p_i – p_j \| > d_{a_s}, & \forall i,j \\ \| p_i – p_o \| > d_{ob_s}, & \forall i, o \\ \| p_i – (p_c + \Delta_i) \| = 0, & \forall i \end{cases} $$
Here, \( D_i \) is the distance from drone \( i \) to the target, \( d_{a_s} \) is the safe distance between drones, \( p_o \) is obstacle position, and \( d_{ob_s} \) is the safe distance between drones and obstacles. This formulation captures the multi-objective nature of drone formation control, balancing navigation, formation keeping, and obstacle avoidance.
For motion modeling, we consider a 2D planar environment at constant altitude. The kinematics of drone \( i \) are described by:
$$ v_{i,x}^{t+1} = v_{i,x}^t + a_{i,x} \cdot t_{\text{step}}, \quad v_{i,y}^{t+1} = v_{i,y}^t + a_{i,y} \cdot t_{\text{step}} $$
$$ x_i^{t+1} = x_i^t + v_{i,x}^{t+1} \cdot t_{\text{step}}, \quad y_i^{t+1} = y_i^t + v_{i,y}^{t+1} \cdot t_{\text{step}} $$
where \( (v_{i,x}, v_{i,y}) \) is velocity, \( (x_i, y_i) \) is position, \( (a_{i,x}, a_{i,y}) \) is acceleration control input, and \( t_{\text{step}} \) is time step. The control focuses on the outer loop for position and velocity adjustment, assuming an inner-loop PD controller for attitude. We assume full connectivity among drones for information sharing, enabling distributed execution based on local observations.
The foundation of our approach is the MATD3 algorithm, which extends TD3 to multi-agent settings. MATD3 uses a centralized training and decentralized execution framework, where each agent has independent Actor and Critic networks. During training, Critic networks access global state and action information to mitigate non-stationarity, while Actors rely on local observations. For agent \( i \), the action is \( a_i = \mu_i(o_i | \theta^\mu_i) \), where \( \theta^\mu_i \) is Actor parameters. The two Critic networks compute action-value functions \( Q^j_i(o, a_1, \dots, a_N; \theta^j_i) \) for \( j=1,2 \). Updates involve target policy smoothing with clipped noise \( \epsilon \sim \text{clip}(\mathcal{N}(0,1), -c, c) \), TD target calculation, and loss minimization. The target networks are softly updated with rate \( \tau \). This structure enhances stability and performance in continuous control tasks for drone formation scenarios.
Our proposed MO-MATD3 algorithm integrates multi-objective planning to handle conflicting goals in drone formation control. We design a comprehensive observation space for each drone \( i \), including: relative positions to target, neighboring drones, formation center, and obstacles; its own position and velocity; and the angle \( \theta \) between its velocity vector and the formation navigation direction. The action space is continuous 2D acceleration \( (a_x, a_y) \). The reward function is dense and shaped using artificial potential field principles to encourage efficient learning. Key components include:
- Collective target reward: \( r_{\text{target}} = -\alpha_{\text{target}} \cdot d_{\text{target}} \), where \( d_{\text{target}} \) is distance from formation center to target.
- Boundary reward: For drone \( i \), if distance to target \( d_i < d_{\text{arr_edge}} \), \( r_{\text{boundary},i} = 10 \sin\left(1.25\pi \cdot \frac{d_i + 3}{8}\pi\right) + 10 \).
- Success reward: \( r_{\text{success}} = 3000 \) if \( d_{\text{target}} < 0.2 \).
- Formation direction reward: \( r_{\theta} = 0 \) if \( \theta < 20^\circ \), else \( r_{\theta} = (\theta – 20)^2 / 100 \).
- Formation keeping reward: \( r_{\text{formation},i} = -\| p_i – (p_c + \Delta_i) \|^2 \).
- Collision penalty: For drone-drone distance \( d_{i,j} \) and drone-obstacle distance \( d_{i,ob} \), penalties are:
$$ r_{a_c} = -\alpha_{c_p} \cdot \left( \frac{1}{d_{i,j}} – \frac{1}{d_{a_s}} \right)^2 \text{ if } 0.04 < d_{i,j} < d_{a_s} $$
$$ r_{ob_c} = -\alpha_{c_p} \cdot \left( \frac{1}{d_{i,ob}} – \frac{1}{d_{ob_s}} \right)^2 \text{ if } 0.05 < d_{i,ob} < d_{ob_s} $$
$$ r_{ex} = -10 \text{ if } d_{i,j} \leq 0.04 \text{ or } d_{i,ob} \leq 0.05 $$
$$ r_{\text{collision}} = r_{a_c} + r_{ob_c} + r_{ex} $$
To manage multi-objective conflicts, we introduce a dynamic mode-switching mechanism based on multi-objective planning. When no obstacles are within a threshold \( d_{\text{switch}} \), the system operates in normal mode, optimizing formation keeping and navigation. If any drone detects an obstacle within \( d_{\text{switch}} \), it switches to avoidance mode, where formation and direction penalties are relaxed to prioritize obstacle avoidance. The total reward \( R \) is:
$$ R = \begin{cases} r_{\text{target}} + \sum_{i=1}^N r_{\text{boundary},i} + r_{\text{success}} + r_{\text{collision}} + \sum_{i=1}^N r_{\text{formation},i} + r_{\theta}, & \text{normal mode} \\ r_{\text{target}} + \sum_{i=1}^N r_{\text{boundary},i} + r_{\text{success}} + r_{\text{collision}}, & \text{avoidance mode} \end{cases} $$
This allows the drone formation to adaptively balance objectives, ensuring safety while maintaining task performance. The MO-MATD3 algorithm pseudocode is summarized in Table 1.
| Step | Description |
|---|---|
| 1 | Initialize Critic networks \( Q^1_i, Q^2_i \), Actor networks \( \mu_i \), and target networks for all N agents. |
| 2 | Initialize experience replay buffer \( B \). |
| 3 | For each episode up to MaxEpisode: |
| 4 | Reset environment to initial state \( o \). |
| 5 | For each time step up to MaxStep: |
| 6 | For each agent i, select action \( a_i = \mu_i(o_i) + \epsilon \). |
| 7 | Execute actions, observe rewards based on mode, and get next state \( o’ \). |
| 8 | Store transition \( (o, a, r, o’) \) in \( B \). |
| 9 | Sample batch from \( B \) and update Critic networks via TD loss. |
| 10 | Update Actor networks using policy gradient. |
| 11 | Soft-update target networks with rate \( \tau \). |
| 12 | End loops. |
We validate our approach through simulations in a 2D square environment built on OpenAI Gym with Python and TensorFlow. The setup includes a target point, N drones, and M dynamic obstacles with random velocities. Key parameters are listed in Table 2. Experiments are conducted on a Windows 11 system with Intel Core i7 processor, 32 GB RAM, and 6 GB GPU memory.
| Parameter | Value | Parameter | Value |
|---|---|---|---|
| Number of drones (N) | 3 (default) | Number of obstacles (M) | 8 (default) |
| Map size | 4×4 units | Drone radius | 0.02 |
| Obstacle radius | 0.03 | Obstacle speed range | [-0.02, 0.02] |
| Drone-drone safe distance | 0.07 | Drone-obstacle safe distance | 0.08 |
| Mode switch threshold | 0.3 | Learning rate (lr) | 0.001 |
| Discount factor (γ) | 0.95 | Replay buffer size | 1,000,000 |
| Batch size | 1024 | Max episodes | 10,000 |
| Max steps per episode | 45 | Target update rate (τ) | 0.01 |
| Noise clip (c) | 0.1 | Network units | 256/512 |
| Update interval (d) | 45 |
In comparative experiments, MO-MATD3 is tested against standard MATD3 in dynamic obstacle environments. Both algorithms use identical hyperparameters. The reward curves, plotted as total per-episode rewards versus training episodes, show that MO-MATD3 achieves faster convergence and higher stable rewards. Specifically, MO-MATD3 stabilizes in about 6,500 episodes, reducing the required episodes by approximately 23.53% compared to MATD3’s 8,500 episodes. The average post-convergence reward for MO-MATD3 is 8,802.99, outperforming MATD3’s 8,284.57, with smaller fluctuations. This indicates that MO-MATD3 enhances learning efficiency and performance in drone formation tasks, potentially lowering latency for real-world applications.
To assess generalization, we conduct experiments with increased drone counts and more challenging obstacle settings. First, for scalability, we increase the number of drones in the formation. Results demonstrate that MO-MATD3 maintains effective convergence and task completion, showing robust generalization across different swarm sizes. This underscores the algorithm’s adaptability to varying drone formation configurations. Second, we intensify environmental complexity by raising obstacle count to 10 and 12, with speed ranges expanded to [-0.04, 0.04] and [-0.06, 0.06], respectively. Despite these hurdles, MO-MATD3 continues to train successfully, enabling drones to navigate, avoid dynamic obstacles, and reach targets while preserving formation integrity. These tests confirm the algorithm’s strong generalization in complex dynamic settings for multi-drone operations.
Robustness is evaluated through parameter perturbation experiments. We modify key parameters and observe algorithm stability: discount factor γ from 0.95 to 0.92, learning rate lr from 0.006 to 0.00601, target update rate τ from 0.01 to 0.0101, and noise clip range from [-0.5, 0.5] to [-0.51, 0.51]. Under these variations, MO-MATD3 exhibits consistent learning trends, with reward curves gradually rising and converging despite initial increased volatility. This resilience highlights the algorithm’s robustness to parameter uncertainties, ensuring reliable performance in practical drone formation deployments where exact tuning may be difficult.
In summary, this work presents a novel MO-MATD3 algorithm for multi-drone formation control in dynamic obstacle environments. By integrating multi-objective planning with reinforcement learning, we enable adaptive strategy switching between formation keeping and obstacle avoidance. Dense reward design based on artificial potential fields accelerates learning, while virtual center coordination ensures formation stability. Extensive experiments validate the algorithm’s effectiveness, generalization, and robustness, showcasing its potential for real-world UAV swarm applications. Future work may explore 3D environments, heterogeneous drones, and real-time implementation challenges to further advance drone formation technology.
The drone formation control problem is inherently multi-faceted, requiring synchronization of navigation, collision avoidance, and geometric consistency. Our approach addresses these aspects through a unified RL framework, emphasizing the importance of multi-objective decision-making. The use of continuous rewards and mode switching allows drones to prioritize safety without sacrificing mission goals, a critical aspect for autonomous systems. Moreover, the scalability tests confirm that the algorithm can handle larger formations, which is essential for applications like aerial shows or coordinated surveillance. The robustness experiments further assure that slight parameter deviations won’t degrade performance, making it suitable for unpredictable real-world conditions.
From a technical perspective, the integration of MATD3 with multi-objective planning represents a significant step in MARL. The centralized Critic networks leverage global information during training to improve cooperation among drones, while decentralized execution ensures scalability and reduced communication overhead. The reward functions are carefully crafted to provide dense feedback, avoiding sparse rewards that hinder learning in complex tasks. For instance, the boundary reward uses a sinusoidal function to smooth approach behavior, reducing erratic movements near targets. The collision penalties are designed with inverse distance terms to create strong repulsive forces near obstacles, enhancing safety in dense environments.
The mode-switching mechanism is a key innovation, drawing from multi-objective optimization principles. By dynamically adjusting reward components based on environmental threats, the system mimics human-like prioritization: when danger is imminent, formation strictness is temporarily relaxed to enable evasive maneuvers. This flexibility is crucial for drone formation survival in cluttered spaces, as rigid adherence to formation could lead to collisions. The threshold \( d_{\text{switch}} \) acts as a sensor-based trigger, allowing proactive avoidance rather than reactive responses. This design balances the often conflicting demands of formation integrity and obstacle avoidance, a common challenge in multi-robot systems.
In simulation, the environment is designed to replicate real-world dynamics. Obstacles move with random velocities, testing the drones’ ability to predict and react to unpredictable motions. The fixed initial positions for drones and random target locations ensure diverse training scenarios, preventing overfitting. The use of a 2D plane simplifies computation while retaining core challenges, though extension to 3D would add altitude control complexities. The neural network architecture, with 256/512 units per layer, provides sufficient capacity to learn complex policies without excessive over-parameterization. Experience replay with a large buffer stabilizes training by decorrelating samples, while target network delays reduce Q-value overestimation—a known issue in actor-critic methods.
Comparative results clearly show MO-MATD3’s superiority over baseline MATD3. The reduction in convergence episodes implies faster training times, which is economically beneficial for real-world deployment where simulation costs matter. Higher average rewards indicate better task performance, meaning drones reach targets more efficiently while maintaining formation and avoiding obstacles. The lower reward variance post-convergence suggests more stable policies, reducing the risk of catastrophic failures during operation. These metrics collectively demonstrate that multi-objective planning enhances both learning speed and final policy quality in drone formation control.
Generalization experiments reveal the algorithm’s adaptability. Increasing drone numbers tests coordination scalability, as more agents introduce higher-dimensional state and action spaces. The successful outcomes suggest that the centralized training phase effectively captures inter-agent dependencies, allowing decentralized execution to scale. Similarly, harder obstacle settings simulate more chaotic environments, akin to urban canyons or disaster zones. The algorithm’s persistence shows that the reward shaping and mode switching generalize well to unseen difficulties, a promising sign for field applications where environmental conditions are unpredictable.
Robustness to parameter changes is vital because exact hyperparameter tuning is often impractical in real systems. The minor perturbations in discount factor, learning rate, etc., mimic calibration errors or sensor noises. MO-MATD3’s maintained convergence indicates that it is not overly sensitive to these changes, reducing deployment risks. This robustness stems from the algorithm’s dual Critic design and soft updates, which smooth learning dynamics. Additionally, the dense reward structure may provide richer gradients, making optimization more forgiving to parameter shifts.
Looking ahead, several extensions can build on this work. Incorporating vision-based sensors would allow drones to detect obstacles without pre-known positions, moving towards full autonomy. Multi-task learning could enable formations to switch between different geometric patterns on-the-fly. Energy efficiency objectives could be added to reward functions to prolong flight times. Furthermore, hardware-in-the-loop simulations and field tests with physical drones would validate simulation findings and uncover practical issues like communication delays or wind disturbances.
In conclusion, drone formation control is a pivotal area in robotics and AI, with wide-ranging applications from logistics to entertainment. Our MO-MATD3 algorithm offers a robust solution by marrying multi-agent reinforcement learning with multi-objective planning. Through innovative reward design and dynamic mode switching, it ensures that drones can navigate complex environments while preserving formation integrity. The empirical validation across multiple scenarios underscores its effectiveness, paving the way for more intelligent and adaptable UAV swarms in the future.