The field of unmanned aerial vehicle (UAV) technology has witnessed exponential growth, driven by advances in artificial intelligence and wireless communication. While single-UAV systems have proven valuable, multi-UAV systems, or drone formations, offer superior robustness, fault tolerance, and economic efficiency for complex tasks such as surveillance, agricultural mapping, and communication relays. The core challenge in realizing these benefits lies in effective formation control—the ability to guide a group of UAVs to maintain a specific geometric configuration while navigating through potentially cluttered and unknown environments.
Traditional control methods, including leader-follower, virtual structure, and consensus-based approaches, often rely on precise models of vehicle dynamics and environmental disturbances. However, in real-world scenarios, the dynamics are nonlinear and time-varying, and systems are subject to sensor noise and environmental perturbations, making accurate modeling difficult. This limitation has spurred significant interest in data-driven approaches, particularly Deep Reinforcement Learning (DRL). DRL enables an agent to learn optimal control policies through direct interaction with the environment, without requiring an explicit model. This end-to-end learning paradigm is highly promising for complex, non-linear tasks like drone formation and obstacle avoidance.
Recent studies have demonstrated the potential of DRL for multi-robot formation control. For instance, algorithms like DQN and PPO have been adapted to enable agents to navigate obstacle fields while maintaining formation structures. However, a critical gap persists in these methods: the lack of safety guarantees during the learning and deployment phases. In DRL, agents must explore unknown state-action spaces to learn optimal behavior. This exploration can lead to the execution of unsafe actions, such as collisions with obstacles or other agents within the drone formation. This safety-risk is exacerbated in continuous, high-dimensional action spaces typical of UAV control. Consequently, the development of Safe Reinforcement Learning (SRL) frameworks, which explicitly incorporate safety constraints into the learning process, has become a paramount research direction.
This article addresses the safety challenge in learning-based drone formation control. I present a comprehensive SRL framework that integrates a novel multi-agent policy learning algorithm with a certifiable safety filter. The core of our approach is a Max-pooling Multi-Agent Deep Deterministic Policy Gradient (M2ADDPG) algorithm, which efficiently learns a cooperative formation policy from local observations. Crucially, this nominal policy is safeguarded by a Control Barrier Function (CBF)-based safety filter. This filter acts as a corrective layer, minimally modifying the learning algorithm’s output actions to ensure they satisfy formal collision-avoidance constraints. The synergy between the learning-based policy and the model-based safety filter enables robust, safe, and efficient drone formation flight in unknown environments.
The visual representation above illustrates the dynamic and coordinated nature of a drone formation navigating through space. Achieving and maintaining such precise, adaptive configurations in the presence of obstacles and disturbances is the central problem tackled by the methodology described in the following sections.
1. Problem Formulation and Modeling
I begin by formally defining the multi-UAV formation and obstacle avoidance problem. The objective is to control a set of follower UAVs to adopt and maintain a pre-defined geometric configuration relative to a virtual leader, while avoiding static obstacles and inter-agent collisions. The virtual leader is not a physical entity but a reference point whose trajectory is planned separately, providing a global path for the drone formation to follow.
1.1 UAV Kinematic Model
Considering operation in a 2D plane (altitude hold), each follower UAV $i$ is modeled with the following kinematic equations, accounting for bounded disturbances:
$$
\dot{\xi}_i = \frac{d}{dt} \begin{bmatrix} x_i \\ y_i \\ \psi_i \\ v_i \end{bmatrix} = \begin{bmatrix} v_i \cos\psi_i \\ v_i \sin\psi_i \\ \omega_i + \eta_{\psi,i} \\ u_i + \eta_{v,i} \end{bmatrix}
$$
where $(x_i, y_i)^T \in \mathbb{R}^2$ is the position, $\psi_i \in [-\pi, \pi]$ is the heading angle, $v_i \in \mathbb{R}$ is the speed, and $u_i$ and $\omega_i$ are the control inputs for linear acceleration and angular velocity, respectively. The terms $\eta_{\psi,i}$ and $\eta_{v,i}$ represent bounded disturbances affecting the heading and speed channels, satisfying $|\eta_{\psi,i}| \leq \sigma_{\psi}$ and $|\eta_{v,i}| \leq \sigma_{v}$. Control inputs are constrained: $u_i \in [u_{min}, u_{max}]$, $\omega_i \in [\omega_{min}, \omega_{max}]$.
1.2 POMDP Model for Drone Formation
The problem is modeled as a Partially Observable Markov Decision Process (POMDP) for a team of $N$ follower agents.
- Observation Space ($o_i$): Each follower $i$ has a localized, partial view of the world:
$$ o_i = [o_i^s, o_i^g, o_i^n, o_i^o] $$- $o_i^s = [v_i, \psi_i]$: Self-state (speed, heading).
- $o_i^g = [\Delta d_i^g, \Delta \psi_i^g, v^l, \psi^l]$: Information related to its assigned formation goal (relative distance $\Delta d_i^g$, relative bearing $\Delta \psi_i^g$) and the virtual leader’s state $(v^l, \psi^l)$.
- $o_i^n = [\Delta d_{ij}^n, \Delta \psi_{ij}^n, \dots]$: Observations of neighboring followers $j$ within a perception radius (relative distances and bearings).
- $o_i^o = [\Delta d_i^o, \Delta \psi_i^o]$: Observation of the nearest obstacle within perception radius (distance and bearing).
- Action Space ($a_i$): The control inputs, $a_i = (u_i, \omega_i)$.
- Reward Function ($r_i$): Designed to incentivize formation keeping, obstacle avoidance, and inter-agent collision avoidance:
$$ r_i = r_i^f + r_i^o + \sum_{j \neq i} r_{ij}^c $$- Formation Reward: $r_i^f = e^{k_1 – k_2 \cdot \Delta d_i^g}$. This exponential shaping provides strong gradient information, encouraging precise convergence to the target formation position.
- Obstacle Penalty: $r_i^o = -P_1$ if $\Delta d_i^o \leq R_s$, else $0$.
- Collision Penalty: $r_{ij}^c = -P_2$ if $\Delta d_{ij}^n \leq R_s$, else $0$.
Here, $R_s$ is a safety radius, and $P_1, P_2$ are penalty constants.
2. Methodology: A Safe RL Framework for Drone Formation
The proposed framework, termed M2ADDPG-CBF, consists of two main components: 1) a learning-based nominal policy trained via the M2ADDPG algorithm, and 2) a certifiable safety filter based on Control Barrier Functions (CBF). The overall architecture ensures that the intelligent, cooperative behavior learned by the policy is always executed within safe bounds.
2.1 Max-pooling Multi-Agent Deep Deterministic Policy Gradient (M2ADDPG)
I adopt the centralized training with decentralized execution (CTDE) paradigm. The challenge with local observations in a drone formation is the variable dimensionality of $o_i^n$ due to a changing number of neighbors. To handle this, M2ADDPG introduces an observation embedding layer followed by a max-pooling operation.
Network Architecture:
- Observation Embedding: Four separate fully-connected (FC) networks process different parts of the observation:
$$ e_i^s = \text{FC}(o_i^s), \quad e_i^g = \text{FC}(o_i^g), \quad e_i^o = \text{FC}(o_i^o), \quad e_i^n = \text{FC}(o_i^n). $$ - Neighbor Feature Aggregation: Since $e_i^n$ contains features for a variable number of neighbors, a max-pooling layer (MP) is applied across the neighbor dimension to extract the most salient features, resulting in a fixed-size vector $\bar{e}_i^n = \text{MP}(e_i^n)$.
- Feature Fusion: The processed embeddings are concatenated and passed through a final FC layer to produce a fixed-length feature vector for agent $i$:
$$ f_i = \text{FC}(e_i^s \| e_i^g \| e_i^o \| \bar{e}_i^n). $$ - Policy and Value Networks: The policy network (actor) $\mu_{\theta}(a_i | f_i)$ and the centralized value network (critic) $Q_{\omega}(f_1, …, f_N, a_1, …, a_N)$ are then built upon this feature representation. Parameter sharing is employed across all follower agents to improve sample efficiency and promote homogeneous cooperative behavior in the drone formation.
Algorithm Update: The parameters are updated using experiences $(x_t, a_t, r_t, x_{t+1})$ sampled from a shared replay buffer $\mathcal{D}$.
- Policy Gradient: The shared policy parameters $\theta$ are updated to maximize the expected return:
$$ \nabla_{\theta} J(\mu) = \mathbb{E}_{x,a \sim \mathcal{D}} \left[ \frac{1}{N} \sum_{i=1}^N \nabla_{\theta} \mu(a_i | f_i) \nabla_{a_i} Q_{\omega}(x, a_1, …, a_N) \big|_{a_i = \mu(f_i)} \right]. $$ - Critic Loss: The value network parameters $\omega$ are updated by minimizing the temporal-difference error:
$$ \mathcal{L}(\omega) = \mathbb{E}_{x,a,r,x’ \sim \mathcal{D}} \left[ \frac{1}{N} \sum_{i=1}^N \left( Q_{\omega}(x, a) – y_i \right)^2 \right], $$
where the target $y_i = r_i + \gamma Q_{\omega’}(x’, \mu'(f_1′), …, \mu'(f_N’))$, and $\omega’$, $\theta’$ are parameters of target networks updated via soft updates.
2.2 Control Barrier Function (CBF) Safety Filter
The M2ADDPG policy outputs a nominal action $\bar{a}_i = (\bar{u}_i, \bar{\omega}_i)$. The CBF safety filter acts as a downstream module that minimally modifies this action, primarily the angular velocity $\bar{\omega}_i$, to ensure collision avoidance with the nearest obstacle.
CBF Design for Obstacle Avoidance: For a UAV with state $(p, \psi)$, where $p=(x,y)$, and the nearest obstacle centered at $p_o$ with radius $R_o$, I define a candidate barrier function $h(p, \psi)$:
$$ h(p, \psi) = \|p – p_o\|^2 – \alpha_o \cos^2(\psi – \psi_o) – (R_o + R_s)^2, $$
where $\psi_o$ is the bearing to the obstacle center, $\alpha_o > 0$ is a tuning parameter, and $R_s$ is the safety margin. The set $\mathcal{C} = \{ (p, \psi) | h(p, \psi) \geq 0 \}$ defines the safe states. The goal is to keep the system in $\mathcal{C}$.
Taking the time derivative along the system dynamics (including disturbance $\eta_{\psi}$) and applying the CBF condition to ensure safety yields a constraint on the admissible angular velocity $\omega$:
$$ \dot{h} = L_f h + L_g h \, \omega \geq -\kappa(h) – \alpha_o |\sin(2(\psi-\psi_o))| \sigma_{\psi}. $$
After derivation and simplification for the worst-case disturbance, the safety condition becomes a linear constraint on $\omega$:
$$ A_i \cdot \omega_i \leq b_i, $$
where
$$ A_i = -\alpha_o \sin(2(\psi_i – \psi_{o,i})), \quad b_i = 2(p_i – p_{o,i})^T \nu_i – \alpha_o |\sin(2(\psi_i-\psi_{o,i}))| \sigma_{\psi}. $$
Here, $\nu_i = (v_i \cos\psi_i, v_i \sin\psi_i)^T$ is the velocity vector.
Safety-Action Projection: The final safe action $a_i^* = (u_i^*, \omega_i^*)$ is obtained by solving a simple quadratic program (QP) that projects the nominal angular velocity onto the set of safe controls:
$$
\begin{aligned}
\omega_i^* = & \underset{\omega}{\text{argmin}} \quad \| \omega – \bar{\omega}_i \|^2 \\
& \text{subject to} \quad A_i \omega \leq b_i, \\
& \quad \quad \quad \omega_{min} \leq \omega \leq \omega_{max}.
\end{aligned}
$$
The linear acceleration $u_i^*$ is typically kept as the nominal output $\bar{u}_i$. This QP is solved in real-time. If the nominal action is already safe ($A_i \bar{\omega}_i \leq b_i$), then $\omega_i^* = \bar{\omega}_i$. If not, the QP finds the closest safe angular velocity, effectively creating a repulsive steering command to avoid the obstacle.
The integration of M2ADDPG and CBF creates a powerful SRL framework for drone formation. The learning algorithm handles the high-level coordination and goal-seeking behavior, while the CBF filter provides a low-level, mathematically guaranteed safety net against collisions.
3. Experimental Validation and Analysis
I conducted extensive numerical simulations and a real-world flight experiment to validate the proposed M2ADDPG-CBF framework for safe drone formation control.
3.1 Simulation Setup and Training
The training environment was a 50m x 25m area with randomly placed static obstacles. A virtual leader, equipped with a pre-trained DDPG path planner, guided a drone formation of four followers. The followers’ task was to maintain a square formation while following the leader and avoiding obstacles. Key training parameters are summarized below.
| Parameter | Value | Parameter | Value |
|---|---|---|---|
| Actor Learning Rate ($l_r^a$) | 0.0001 | $u_{min}$ | -1 m/s² |
| Critic Learning Rate ($l_r^c$) | 0.001 | $u_{max}$ | 1 m/s² |
| Discount Factor ($\gamma$) | 0.95 | $\omega_{min}$ | $-\pi/4$ rad/s |
| Soft Update Rate ($\tau$) | 0.01 | $\omega_{max}$ | $\pi/4$ rad/s |
| Batch Size ($N_B$) | 64 | Disturbance Bound ($\sigma_{\psi}, \sigma_v$) | 0.05 |
I compared M2ADDPG-CBF against three baseline algorithms: MADDQN, standard MADDPG, and the standalone M2ADDPG (without the CBF filter). All used parameter sharing.
3.2 Training Performance and Quantitative Evaluation
The learning curves showed that M2ADDPG converged faster and to a higher average return than MADDPG, demonstrating the benefit of the neighbor feature aggregation via max-pooling. Crucially, M2ADDPG-CBF achieved the highest and most stable return, indicating that the safety filter prevented catastrophic failures during training, allowing the policy to learn more efficiently.
For quantitative evaluation, I defined three metrics over 100 test episodes in environments with 6, 7, and 8 obstacles:
- Average Return per Step ($G$): Higher is better.
- Average Formation Error ($\bar{\rho}$): Euclidean distance to assigned formation point, lower is better.
- Average Collisions per Episode ($\bar{N}_c$): Lower is better.
The results are consolidated in the table below. M2ADDPG-CBF consistently achieved the highest safety (lowest $\bar{N}_c$) while maintaining competitive formation accuracy. The CBF filter drastically reduced collisions compared to the unfiltered M2ADDPG, validating its critical role in ensuring safe drone formation operations.
| # Obstacles | Algorithm | $G$ (Higher Better) | $\bar{\rho}$ [m] (Lower Better) | $\bar{N}_c$ (Lower Better) |
|---|---|---|---|---|
| 6 | MADDQN | 553.1 ± 209.5 | 2.33 ± 1.43 | 6.32 ± 7.90 |
| MADDPG | 759.0 ± 176.6 | 1.81 ± 1.23 | 2.49 ± 4.52 | |
| M2ADDPG | 867.8 ± 151.8 | 1.38 ± 1.11 | 2.75 ± 4.93 | |
| M2ADDPG-CBF | 875.3 ± 133.2 | 1.37 ± 0.46 | 0.67 ± 1.95 | |
| 7 | MADDQN | 535.4 ± 231.9 | 2.51 ± 1.42 | 8.20 ± 9.47 |
| MADDPG | 763.4 ± 167.9 | 1.70 ± 0.69 | 2.63 ± 4.41 | |
| M2ADDPG | 861.0 ± 150.9 | 1.41 ± 1.11 | 2.68 ± 4.62 | |
| M2ADDPG-CBF | 864.9 ± 129.5 | 1.44 ± 0.54 | 0.93 ± 2.19 | |
| 8 | MADDQN | 505.5 ± 226.2 | 2.47 ± 1.06 | 8.51 ± 7.26 |
| MADDPG | 752.2 ± 173.6 | 1.75 ± 0.80 | 3.04 ± 4.35 | |
| M2ADDPG | 830.0 ± 186.5 | 1.42 ± 0.53 | 4.16 ± 6.46 | |
| M2ADDPG-CBF | 854.6 ± 145.8 | 1.47 ± 0.62 | 1.04 ± 2.55 |
3.3 Robustness and Trajectory Analysis
A robustness test was conducted by increasing the maximum disturbance bound $\sigma$ during evaluation. The drone formation controlled by M2ADDPG-CBF maintained a low collision rate even when $\sigma$ exceeded the nominal training value by 50%, demonstrating the algorithm’s resilience to unmodeled dynamics and perturbations.
Trajectory visualization revealed distinct behavioral strategies. In dense obstacle fields, the unfiltered M2ADDPG policy often led the entire drone formation into narrow, high-risk passages to preserve formation shape, sometimes resulting in collisions. In contrast, the M2ADDPG-CBF controller prioritized safety: followers would temporarily break the strict formation to maneuver safely around obstacles, seamlessly re-establishing the formation once the threat passed. This illustrates the effective trade-off managed by the SRL framework between cooperative task performance and hard safety constraints.
3.4 Real-World Flight Experiment
To validate practical applicability, I deployed the trained M2ADDPG-CBF policy on a physical system of four Crazyflie 2.0 micro-UAVs in an 11.2m x 5.2m indoor motion-capture arena. A central computer ran the policy, processing real-time positional data and sending control commands to each UAV at 20 Hz. Obstacles were virtually defined in the software, with only those within a UAV’s perception radius being observed.
The drone formation successfully transitioned from a compact initial shape to a wider square formation and navigated through the virtual obstacle field. The CBF filter actively intervened when obstacles were detected, causing slight deviations from the ideal formation points to ensure collision avoidance. The formation error throughout the experiment remained below 0.6 meters, confirming that the policy effectively balanced the formation-keeping objective with the imperative of safe navigation. This real-world experiment conclusively demonstrates the viability and reliability of the proposed safe RL framework for physical drone formation control.
4. Conclusion and Future Directions
This article presented a novel Safe Reinforcement Learning framework to address the critical challenge of safety in learning-based multi-UAV formation control. By integrating the M2ADDPG algorithm—which enhances multi-agent coordination through structured observation processing—with a certifiable CBF-based safety filter, I developed a system where intelligent, learned cooperative behavior is executed within formally defined safe bounds. The M2ADDPG component efficiently learns a policy for maintaining a drone formation and following a leader, while the CBF filter acts as a safeguard, minimally modifying actions to prevent collisions with obstacles. This combination ensures that the drone formation can operate reliably in unknown and cluttered environments.
Extensive numerical simulations and a successful real-world flight experiment validated the framework’s effectiveness. Compared to standard multi-agent RL baselines, M2ADDPG-CBF achieved superior safety performance (significantly fewer collisions) while maintaining excellent formation accuracy and robustness to disturbances. The results underscore the importance of incorporating explicit safety mechanisms into the RL pipeline for deployment in safety-critical systems like autonomous drone formations.
Future work will focus on several exciting extensions. First, integrating the safety filter directly into the policy learning process, perhaps through constrained policy optimization or differentiable CBFs, could lead to even more efficient and inherently safe policies. Second, extending the CBF formulation to actively guarantee inter-agent collision avoidance within the drone formation, in addition to obstacle avoidance, is a logical next step. Third, scaling the approach to larger swarms and more complex 3D environments with dynamic obstacles presents a significant challenge and opportunity. Finally, investigating the transfer of policies learned in simulation to more diverse and uncertain real-world conditions remains a key research avenue for robust autonomous drone formation systems.