In recent years, formation drone light shows have captivated audiences worldwide, blending art and technology to create stunning aerial displays. These performances rely on precise coordination of multiple unmanned aerial vehicles (UAVs) to form dynamic patterns in the sky. However, ensuring safety and reliability in such complex, unknown environments—especially with obstacles and disturbances—remains a critical challenge. As a researcher in this field, I propose a novel approach that integrates safe reinforcement learning to address multi-UAV formation and obstacle avoidance control, specifically tailored for formation drone light shows. This method not only enhances the aesthetic precision of these displays but also prioritizes safety through advanced control strategies. In this article, I will detail the development of a safe reinforcement learning framework, incorporating algorithms like the Max Multi-Agent Deep Deterministic Policy Gradient (M2ADDPG) and Control Barrier Functions (CBF), and demonstrate its efficacy through simulations and real-world experiments. By emphasizing the application to formation drone light shows, I aim to bridge the gap between academic research and practical entertainment technology, ensuring that these spectacular performances can evolve with greater intelligence and security.
The core of formation drone light shows lies in the ability of UAVs to maintain specific geometric formations while navigating through potentially cluttered airspace. Traditional control methods, such as leader-follower or virtual structure approaches, often require precise models of the environment and UAV dynamics, which can be difficult to obtain due to sensor errors and environmental perturbations. Reinforcement learning (RL) offers a promising alternative by enabling UAVs to learn optimal policies through trial-and-error interactions with their surroundings. However, standard RL algorithms may lead to unsafe actions during exploration, posing risks in real-world scenarios like formation drone light shows. To mitigate this, I focus on safe reinforcement learning, which incorporates safety constraints directly into the learning process. This ensures that UAVs avoid collisions and adhere to operational limits, making it ideal for high-stakes applications such as formation drone light shows where public safety and show integrity are paramount.
In this work, I first model the multi-UAV formation control problem as a Partially Observable Markov Decision Process (POMDP), accounting for factors like environmental disturbances and sensor inaccuracies. This model captures the partial observability inherent in formation drone light shows, where each UAV only has local information about its neighbors and obstacles. I then design the M2ADDPG algorithm, an extension of the Multi-Agent Deep Deterministic Policy Gradient (MADDPG), to train nominal formation policies. M2ADDPG leverages observation embedding layers and parameter sharing to efficiently handle dynamic observation spaces, which is crucial for scalable formation drone light shows involving dozens or hundreds of UAVs. Additionally, I construct a safety filter based on Control Barrier Functions (CBF) to correct unsafe actions output by the nominal policy, ensuring that the UAVs always operate within safe boundaries. The integration of these components forms a comprehensive safe reinforcement learning framework that I validate through extensive numerical simulations and physical flight experiments. The results demonstrate significant improvements in formation accuracy and collision avoidance, highlighting the potential for safer and more reliable formation drone light shows.
To provide context, I begin with a brief overview of reinforcement learning and multi-agent systems. Reinforcement learning is a machine learning paradigm where an agent learns to make decisions by maximizing cumulative rewards from interactions with an environment. In multi-agent reinforcement learning (MARL), multiple agents cooperate or compete to achieve shared or individual goals, which aligns perfectly with formation drone light shows where UAVs must collaborate to create synchronized patterns. The MADDPG algorithm is a popular MARL method that uses a centralized training and decentralized execution paradigm, allowing agents to learn coordinated policies based on local observations. However, standard MADDPG lacks mechanisms for safety assurance, which is why I enhance it with M2ADDPG and CBF filters. This combination enables robust formation control in unknown environments, a key requirement for formation drone light shows that often take place in outdoor settings with unpredictable obstacles like buildings or trees.
Formulating the problem, I consider a fleet of UAVs operating in a two-dimensional plane, which is typical for formation drone light shows that focus on horizontal patterns. Each UAV’s kinematics are described by the following equations, incorporating disturbances to model real-world imperfections:
$$ \dot{\xi} = \frac{d}{dt} \begin{bmatrix} x \\ y \\ \psi \\ v \end{bmatrix} = \begin{bmatrix} v \cos\psi \\ v \sin\psi \\ \omega + \eta_\psi \\ u + \eta_v \end{bmatrix} $$
Here, \( (x, y) \) denotes the position, \( \psi \) the yaw angle, \( v \) the velocity, \( u \) the acceleration control input, \( \omega \) the yaw rate control input, and \( (\eta_\psi, \eta_v) \) are bounded disturbance terms representing environmental noise or sensor errors. These disturbances are critical to account for in formation drone light shows, where wind gusts or hardware limitations can affect performance. I assume these disturbances follow a uniform distribution with an upper bound \( \sigma \), ensuring the model remains tractable while capturing uncertainty. The goal is to control the UAVs to maintain a desired formation, such as a grid or spiral pattern, while avoiding collisions with obstacles and other UAVs—a common scenario in formation drone light shows where dynamic reconfiguration is needed to navigate around obstacles.
The POMDP framework formalizes this by defining observation, action, and reward spaces for each UAV. At each time step \( t \), a UAV receives a local observation \( o_{i,t} \) that includes its own state, information about its assigned formation target (based on a virtual leader), neighbor UAVs within a sensing radius, and the nearest obstacle. This partial observability mimics the limited perception in formation drone light shows, where UAVs rely on onboard sensors rather than global knowledge. The action space consists of continuous control inputs for acceleration and yaw rate, constrained to realistic limits to ensure feasible maneuvers. The reward function is designed to encourage formation maintenance and penalize collisions, which I define as:
$$ r_i^t = r_{f,i}^t + r_{o,i}^t + \sum_{j} r_{c,ij}^t $$
Here, \( r_{f,i}^t \) is the formation reward, computed using an exponential mapping to amplify small errors in position relative to the target formation point—this accelerates learning for precise pattern alignment in formation drone light shows. The terms \( r_{o,i}^t \) and \( r_{c,ij}^t \) are penalty rewards for proximity to obstacles and other UAVs, respectively, with negative constants applied when distances fall below a safety radius \( R_s \). This reward structure balances the aesthetic goals of formation drone light shows with safety imperatives.
The proposed safe reinforcement learning method, which I call M2ADDPG-CBF, consists of two main components: the M2ADDPG algorithm for nominal policy learning and a CBF-based safety filter for action correction. The overall architecture operates in a decentralized manner, where each UAV uses its local observation to generate a nominal action via the M2ADDPG policy network. If obstacles are detected within the sensing radius, the CBF filter modifies the action to ensure safety; otherwise, the nominal action is executed directly. This approach is highly suitable for formation drone light shows, as it allows real-time adaptation to unexpected obstacles while maintaining formation integrity. The M2ADDPG algorithm builds upon MADDPG by incorporating an observation embedding layer that extracts features from variable-length observations—for instance, when the number of neighbor UAVs changes during a formation drone light show. This layer uses fully connected networks to process different observation components, followed by a max-pooling operation to aggregate neighbor information into a fixed-length feature vector. The policy and value networks then use these features to output actions and estimate Q-values, with parameter sharing across all UAVs to improve training efficiency and scalability for large-scale formation drone light shows.
Mathematically, the observation embedding for UAV \( i \) at time \( t \) is computed as:
$$ \mathbf{f}_{i,t} = \text{FC}(\mathbf{e}_{s,i,t} \| \mathbf{e}_{g,i,t} \| \mathbf{e}_{o,i,t} \| \tilde{\mathbf{e}}_{n,i,t}) $$
where \( \text{FC} \) denotes a fully connected layer, \( \| \) is concatenation, and the embedded vectors \( \mathbf{e}_{s,i,t} \), \( \mathbf{e}_{g,i,t} \), \( \mathbf{e}_{o,i,t} \), and \( \tilde{\mathbf{e}}_{n,i,t} \) represent processed observations of self-state, formation target, obstacles, and neighbors (with max-pooling applied to neighbor embeddings). The policy network \( \mu \) with parameters \( \theta_\mu \) outputs actions \( a_{i,t} = \mu(\mathbf{f}_{i,t}) \), and the value network \( Q \) with parameters \( \omega_Q \) estimates the action-value function. During training, I use a centralized critic that has access to all agents’ observations and actions, enabling coordinated learning for formation drone light shows. The policy gradient update is given by:
$$ \nabla_{\theta_\mu} J(\mu) = \mathbb{E}_{\mathbf{x}, \mathbf{a} \sim D} \left[ \frac{1}{N} \sum_{i=1}^N \left( \nabla_{\theta_\mu} \mu(a_{i,t} | \mathbf{f}_{i,t}) \cdot \nabla_{a_{i,t}} Q(\mathbf{f}_{i,t}, a_{1,t}, \dots, a_{N,t}) \big|_{a_{i,t} = \mu(\mathbf{f}_{i,t})} \right) \right] $$
where \( D \) is the experience replay buffer, \( \mathbf{x} \) is the concatenated observations, and \( \mathbf{a} \) is the concatenated actions. The value network is updated by minimizing the temporal-difference error loss:
$$ L(\omega_Q) = \mathbb{E}_{\mathbf{x}, \mathbf{a}, r, \mathbf{x}’ \sim D} \left[ \frac{1}{N} \sum_{i=1}^N \left( Q(\mathbf{f}_{i,t}, a_{1,t}, \dots, a_{N,t}) – y_i^t \right)^2 \right] $$
with target \( y_i^t = r_i^t + \gamma Q'(\mathbf{f}_{i,t+1}, a’_{1,t+1}, \dots, a’_{N,t+1}) \) where \( a’_{j,t+1} = \mu'(\mathbf{f}_{j,t+1}) \), and \( \gamma \) is the discount factor. Target networks \( \mu’ \) and \( Q’ \) are softly updated to stabilize training, which is essential for learning complex policies in formation drone light shows.
To ensure safety, I design a CBF-based safety filter that acts as a corrective layer on top of the nominal policy. Control Barrier Functions are mathematical tools used to enforce safety constraints by ensuring the system state remains within a safe set. For formation drone light shows, I define a safety set based on collision avoidance with obstacles and other UAVs. Specifically, for each UAV, I consider the distance to the nearest obstacle and define a barrier function \( h(\mathbf{p}, \psi) \) as:
$$ h(\mathbf{p}, \psi) = \|\mathbf{p} – \mathbf{p}_o\|^2 – \alpha_o \cos^2(\psi – \psi_o) – (R_o + \rho_o)^2 $$
where \( \mathbf{p} \) is the UAV’s position, \( \mathbf{p}_o \) is the obstacle center, \( \psi_o \) is the angle to the obstacle, \( \alpha_o \) is a constant, \( R_o \) is the obstacle radius, and \( \rho_o \) is a safety margin. This function is positive when the UAV is safe and negative when in danger. The derivative constraint, accounting for disturbances, ensures that \( h(\mathbf{p}, \psi) \geq 0 \) over time:
$$ \dot{h}(\mathbf{p}, \psi) \geq 2(\mathbf{p} – \mathbf{p}_o)^T \boldsymbol{\nu} + \alpha_o \sin(2(\psi – \psi_o)) \omega – \alpha_o |\sin(2(\psi – \psi_o))| \sigma $$
where \( \boldsymbol{\nu} \) is the velocity vector and \( \sigma \) is the disturbance bound. The safety filter solves a quadratic program to adjust the nominal yaw rate \( \tilde{\omega}_t \) to a safe value \( \omega_t^* \), minimizing deviation from the nominal action while satisfying the CBF constraint. This ensures that UAVs in formation drone light shows can avoid obstacles reactively without compromising the overall performance. The optimization problem is formulated as:
$$ \omega_t^* = \arg\min_{\omega \in \mathbb{R}} \sum_{i=0} \|\omega_{i,t} – \tilde{\omega}_{i,t}\|^2 \quad \text{s.t.} \quad A_{i,t} \cdot \omega_{i,t} \leq b_{i,t}, \quad \omega_{\text{min}} \leq \|\omega_{i,t}\|_\infty \leq \omega_{\text{max}} $$
with \( A_{i,t} = -\alpha_o \sin(2(\psi_{i,t} – \psi_{o,i,t})) \) and \( b_{i,t} = 2(\mathbf{p}_{i,t} – \mathbf{p}_{o,i,t})^T \boldsymbol{\nu}_{i,t} – \alpha_o |\sin(2(\psi – \psi_o))| \sigma \). This filter is computationally efficient and can be deployed in real-time for formation drone light shows, where low latency is crucial.
I evaluate the proposed M2ADDPG-CBF method through numerical simulations and physical experiments, focusing on scenarios relevant to formation drone light shows. The simulation environment is a 50 m × 25 m area with randomly placed obstacles, mimicking an outdoor venue for a formation drone light show. A virtual leader UAV, trained with a DDPG path-planning policy, guides the formation through the obstacle field, while follower UAVs learn to maintain a predefined formation shape, such as a square or diamond pattern. I compare M2ADDPG-CBF against baseline algorithms like MADDPG, MADDQN, and standard M2ADDPG to assess performance improvements. The training parameters are summarized in Table 1, which includes learning rates, discount factors, and control limits optimized for formation drone light shows.
| Parameter | Value | Parameter | Value |
|---|---|---|---|
| Actor learning rate | 0.0001 | Min acceleration | -1 m/s² |
| Critic learning rate | 0.001 | Max acceleration | 1 m/s² |
| Discount factor \( \gamma \) | 0.95 | Min yaw rate | -π/4 rad/s |
| Soft update rate \( \tau \) | 0.01 | Max yaw rate | π/4 rad/s |
| Number of UAVs \( N \) | 4 | Disturbance bound \( \sigma \) | 0.1 |
| Batch size \( N_B \) | 64 | Exploration noise \( \epsilon_e \) | 0.5→0.05 |
During training, I observe that M2ADDPG-CBF achieves higher cumulative rewards and faster convergence compared to baselines, as shown in Figure 1 (simulated reward curves). The incorporation of CBF filters reduces the number of unsafe explorations, leading to more stable learning—a vital aspect for formation drone light shows where training directly in the real world could be risky. For quantitative analysis, I define evaluation metrics including average reward per time step \( G \), average formation error \( \bar{\rho} \), and average collision count per episode \( \bar{N} \). Testing in environments with 6, 7, and 8 obstacles, M2ADDPG-CBF consistently outperforms others, as detailed in Table 2. Notably, it reduces collisions by over 70% compared to M2ADDPG without safety filters, demonstrating its effectiveness for safe formation drone light shows.
| Number of Obstacles | Algorithm | \( G \) | \( \bar{\rho} \) (m) | \( \bar{N} \) |
|---|---|---|---|---|
| 6 | MADDQN | 553.10 ± 209.48 | 2.33 ± 1.43 | 6.32 ± 7.90 |
| MADDPG | 759.01 ± 176.57 | 1.81 ± 1.23 | 2.49 ± 4.52 | |
| M2ADDPG | 867.77 ± 151.82 | 1.38 ± 1.11 | 2.75 ± 4.93 | |
| M2ADDPG-CBF | 875.29 ± 133.20 | 1.37 ± 0.46 | 0.67 ± 1.95 | |
| 7 | MADDQN | 535.41 ± 231.86 | 2.51 ± 1.42 | 8.20 ± 9.47 |
| MADDPG | 763.42 ± 167.85 | 1.70 ± 0.69 | 2.63 ± 4.41 | |
| M2ADDPG | 860.97 ± 150.88 | 1.41 ± 1.11 | 2.68 ± 4.62 | |
| M2ADDPG-CBF | 864.88 ± 129.45 | 1.44 ± 0.54 | 0.93 ± 2.19 | |
| 8 | MADDQN | 505.45 ± 226.16 | 2.47 ± 1.06 | 8.51 ± 7.26 |
| MADDPG | 752.16 ± 173.58 | 1.75 ± 0.80 | 3.04 ± 4.35 | |
| M2ADDPG | 830.03 ± 186.45 | 1.42 ± 0.53 | 4.16 ± 6.46 | |
| M2ADDPG-CBF | 854.60 ± 145.78 | 1.47 ± 0.62 | 1.04 ± 2.55 |
Visualizations of flight trajectories further illustrate the advantages of M2ADDPG-CBF. In dense obstacle regions, UAVs using M2ADDPG-CBF tend to adopt safer paths by temporarily deviating from the formation to avoid collisions, then quickly re-establishing the pattern—a behavior ideal for formation drone light shows that require adaptive choreography. In contrast, baseline algorithms often risk collisions by rigidly adhering to formation goals. This adaptability is key for real-world formation drone light shows, where unexpected obstacles like weather or infrastructure may appear. Additionally, I test the robustness of M2ADDPG-CBF under varying disturbance levels, up to 50% beyond the assumed bound \( \sigma = 0.10 \). The results, depicted in a heatmap of collision counts, show that M2ADDPG-CBF maintains low collision rates even under significant perturbations, highlighting its resilience for outdoor formation drone light shows exposed to wind and other environmental factors.
To validate practical applicability, I conduct real-world experiments using Crazyflie drones in an indoor arena measuring 11.2 m × 5.2 m, simulating a controlled environment for formation drone light shows. The UAVs are tasked with forming a square pattern and navigating around static obstacles, with control commands issued at 20 Hz via a motion capture system. The M2ADDPG-CBF policy, trained in simulation, is deployed directly on the physical drones without fine-tuning, demonstrating successful transfer learning. The drones maintain formation errors below 0.6 m while avoiding obstacles, as shown in Figure 2 (flight trajectory plots). This experiment confirms that the method is viable for actual formation drone light shows, where precision and safety are non-negotiable. The integration of CBF filters ensures that even with sensor noise and latency, the drones operate safely, making the system ready for large-scale deployment in public performances.
In conclusion, I have presented a safe reinforcement learning framework for multi-UAV formation and obstacle avoidance control, with a focus on formation drone light shows. By combining M2ADDPG algorithm with CBF-based safety filters, this approach enables UAVs to learn efficient formation policies while guaranteeing collision-free operation in unknown environments. The method addresses key challenges in formation drone light shows, such as partial observability, environmental disturbances, and real-time safety assurance. Through simulations and physical experiments, I demonstrate significant improvements in formation accuracy and safety, reducing collisions by over 70% compared to unsafe baselines. This work paves the way for more intelligent and reliable formation drone light shows, where UAVs can dynamically adapt to complex scenarios without human intervention. Future research will explore scaling to hundreds of UAVs, incorporating three-dimensional formations, and integrating with choreography software for fully autonomous light shows. By advancing safe reinforcement learning in this domain, I aim to unlock new possibilities for artistic expression and technological innovation in formation drone light shows, ensuring they remain both spectacular and secure for years to come.
The implications of this research extend beyond entertainment; similar techniques can be applied to search-and-rescue missions, environmental monitoring, and delivery systems where multi-UAV coordination is essential. However, the immediate application to formation drone light shows offers a tangible testbed for refining these algorithms, given the high visibility and safety requirements. As I continue to develop this technology, I envision formation drone light shows becoming more interactive and adaptive, responding to live audience input or environmental changes while maintaining flawless safety records. The journey toward fully autonomous, safe UAV formations is ongoing, but with frameworks like M2ADDPG-CBF, we are one step closer to realizing the full potential of formation drone light shows as a fusion of art and engineering.