In recent years, the use of unmanned aerial vehicles (UAVs) in formation drone light shows has gained significant traction for entertainment, advertising, and artistic performances. These shows require precise coordination and path planning to create stunning visual patterns in the sky. However, the path planning for formation drone light shows in unknown dynamic environments poses substantial challenges, including obstacle avoidance, formation maintenance, and real-time adjustments. Traditional methods often rely on pre-defined environments, limiting their adaptability. Reinforcement learning (RL) offers a promising solution by enabling autonomous decision-making through interaction with the environment. In this article, I propose an intelligent decision-making scheme for formation drone light show path planning, incorporating a dynamic reward strategy based on multi-agent reinforcement learning. The key innovation lies in designing a dynamic formation reward function that ensures stable formation structures and allows for autonomous shape adjustments during performances, enhancing the robustness and visual appeal of formation drone light shows.
The core problem in formation drone light show path planning is to guide multiple drones from their starting positions to form specific patterns, navigate around obstacles, and reach target destinations while maintaining formation integrity. This involves two interconnected aspects: dynamic formation control and efficient path planning. For formation drone light shows, the drones must fly in stable formations, such as geometric shapes or logos, and adjust their formations in real-time to avoid obstacles or adapt to environmental changes. The essence is to keep the spacing between each pair of drones relatively stable while allowing minor tweaks based on external factors. This requires a sophisticated reward function that balances formation maintenance with obstacle avoidance. In this context, I extend sparse reward functions for obstacle-free environments and design improved reward functions for complex obstacle settings, focusing on the unique demands of formation drone light shows.
To model the formation drone light show path planning problem, consider a set of N drones operating in a 2D or 3D space. Each drone’s state includes its position, heading angle, velocity, and distances to other drones. The state space for drone i is defined as:
$$ s_i = [x_i, y_i, \theta_i, v_i, \mathbf{d}_i], \quad i=1,2,\dots,N $$
where \(x_i\) and \(y_i\) are the coordinates, \(\theta_i\) is the heading angle, \(v_i\) is the speed, and \(\mathbf{d}_i\) is an array of Euclidean distances to other drones. The action space consists of angular velocity and acceleration:
$$ A_i = [\omega_i, a_i], \quad i=1,2,\dots,N $$
subject to constraints such as maximum speed and angular limits. The kinematics are governed by:
$$ \dot{x}_i = v_i \cos \theta_i, \quad \dot{y}_i = v_i \sin \theta_i, \quad \dot{\theta}_i = \omega_i, \quad \dot{v}_i = a_i $$
For discrete time steps with interval \(\Delta T\), the state update is:
$$ x_i^{t+1} = x_i^t + v_i^t \Delta T \cos \theta_i^t, \quad y_i^{t+1} = y_i^t + v_i^t \Delta T \sin \theta_i^t, \quad \theta_i^{t+1} = \theta_i^t + \omega_i \Delta T, \quad v_i^{t+1} = v_i^t + a_i \Delta T $$
This model forms the basis for reinforcement learning in formation drone light shows, where drones must collaborate to achieve collective goals.
In formation drone light shows, the reward function is critical for guiding drones toward desired behaviors. For obstacle-free environments, I design a combination of sparse and guided rewards. The total reward for drone i is:
$$ R_i = \alpha_1 r_1 + \alpha_2 r_2 + \alpha_3 r_3 + \alpha_4 r_4 + \alpha_5 r_5 $$
where \(\alpha\) coefficients are set to 1 for simplicity. The components include: \(r_1\) for reaching the destination (e.g., positive reward based on distance and angle thresholds), \(r_2\) for collision penalties, \(r_3\) for distance-based progress toward the destination, \(r_4\) for alignment with the destination angle, and \(r_5\) for a step penalty to discourage unnecessary movements. For complex obstacle environments, additional rewards are introduced. An improved angle reward \(r_6\) encourages drones to follow optimal paths around obstacles, and a distance reward \(r_7\) uses a queue-based mechanism to reward consistent progress. A collision avoidance reward \(r_8\) penalizes inter-drone collisions. The total reward becomes:
$$ R_i = \alpha_1 r_1 + \alpha_2 r_2 + \alpha_5 r_5 + \alpha_6 r_6 + \alpha_7 r_7 + \alpha_8 r_8 $$
These rewards are tailored for formation drone light shows, ensuring that drones not only avoid obstacles but also maintain visual patterns.
To address dynamic formation control in formation drone light shows, I propose a dynamic formation reward function based on optimal and current distances between drone pairs. For drones i and j, the reward is:
$$ r_{d,ij} = -100 \left( \frac{d_i(j)}{d_{\text{opt},ij}} – 1 \right) \left( \frac{d_i(j)}{d_{\text{opt},ij}} – 0.9 \right) $$
where \(d_i(j)\) is the current distance, and \(d_{\text{opt},ij}\) is the optimal distance for the desired formation pattern. This function peaks when the distance is optimal, encouraging stability. The total dynamic formation reward for drone i is:
$$ r_9 = \sum_{j=1, j \neq i}^N r_{d,ij} $$
For obstacle-free formation drone light shows, the overall reward incorporates this as:
$$ R_i = \alpha_1 r_1 + \alpha_2 r_2 + \alpha_3 r_3 + \alpha_4 r_4 + \alpha_5 r_5 + \alpha_8 r_8 + \alpha_9 r_9 $$
For complex environments, it becomes:
$$ R_i = \alpha_1 r_1 + \alpha_2 r_2 + \alpha_5 r_5 + \alpha_6 r_6 + \alpha_7 r_7 + \alpha_8 r_8 + \alpha_9 r_9 $$
with \(\alpha_9 = 1\). This dynamic reward strategy enables formation drone light shows to maintain shapes while adapting to real-time challenges.
I integrate this reward function with the Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (MATD3) algorithm, proposing the MATD3-Incorporating Dynamic Formation Reward Function (MATD3-IDFRF) algorithm for formation drone light show path planning. MATD3 is an advanced multi-agent RL algorithm that uses twin critics to reduce overestimation and centralized training with decentralized execution. The algorithm involves policy networks \(\mu\) and critic networks \(q\) for each drone. The target Q-value for drone i is:
$$ y_i = r_i + \gamma \min_{j=1,2} q(\mathbf{o}, \mathbf{a}; w’_{i,j}) $$
where \(\gamma\) is the discount factor, \(\mathbf{o}\) is the joint observation, and \(\mathbf{a}\) is the joint action. Critics are updated by minimizing the loss:
$$ L(w_{i,j}) = \frac{1}{N} \sum_{i=1}^N (y_i – q(\mathbf{o}, \mathbf{a}; w_{i,j}))^2 $$
and policies are updated via gradient ascent. The dynamic formation reward \(r_9\) is included in \(r_i\), enhancing formation stability for formation drone light shows. The training process involves collecting experiences and updating networks iteratively, as summarized in the algorithm below.
| Parameter | Symbol | Value |
|---|---|---|
| Discount Factor | \(\gamma\) | 0.990 |
| Soft Update Coefficient | \(\tau\) | 0.010 |
| Batch Size | \(m\) | 1024 |
| Actor Learning Rate | \(\alpha_A\) | 0.008 |
| Critic Learning Rate | \(\alpha_C\) | 0.010 |
| Action Noise Standard Deviation | \(\sigma\) | 0.200 |
| Max Episodes | MaxEpisode | 10,000 |
| Max Steps per Episode | MaxStep | 100 |
To evaluate the MATD3-IDFRF algorithm for formation drone light shows, I design simulation experiments in complex obstacle environments. The setup includes static obstacles (e.g., buildings) and dynamic obstacles (e.g., moving threats) to mimic real-world scenarios. A wedge formation is used for the formation drone light show, with five drones arranged in a pattern. Key metrics include path smoothness, formation deformation rate, average reward, success rate, and convergence performance. The formation deformation rate is defined as:
$$ \text{Deformation\_rate} = \sum_{i=1}^n \text{offset}_i, \quad \text{offset}_i = \sum_{j=1}^n \frac{|d_i(j) – d_{\text{opt},ij}|}{d_{\text{opt},ij}} $$
This measures how much the formation deviates from the ideal pattern during the formation drone light show. Success rate is the proportion of episodes where all drones reach destinations without collisions. The results are compared against baseline algorithms like MADDPG and standard MATD3.
In the simulations, the MATD3-IDFRF algorithm demonstrates superior performance for formation drone light shows. The drones quickly form stable formations, adjust shapes to avoid obstacles, and reassemble afterward. The dynamic formation reward reduces deformation by up to 97% compared to MATD3, ensuring visually consistent patterns. The average reward converges faster and higher, indicating more efficient learning. For instance, the success rate improves by 6.8%, and the converged reward average increases by 2.3%. These enhancements are crucial for formation drone light shows, where precision and reliability are paramount. The table below summarizes key results.
| Algorithm | Average Path Length (m) | Formation Deformation Rate (%) | Converged Reward Average |
|---|---|---|---|
| MADDPG | 10,763 | 55.50 | 211 |
| MATD3 | 10,438 | 22.60 | 220 |
| MATD3-IDFRF | 10,536 | 0.68 | 575 |
The path planning for formation drone light shows benefits significantly from the dynamic reward strategy. By incorporating distance-based rewards, drones maintain optimal spacing, leading to stable formations. In obstacle-rich environments, the reward functions guide drones to navigate efficiently while preserving formation integrity. For example, the improved angle reward \(r_6\) calculates an optimal flight angle \(\theta_{\text{best}}\) based on obstacle tangents, defined as:
$$ \theta_{\text{best}} = \theta_{L} \pm \theta_{\epsilon} $$
where \(\theta_{L}\) is the tangent angle and \(\theta_{\epsilon}\) is a small deviation. The reward \(r_6\) is then:
$$ r_6 = \begin{cases} 2, & |\theta_{\text{act}} – \theta_{\text{best}}| \leq 5^\circ \\ 0, & 5^\circ < |\theta_{\text{act}} – \theta_{\text{best}}| \leq 30^\circ \\ -2, & \text{otherwise} \end{cases} $$
This encourages precise maneuvering in formation drone light shows. Similarly, the distance reward \(r_7\) uses a queue of recent distances to reward monotonic progress toward destinations, enhancing robustness. These components work together to create a cohesive system for formation drone light show path planning.
The MATD3-IDFRF algorithm’s training involves centralized critics that observe all drones’ states and actions, enabling coordinated learning for formation drone light shows. Each drone’s policy network outputs actions based on its own state, promoting scalability. The dynamic formation reward \(r_9\) is computed online, adding a layer of feedback that reinforces formation patterns. This is particularly important for formation drone light shows, where aesthetic patterns like spirals or waves require strict adherence to relative positions. The algorithm’s pseudo-code outlines the steps: initialize networks, collect experiences, update critics and policies, and repeat over episodes. Key equations include the target Q-value and loss functions, as shown earlier.
In terms of scalability, the approach can be extended to larger formation drone light shows with hundreds of drones. The reward functions can be adapted for 3D environments, allowing for more complex patterns. Future work may involve integrating real-time visual feedback or audience interactions into the reward system, making formation drone light shows more dynamic and engaging. The use of reinforcement learning enables continuous improvement, as drones learn from each performance to optimize paths and formations.
To further illustrate the reward mechanisms, consider the following breakdown for a formation drone light show. The sparse rewards \(r_1\) to \(r_5\) provide basic guidance, while the dynamic rewards \(r_6\) to \(r_9\) handle complex scenarios. The overall reward for drone i in a complex environment is:
$$ R_i = \sum_{k=1}^{9} \alpha_k r_k $$
with coefficients typically set to 1. This modular design allows for customization based on specific formation drone light show requirements, such as emphasizing formation stability over speed or vice versa. The table below lists the reward components and their purposes.
| Reward Component | Purpose | Formula |
|---|---|---|
| \(r_1\) | Destination reached | Based on distance and angle thresholds |
| \(r_2\) | Collision penalty | Fixed negative value |
| \(r_3\) | Distance progress | \(k \cdot (d^{t-1} – d^t)\) |
| \(r_4\) | Angle alignment | Reward for small angle differences |
| \(r_5\) | Step penalty | Constant negative value |
| \(r_6\) | Obstacle angle | Based on optimal flight angle |
| \(r_7\) | Distance consistency | Queue-based monotonic check |
| \(r_8\) | Inter-drone collision | Fixed negative value |
| \(r_9\) | Dynamic formation | \( \sum r_{d,ij} \) for distance stability |
The success of formation drone light shows hinges on efficient path planning and formation control. The proposed dynamic reward strategy addresses both aspects by incentivizing drones to maintain patterns while navigating dynamically. In simulation tests, drones using MATD3-IDFRF achieve smoother trajectories and lower deformation rates compared to alternatives. For instance, in a scenario with moving obstacles, the formation drone light show adapts by temporarily widening gaps, then reconverging, all guided by the reward function. This flexibility is key to resilient performances in unpredictable environments.
Moreover, the reinforcement learning framework allows for offline training and online execution, reducing computational demands during actual formation drone light shows. The policy networks can be deployed on drones with limited resources, enabling real-time decision-making. This is crucial for large-scale formation drone light shows, where centralized control may be impractical. The decentralized nature of the algorithm, combined with centralized training, offers a balance between coordination and autonomy.
In conclusion, the integration of dynamic reward strategies into multi-agent reinforcement learning significantly enhances formation drone light show path planning. The MATD3-IDFRF algorithm, with its tailored reward functions, enables drones to form stable patterns, avoid obstacles, and adapt in real-time. This leads to more reliable and visually appealing formation drone light shows, capable of handling complex environments. Future directions include incorporating 3D dynamics, integrating with swarm intelligence, and optimizing for energy efficiency. As formation drone light shows continue to evolve, such advanced path planning methods will play a pivotal role in pushing the boundaries of aerial entertainment.
The mathematical formulations and experimental results underscore the efficacy of this approach. The dynamic formation reward function, in particular, ensures that formation drone light shows maintain their artistic integrity under various conditions. By leveraging reinforcement learning, drones can learn optimal behaviors through experience, reducing the need for manual programming. This autonomy is transformative for formation drone light shows, allowing for more creative and complex performances. Overall, the proposed method represents a step forward in intelligent path planning for formation drone light shows, blending technology and art seamlessly.