In recent years, unmanned aerial vehicles (UAVs), commonly known as drones, have garnered significant attention due to their versatility and applicability across various domains. However, a single drone often struggles to handle complex multi-task scenarios due to hardware and software limitations. To address this, multi-drone formations, or drone swarms, have emerged as a promising solution, enabling autonomous cooperation and enhanced performance through coordinated information exchange. The control of such drone formations focuses on aggregation, establishment, and maintenance during movement, which presents substantial challenges, especially as the number of drones increases. Traditional methods, such as leader-follower, virtual structure, consensus theory, and backstepping control, offer precision but face scalability issues. Inspired by natural ecosystems, self-organizing swarm control methods utilize principles like separation, cohesion, and velocity alignment, allowing each drone to act as an intelligent entity. Nonetheless, extending these approaches to large-scale drone formations often leads to convergence difficulties and computational inefficiencies.
To overcome these limitations, we propose a novel drone formation model based on the Menger sponge fractal structure. This model leverages the self-similar properties of fractals to simplify formation construction and enhance scalability. By organizing drones into hierarchical formations, we can efficiently manage large clusters while maintaining stability and adaptability. Additionally, we introduce a reinforcement learning algorithm called MIX-MAPPO, which integrates multi-agent proximal policy optimization (MAPPO), proximal policy optimization (PPO), and an attention mechanism. This algorithm trains formation control strategies, reduces input dimensionality, and accelerates convergence, enabling drones to autonomously learn and execute tasks in dynamic environments. Our experimental results demonstrate that this approach outperforms existing methods like DDPG, PPO, MADDPG, and MAPPO in terms of convergence speed, reward values, and formation completion rates, highlighting its superiority in drone formation control.
The core of our work lies in the integration of fractal geometry with reinforcement learning for drone formation control. The Menger sponge, a three-dimensional fractal, exhibits self-similarity, meaning that parts of the structure resemble the whole. We exploit this property to design drone formations where sub-formations mirror the overall structure, enabling seamless aggregation and scalability. In this model, a primary drone formation consists of five drones: one leader drone positioned at the center and four follower drones evenly distributed around it. This configuration protects the leader from external threats and facilitates centralized communication within the sub-formation. The primary formation can be treated as a single entity, allowing it to aggregate into higher-level formations, such as secondary formations, through iterative processes. This hierarchical approach supports both small-scale centralized control and large-scale distributed coordination, making it ideal for complex multi-task scenarios.
To model the drone movements, we simplify each drone as a point mass in a Cartesian coordinate system. The motion of drone \(i\) is described by the following equations:
$$ x_i = \nu_i \cos \theta_i \cos \phi_i $$
$$ y_i = \nu_i \cos \theta_i \sin \phi_i $$
$$ z_i = \zeta $$
Here, \(\nu_i\) represents the flight speed, \(\theta_i\) is the pitch angle, \(\phi_i\) is the heading angle, and \(\zeta\) denotes a constant height in the z-axis direction, assuming level flight. The control inputs are applied to guide the drone’s trajectory, as defined by:
$$ \dot{p_i} = V_i $$
$$ \dot{V_i} = u_i $$
where \(p_i\) and \(V_i\) are the position and velocity of drone \(i\), respectively, and \(u_i = [u^x_i, u^y_i]\) represents the control inputs in the X and Y directions. These inputs adjust the drone’s speed and orientation, enabling precise formation control.
The drone formation structure is designed using the Menger sponge fractal. A primary formation, as mentioned, includes one leader and four followers, with a radius \(d_{ij}\) defining the distance between the leader and followers. The safety range is based on \(d_{safe}\), ensuring collision avoidance. The Laplacian matrix \(L\) for the primary formation reflects the communication links:
$$ L = D – A = \begin{bmatrix} 4 & -1 & -1 & -1 & -1 \\ -1 & 1 & 0 & 0 & 0 \\ -1 & 0 & 1 & 0 & 0 \\ -1 & 0 & 0 & 1 & 0 \\ -1 & 0 & 0 & 0 & 1 \end{bmatrix} $$
where \(D\) is the degree matrix and \(A\) is the adjacency matrix. The rank of 4 indicates mutual communication among the five drones. For higher-level formations, such as secondary formations, four primary formations aggregate using the same strategy, maintaining the fractal structure. This iterative process allows for n-level formations, enhancing scalability. The secondary formation preserves internal communication within primary formations while enabling leader-to-leader communication for distributed control.
To train the drone formation control strategy, we develop the MIX-MAPPO algorithm, which addresses the slow convergence and limited adaptability of MAPPO in multi-agent settings. MIX-MAPPO combines MAPPO, PPO, and an attention mechanism within a grouped framework. Specifically, follower drones use a critic network integrated with an attention mechanism that assigns weights to inputs based on their attention to other drones in the sub-formation. This reduces input dimensionality and improves learning efficiency. Leader drones employ a simplified PPO critic network that only considers their own state and action, speeding up convergence to target positions. The grouping mechanism categorizes drones based on heterogeneous information, such as task types (e.g., search, transport, operation), further optimizing the critic network. The loss function for drone \(i\) is defined as:
$$ L(\theta_i) = \frac{1}{S} \sum_{i=1}^{S} \left[ (Q^{\mu_i}(x_i, G_{X_{T_i}}(i)) – y)^2 \right] $$
with
$$ y = r_i + \gamma Q^{i’}(x_i, G_{X_{T_i}}(i)) \big|_{a_j’ = \mu_j'(o_j)} $$
Here, \(Q^{\mu_i}(x_i, G_{X_{T_i}}(i))\) is the grouped critic function, \(\theta_i\) represents the target network parameters, \(x_i\) is the environment state, \(r_i\) is the reward, and \(\gamma\) is the discount factor. The gradient update is given by:
$$ \nabla_{\theta_i} J(\mu_i) = \frac{1}{S} \sum_{i=1}^{S} \nabla_{\theta_i} \mu_i(a_i | o_i) \nabla_{a_i} Q^{\mu_i}(x_i, G_{X_{T_i}}(i)) $$
The grouping function \(G_{X_{T_i}}(i)\) connects drones based on their types, reducing linear growth in input dimensions and enhancing convergence. Algorithm 1 outlines the MIX-MAPPO process, emphasizing episodic training with experience replay and parameter updates.
Reward functions are crucial for training effective drone formation policies. We design distinct rewards for leader and follower drones, along with collision avoidance and environmental rewards. The leader reward encourages rapid movement to the target position:
$$ r_{lea} = -\min \| p_{lea} – p_{tar} \|^2 $$
where \(p_{lea}\) is the leader’s current position and \(p_{tar}\) is the target position. The follower reward ensures alignment with the leader and formation shape:
$$ r_f = -\min \| p_f – p_l \|^2 $$
with \(p_f\) as the follower’s position and \(p_l\) as the leader’s position. Collision avoidance is enforced through a safety distance \(d_{safe}\):
$$ r_{avoid} = \begin{cases} 0 & \text{if } d_{ij} \geq d_{safe} \\ -(d_{safe} – d_{ij}) & \text{if } d_{ij} < d_{safe} \\ -\alpha_{collision} & \text{if collision occurs} \end{cases} $$
where \(d_{ij}\) is the relative distance between drones, and \(\alpha_{collision}\) is a penalty constant. Environmental rewards simulate adverse conditions, attracting drones to the origin:
$$ r_{env} = -\min 0.1 \times \| p_{lea} – p_0 \|^2 $$
with \(p_0\) as the origin. These rewards collectively guide the drone formation toward stable aggregation and task completion.
We conduct extensive experiments to validate our approach. The training platform is built on OpenAI’s MPE library, with hardware including a Xenon E5-2620 CPU, 32GB RAM, and an NVIDIA 2070 Super GPU. Key parameters are summarized in Table 1, which includes learning rates, batch sizes, and network configurations. We compare MIX-MAPPO against DDPG, PPO, MADDPG, and MAPPO algorithms in terms of reward curves, training time, and formation completion rates.
| Parameter | Description | Value |
|---|---|---|
| \(\gamma\) | Discount factor | 0.95 |
| \(lr_C\) | Critic network learning rate | 0.01 |
| \(lr_A\) | Actor network learning rate | 0.001 |
| batch-size | Number of samples per batch | 1024 |
| num-units | Hidden layer neurons | 128 |
| num-layer | Fully connected network layers | 4 |
| max-episode-size | Maximum steps per episode | 100 |
| num-episodes | Total training episodes | 8000 |
| \(M\) | Experience replay buffer size | \(1 \times 10^6\) |
The reward curves, as shown in our experiments, indicate that MIX-MAPPO achieves higher cumulative rewards compared to other methods. Multi-agent algorithms like MIX-MAPPO, MAPPO, and MADDPG outperform single-agent DDPG and PPO, highlighting the importance of coordinated learning in drone formation control. Specifically, MIX-MAPPO exhibits faster convergence and superior stability, attributed to its grouped attention mechanism and fractal-based structure. Training time is another critical metric; Table 2 presents the time required for each algorithm to complete 8000 episodes. MIX-MAPPO has the shortest training time due to reduced input dimensionality and optimized network efficiency.
| Algorithm | Training Time (hours) |
|---|---|
| DDPG | 4.21 |
| PPO | 3.87 |
| MADDPG | 4.93 |
| MAPPO | 4.88 |
| MIX-MAPPO | 3.59 |
Formation completion rate (CR) measures the percentage of drones that reach their target positions in 100 experimental runs. As summarized in Table 3, MIX-MAPPO achieves a 97% completion rate, significantly higher than other algorithms. This demonstrates its effectiveness in ensuring robust drone formation aggregation and maintenance, even in dynamic environments.
| Algorithm | Completion Rate (%) |
|---|---|
| DDPG | 37 |
| PPO | 41 |
| MADDPG | 79 |
| MAPPO | 91 |
| MIX-MAPPO | 97 |
To further validate the practical applicability, we perform motion capture experiments using real drones. The trained policy from MIX-MAPPO successfully controls five drones from random positions to form a stable formation, maintain it during movement, and land smoothly. The distances between followers and the leader stabilize quickly and remain consistent, confirming the policy’s robustness. This real-world implementation underscores the feasibility of our approach for actual drone formation control scenarios, such as search and rescue or surveillance missions.
In conclusion, we present a comprehensive framework for drone formation control that combines fractal geometry with advanced reinforcement learning. The Menger sponge-based structure provides scalability and simplicity, while the MIX-MAPPO algorithm enables efficient, autonomous learning through grouped attention mechanisms. Our results show that this method outperforms existing algorithms in convergence speed, reward attainment, and formation completion, making it a superior choice for managing large-scale drone swarms. Future work will explore three-dimensional formations based on self-similar properties, extend validation to more drones, and refine grouping strategies for even more complex multi-agent tasks. The integration of fractal principles and reinforcement learning opens new avenues for intelligent drone formation control, promising enhanced performance in diverse applications.
The drone formation field continues to evolve, with challenges like environmental adaptability and real-time decision-making remaining at the forefront. Our approach addresses these by leveraging self-organizing structures and machine learning, offering a path toward more resilient and autonomous swarms. As drone technology advances, methods like MIX-MAPPO will play a crucial role in enabling coordinated behaviors, from artistic light shows to critical disaster response. The key takeaway is that fractal-based designs and attention-driven learning can significantly improve the efficiency and reliability of drone formation control, paving the way for next-generation aerial systems.