The rapid advancement of unmanned aerial vehicle (UAV) technology has revolutionized numerous fields, from military reconnaissance and disaster response to environmental monitoring and logistics delivery. As the complexity of mission requirements grows, single-drone systems often struggle with limitations in sensing range, payload capacity, and fault tolerance. This has motivated the development of drone technology in the form of swarm coordination, where multiple UAVs collaborate to achieve tasks that surpass the sum of individual capabilities. The core enabler for such collective intelligence is autonomous coordination, which demands distributed decision-making under dynamic and partially observable conditions.
In recent years, multi-agent reinforcement learning (MARL) has emerged as a powerful paradigm for addressing coordination problems in UAV swarms. Unlike traditional rule-based or optimization-based methods, MARL allows each drone to learn optimal policies through interaction with the environment, enabling emergent cooperative behaviors without explicit programming. This survey provides a systematic review of MARL approaches applied to drone technology, covering fundamental theories, key techniques, typical applications, and open challenges. The discussion is organized into six sections: fundamentals of MARL, key technical frameworks, application scenarios, current limitations, future directions, and concluding remarks.
1. Fundamentals of Multi-Agent Reinforcement Learning
Reinforcement learning (RL) models sequential decision-making via the Markov decision process (MDP), defined by the tuple $$M = (S, A, P, R, \gamma)$$ where \(S\) is the state space, \(A\) the action space, \(P\) the transition probability, \(R\) the reward function, and \(\gamma\) the discount factor. The agent maximizes the expected cumulative reward $$J(\pi) = \mathbb{E}_\pi\left[\sum_{t=0}^\infty \gamma^t r_t\right].$$
In multi-agent settings, the environment is modeled as a partially observable stochastic game (POSG): $$G = (N, S, \{A_i\}, \{O_i\}, P, \{R_i\}, \gamma),$$ where \(N\) is the set of agents, each with its own action space \(A_i\) and observation space \(O_i\). The joint action \(\mathbf{a} = (a_1, \ldots, a_N)\) determines state transitions and rewards. This framework captures the non-stationarity caused by concurrently learning agents.
MARL algorithms can be classified along several dimensions, as summarized in Table 1.
| Dimension | Category | Description |
|---|---|---|
| Agent relationship | Fully cooperative | All agents share a common reward; team objective maximized. |
| Fully competitive | Zero-sum reward structure; Nash equilibrium sought. | |
| Mixed | General-sum games; requires opponent modeling or role assignment. | |
| Learning paradigm | Centralized training centralized execution (CTCE) | Central controller accesses full state and joint actions. |
| Decentralized training decentralized execution (DTDE) | Each agent learns independently, treating others as part of environment. | |
| Centralized training decentralized execution (CTDE) | Training uses global information; execution uses only local observations. | |
| Communication | Explicit communication | Agents exchange messages or gradients directly. |
| Implicit communication | Information is conveyed through actions or environmental effects. | |
| Optimization target | Value-based | Learn value function (e.g., Q-learning) and derive policy. |
| Policy-based | Directly optimize policy parameters (e.g., PPO, DDPG). |
Among these, CTDE has become dominant for drone swarms because it balances coordination performance with decentralized execution. Key examples include MADDPG, MAPPO, and QMIX. The CTDE paradigm is illustrated by the following formulation: during training, the centralized critic evaluates joint actions using global state, while at execution, each actor takes actions based solely on its local observation.
2. Key Techniques in MARL for Drone Swarms
2.1 Learning Paradigms
CTCE (centralized training centralized execution) treats the swarm as a single agent, suitable for small-scale systems but suffers from the curse of dimensionality. DTDE (e.g., IQL, IPPO) is scalable but struggles with non-stationarity. CTDE (e.g., MADDPG, MAPPO, MASAC) is the most widely adopted. For instance, the MAPPO algorithm extends PPO to multi-agent settings by sharing actor parameters and using a centralized value function. Its actor loss is:
$$\mathcal{L}(\pi) = \frac{1}{N}\sum_{i=1}^N \mathbb{E}_{o_i, a_i} \left[ \min\left( \frac{\pi_i(a_i|o_i)}{\pi_i^{\text{old}}(a_i|o_i)} A_i, \text{clip}\left(\frac{\pi_i(a_i|o_i)}{\pi_i^{\text{old}}(a_i|o_i)}, 1-\epsilon, 1+\epsilon\right) A_i \right) \right].$$
2.2 Communication Mechanisms
Effective information exchange is critical for coordination. Rule-based methods (e.g., event-triggered communication) are simple but rigid. Learnable communication has become mainstream. In CommNet, each agent broadcasts a message aggregated via average pooling: $$c_i^t = \frac{1}{N}\sum_{j} h_j^t,$$ where \(h_j^t\) is the hidden state of agent \(j\). Attention-based methods (e.g., TarMAC) dynamically weight messages: $$\alpha_{ij} = \frac{\exp(\text{score}(q_i, k_j))}{\sum_{k \neq i} \exp(\text{score}(q_i, k_k))},$$ with score computed as \(\text{score}(q,k) = q \cdot k / \sqrt{d}\). Gating mechanisms allow agents to decide when to communicate, reducing overhead. For drone swarms operating under bandwidth constraints, joint learning of action and communication policies is essential.
2.3 Credit Assignment
A key challenge in cooperative MARL is how to distribute team rewards among individual agents. Common approaches include:
- Difference rewards: Measure the marginal contribution of an agent by computing the team reward difference when the agent is included versus excluded.
- Value function factorization: VDN uses additive decomposition: \(Q_{\text{tot}}(s,\mathbf{a}) = \sum_i Q_i(o_i,a_i)\). QMIX enforces monotonicity via a mixing network: \(Q_{\text{tot}}(s,\mathbf{a}) = f_{\text{mix}}(Q_1, \ldots, Q_N; s)\) with \(\partial f_{\text{mix}} / \partial Q_i \geq 0\).
- Counterfactual reasoning: COMA computes individual advantage as \(A_i(s,\mathbf{a}) = Q(s,\mathbf{a}) – \sum_{a_i’} \pi_i(a_i’|o_i) Q(s, (a_i’, \mathbf{a}_{-i}))\).
Accurate credit assignment improves learning efficiency and policy interpretability in drone swarms.
2.4 Scalability
As swarm size grows, state-action space explodes. Graph neural networks (GNNs) model dynamic agent topologies: each node \(v_i\) aggregates messages from neighbors via permutation-invariant operations. Mean-field theory approximates interactions using average effects, reducing complexity. Hierarchical methods decompose tasks into high-level subtasks and low-level controls. For example, a two-level framework may have a high-level policy for target assignment and low-level controllers for trajectory generation.
3. Typical Applications in Drone Swarm Coordination
3.1 Formation Control
Formation control includes geometry maintenance, dynamic reconfiguration, and collision avoidance. MADDPG with attention-augmented critics has been used to keep “V” or “O” formations in GPS-denied environments. For dynamic reconfiguration, a hierarchical PPO framework uses a top-level strategy selector and bottom-level controller. Obstacle avoidance often integrates artificial potential fields with Dueling DQN or graph-based attention (PGAT). The reward function typically penalizes collisions and rewards formation error reduction and goal completion.
3.2 Path Planning
Two types of path planning are common: goal-oriented and task-oriented. Goal-oriented planning aims to guide drones from start to goal safely. Multi-agent soft actor–critic (MASAC) has been applied to heterogeneous UAV navigation with collision avoidance. Task-oriented planning addresses coverage and persistent surveillance. For coverage, a dual-grid input representation combined with double DQN enables generalization to varying swarm sizes. A MATD3+LSTM framework jointly optimizes trajectory and communication scheduling. The value function is decomposed into individual and joint components:
$$J(\pi) = \mathbb{E}_{\pi}\left[ \sum_{t=0}^T \gamma^t \left( r_i^{\text{self}} + r^{\text{joint}}\right) \right].$$
3.3 Task Allocation
Heterogeneous task allocation considers different drone capabilities and mission requirements. Overlapping coalition game theory combined with distributed RL allows drones to self-organize into task teams. Dynamic task allocation leverages multi-round decision-making to adapt to unexpected targets. Joint optimization of task assignment and path planning is modeled as a sequential decision process and solved via MADDPG, where the reward integrates travel distance, collision penalty, and completed tasks.
3.4 Target Tracking and Adversarial Games
For cooperative target tracking, two-stage methods use behavior cloning for warm-start followed by fine-tuning with prioritized experience replay. Pointwise mutual information (PMI) networks capture inter-agent dependencies: $$d_{ij} = \log \frac{p(f_i(a_i), f_j(a_j))}{p(f_i(a_i)) p(f_j(a_j))},$$ which weights neighbor contributions. In adversarial scenarios, self-play with a population of strategies enhances robustness. Evolutionary MARL introduces crossover and mutation operators to maintain diversity. Hierarchical reinforcement learning decomposes high-level tactical decisions (e.g., target assignment) from low-level continuous control.
4. Current Challenges
4.1 Sample Efficiency
MARL typically requires millions of interactions to converge, which is impractical for real drone deployments. Solutions include imitation learning (warm-start from expert demonstrations), curriculum learning (gradually increasing task difficulty), offline RL (learning from logged data), and model-based RL (using a learned dynamics model to generate synthetic trajectories). However, these techniques often compromise policy optimality or require high-quality demonstrations.
4.2 Scalability and Computational Cost
As the number of drones increases, the joint action space grows exponentially. Centralized critics become infeasible. While GNNs and mean-field approximations help, they assume locality or homogeneity. Hierarchical methods reduce complexity but may introduce suboptimality. Furthermore, real-time inference on resource-constrained onboard computers demands lightweight network architectures, which can degrade policy expressiveness.
4.3 Partial Observability and Information Uncertainty
Real-world sensors provide noisy, limited observations. Partial observability leads to aliasing and non-stationarity. Recurrent networks (LSTM/GRU) and attention mechanisms mitigate this by maintaining internal memory. However, they increase computational burden. In adversarial environments, GPS denial and jamming exacerbate uncertainty. Robust policy learning under such conditions remains an open problem.
4.4 Safety and Reliability
Deploying learned policies in physical drones risks collisions, system failures, or unsafe behaviors. Safety-aware MARL incorporates control barrier functions (CBFs) into the optimization objective: $$\min_{\pi} \mathcal{L}(\pi) \quad \text{s.t.} \quad \Delta h(s) + \alpha h(s) \geq 0,$$ where \(h(s)\) is a safety metric. Adversarial training improves robustness to disturbances, but balancing performance and safety is delicate. Redundant communication and heterogeneous sensor fusion can enhance fault tolerance.
4.5 Sim-to-Real Transfer
Policies trained in simulation often fail in reality due to model mismatch. Domain randomization varies simulation parameters (e.g., gravity, drag) to increase robustness. System identification and adaptive dynamics models align the simulator with real data. Domain adaptation aligns feature distributions. Despite progress, zero-shot transfer remains unreliable; incremental deployment with safety constraints is preferred.
5. Future Directions
From the perspective of system architecture, future drone swarms will likely employ hierarchical and modular designs. Sub-swarm clustering reduces communication overhead, while standardized interfaces separate high-level planning from low-level control. Open architectures support plug-and-play for heterogeneous platforms and dynamic topology reconfiguration under failures.
In algorithm design, research should focus on hybrid learning frameworks combining imitation, transfer, meta-learning, and offline RL to drastically reduce sample requirements. Lightweight neural models via knowledge distillation or sparse GNNs will enable onboard deployment. Confidence-aware policies that quantify uncertainty can trigger conservative actions when information is insufficient. Explainable MARL through credit attribution and decision visualization will foster trust and debugging.
On the platform side, cloud-edge collaborative training pipelines will leverage distributed simulation and hardware acceleration. Lightweight inference engines optimized for edge devices (e.g., NVIDIA Jetson, Qualcomm Snapdragon) will be essential. Standardized middleware (e.g., ROS2 + MQTT) will abstract hardware differences and provide consistent interfaces. Comprehensive testbeds combining software-in-the-loop, hardware-in-the-loop, and field trials will systematically validate robustness before deployment.
6. Conclusion
This survey has presented a holistic overview of multi-agent reinforcement learning approaches for autonomous coordination in UAV swarms, a pivotal area within modern drone technology. We reviewed fundamental theories, key technical frameworks (learning paradigms, communication, credit assignment, scalability), and typical applications (formation, planning, allocation, tracking, adversary). We identified critical challenges including sample efficiency, scalability, partial observability, safety, and sim-to-real transfer, and outlined promising future directions in system architecture, algorithm design, and platform support. The synergy between MARL and drone technology continues to push the boundaries of what autonomous swarms can achieve, and we anticipate that ongoing research will lead to more robust, scalable, and safe deployments in real-world scenarios.