In the rapidly evolving landscape of intelligent transportation systems, the integration of drone technology has emerged as a transformative approach to address the growing challenges of urban traffic congestion and monitoring coverage limitations. Our research focuses on developing a novel multi-agent reinforcement learning framework to solve the cooperative path planning problem for multiple unmanned aerial vehicles engaged in traffic monitoring missions, with explicit consideration of battery replacement station constraints. This work represents a significant advancement in applying drone technology to real-world traffic surveillance applications, where the ability to coordinate multiple aerial platforms efficiently can dramatically enhance the spatial and temporal coverage of traffic data collection.
The motivation for our research stems from the inherent limitations of conventional traffic monitoring infrastructure. Fixed sensors such as inductive loop detectors, surveillance cameras, and roadside radar units suffer from restricted coverage areas and vulnerability to occlusion caused by buildings, vegetation, and other vehicles. While mobile sensing approaches using probe vehicles and floating car data provide some flexibility, they cannot offer the comprehensive aerial perspective that drone technology affords. Unmanned aerial vehicles equipped with high-resolution cameras and LiDAR sensors can traverse large areas rapidly, providing real-time traffic flow parameters, incident detection, and congestion monitoring capabilities that far surpass traditional ground-based methods. However, the practical deployment of drone technology for traffic monitoring faces critical challenges, particularly regarding limited battery endurance and the need for coordinated multi-vehicle operations.
In typical urban traffic monitoring scenarios, a fleet of drones must simultaneously monitor multiple critical locations including major intersections, expressway entrances and exits, and accident-prone segments. These target points are predetermined based on historical traffic data and urban planning considerations. The drones collect traffic parameters such as vehicle counts, speeds, and densities through onboard sensors, transmitting this information via 5G wireless communication networks to traffic management centers in real time. The acquired data supports various applications including traffic flow analysis, incident detection and response, signal control optimization, and traffic management decision-making. Given the finite battery capacity of each drone and the requirement to complete monitoring missions within specified time windows, the problem of maximizing coverage becomes a complex combinatorial optimization challenge.
Our work formulates this multi-UAV cooperative traffic monitoring problem as a variant of the Team Orienteering Problem (TOP), where the objective is to maximize the total reward collected from visited target nodes while respecting battery endurance constraints and mission time limits. We introduce battery replacement stations as intermediate replenishment points that allow drones to extend their operational range significantly. This innovation in drone technology application enables persistent monitoring capabilities that were previously unattainable with single-dispatch approaches. The battery replacement stations are strategically located using clustering methods to ensure uniform spatial distribution across the monitoring area, thereby maximizing the accessibility of charging infrastructure for all operational drones.
Problem Formulation and Mathematical Modeling
The multi-UAV cooperative traffic monitoring problem is defined on a directed graph G = (V, E), where V represents the set of all nodes including the depot (nodes 0 and n+1), the set of target points V’ = {1, …, n}, and the set of battery replacement stations S = {s₁, s₂, …, sk}. The edge set E carries weights t_ij representing the travel time between nodes i and j, calculated as the Euclidean distance divided by the drone’s constant flight velocity. Each target point j ∈ V’ requires a hovering monitoring time t_j and provides a reward r_j upon successful visitation. The depot serves as both the starting point and the final destination for all drones, while battery replacement stations can be visited multiple times by different drones to replenish their energy reserves.
The Mixed Integer Linear Programming (MILP) formulation of this problem captures all essential operational constraints. Let φ_ij,a be a binary decision variable indicating whether drone a traverses from node i to node j. The objective function maximizes the total collected reward:
$$ \max R = \sum_{j \in V’} r_j \sum_{i \in V} \sum_{a \in A} \phi_{ij,a} $$
subject to the following constraints that ensure operational feasibility. The departure constraint guarantees that all drones depart from the depot:
$$ \sum_{j=1}^{n+1} \phi_{0j} + \sum_{s \in S} \phi_{0s} = |A| $$
The return constraint ensures all drones return to the depot after completing their missions:
$$ \sum_{i=0}^{n} \phi_{i(n+1)} + \sum_{s \in S} \phi_{s(n+1)} = |A| $$
Flow conservation constraints maintain the continuity of each drone’s trajectory. For every target node j ∈ V’, the inflow must equal the outflow:
$$ \sum_{i=0}^{n} \phi_{ij} + \sum_{s \in S} \phi_{sj} = \sum_{i’=1}^{n+1} \phi_{ji’} + \sum_{s \in S} \phi_{js} $$
The total mission time constraint limits the cumulative flight and monitoring duration for each drone:
$$ \sum_{i=0}^{n} \sum_{j=1}^{n+1} t_{ij}\phi_{ij,a} + \sum_{i \in V} \sum_{s \in S} t_{is}\phi_{is,a} + \sum_{s \in S} \sum_{j \in V} t_{sj}\phi_{sj,a} \leq T $$
Sub-tour elimination constraints prevent the formation of cycles among target nodes:
$$ u_{i,a} – u_{j,a} + n \cdot \phi_{ij,a} \leq n-1 \quad \forall i,j \in V’ $$
Battery capacity constraints ensure that the remaining flight time at any target node is positive and does not exceed the maximum battery capacity B:
$$ 0 < b_{j,a} \leq B \quad \forall j \in V’ $$
The battery update equation models energy consumption during flight and monitoring:
$$ b_{j,a} = b_{i,a} – (t_{ij} + t_j) \cdot \phi_{ij,a} $$
Battery replacement stations restore the drone to full capacity upon visitation:
$$ b_{j,a} = B \quad \forall j \in \{0, n+1\} \cup S $$
This comprehensive MILP model provides an exact mathematical representation of the multi-UAV cooperative traffic monitoring problem. However, due to its NP-hard nature, solving large-scale instances using commercial solvers becomes computationally prohibitive. This limitation motivates our development of a learning-based approach that can generate high-quality feasible solutions with minimal computational overhead.
Battery Replacement Station Location Using Clustering Methods
The spatial distribution of battery replacement stations critically influences the efficiency of drone operations in traffic monitoring missions. We employ the K-Means Constrained (KMC) clustering algorithm to determine optimal station locations based on target point distribution. This approach ensures that each battery replacement station serves a balanced subset of target points, minimizing the maximum distance between any target point and its nearest charging facility. The coordinates of each battery replacement station are computed as the centroid of its assigned target point cluster:
$$ x_k = \frac{1}{|C_s|} \sum_{x_i \in C_s} x_i $$
$$ y_k = \frac{1}{|C_s|} \sum_{y_i \in C_s} y_i $$
where C_s represents the s-th cluster with |C_s| target points, and (x_k, y_k) denotes the coordinates of the corresponding battery replacement station. To validate the effectiveness of KMC, we conducted comprehensive comparisons against four alternative clustering methods: Fuzzy C-Means (FCM), Density-Based Spatial Clustering of Applications with Noise (DBSCAN), Spectral Clustering (SC), and Gaussian Mixture Model (GMM).
The evaluation metrics included the Calinski-Harabasz (CH) index measuring cluster compactness and separation, Minimum Distance (MD) between station locations, Coverage (CO) representing the proportion of target points within service radius, and Uniformity (UN) quantifying distribution evenness. Our experimental results across 256 random instances demonstrated that KMC consistently outperforms alternative methods across all four metrics. The average CH index for KMC reached 51.646 compared to 33.629 for DBSCAN, while the coverage rate achieved 74.102% versus 60.547% for DBSCAN. The minimum distance between stations was 0.329 units for KMC, significantly higher than 0.230 for DBSCAN, indicating more uniform spatial distribution. These results confirm that KMC provides superior battery replacement station placement that facilitates efficient drone operations and maximizes mission coverage.
The Multi-Agent Transformer Reinforcement Learning Framework
To overcome the computational limitations of exact optimization methods for large-scale instances, we developed the Multi-agent Transformer Reinforcement Learning (MTRL) framework. This approach models the cooperative path planning problem as a Markov Decision Process and leverages the powerful representation capabilities of Transformer architectures to learn effective coordination strategies. The MTRL framework adopts a centralized encoder-decoder architecture that processes global state information and generates joint action decisions for all drone agents simultaneously.
The encoder component of MTRL employs multi-head self-attention mechanisms to capture global dependencies among all nodes in the problem instance. Each node’s initial embedding is computed through linear projection of its coordinate and time consumption information. The attention mechanism computes compatibility scores between node pairs:
$$ \mu_{ij} = \frac{q_i \cdot k_j^{\top}}{\sqrt{d_k}} $$
where q_i and k_j represent the query and key vectors for nodes i and j respectively, and d_k denotes the dimensionality of these vectors. The attention weights are obtained through softmax normalization:
$$ y_{ij} = \frac{e^{\mu_{ij}}}{\sum_{j’} e^{\mu_{ij’}}} $$
Through M layers of multi-head attention with residual connections and batch normalization, the encoder produces enhanced node embeddings that incorporate global contextual information. The graph-level embedding is obtained by averaging all node embeddings:
$$ \bar{h} = \frac{1}{|V|} \sum_{i \in V} h_{i,M} $$
The decoder component treats each drone as an independent agent that sequentially selects nodes to visit. At each decision step δ, the decoder receives the current state s_δ comprising static information (node coordinates, rewards, time consumption) and dynamic information (drone positions, remaining battery levels, accumulated mission time, and visitation masks). The action a_δ,j for each drone involves selecting the next node to visit from the set of feasible candidates determined by mask constraints.
The state embedding at step δ integrates graph-level information, node embeddings, and drone battery status:
$$ h_{\delta,c} = \text{cat}(\bar{h}, h_{1,M}:h_{|V|,M}, b_{1,\delta}:b_{|A|,\delta}) $$
The decoder applies multi-head cross-attention to compute compatibility scores between the current state and candidate nodes:
$$ h_{\delta,c}’ = \text{MHA}(h_{\delta,c}, W_{c,q} \cdot h_{\delta,c}, W_{M,k} \cdot h_M, W_{M,v} \cdot h_M) $$
$$ \mu_{i,u,\delta} = \tau \cdot \tanh\left(\frac{q_\delta \cdot k_{i,\delta}^{\top}}{\sqrt{d_k}}\right) $$
To ensure solution feasibility, we introduce a structured masking mechanism that eliminates invalid actions. Six distinct mask types are defined: repetition constraint (M₁) prevents revisiting target nodes already visited by any drone; energy constraint (M₂) masks nodes unreachable due to insufficient remaining battery; cumulative time constraint (M₃) ensures the total mission time does not exceed the limit; consecutive station visit constraint (M₄) prevents inefficient transitions between battery replacement stations; safety margin constraint (M₅) ensures drones maintain sufficient battery to reach at least one station after visiting a target; and depot access rule (M₆) controls the starting and ending depot access. The final mask is obtained through logical OR operation:
$$ M_{\delta} = M_1 \cup M_2 \cup M_3 \cup M_4 \cup M_5 \cup M_6 $$
The reward function is designed to maximize the number of target nodes visited collectively by all drones:
$$ R_t = \sum_{a \in A} \sum_{\delta=1}^{T_a} r_{\delta,a} $$
where r_{\delta,a} equals 1 when drone a visits a target node at step δ, and 0 otherwise (for depot and battery replacement stations). This reward structure incentivizes the team to coordinate effectively and maximize coverage within the given constraints.
The training process employs policy gradient methods with a baseline network for variance reduction. The policy network σ learns to compute probability distributions over actions, while the baseline network σ_BL generates baseline rewards using greedy action selection. Both networks share identical architecture. A one-sided t-test determines whether the policy network significantly outperforms the baseline network; when this condition is met, the baseline network parameters are updated to match the policy network parameters. This approach ensures stable and efficient training convergence.
Experimental Setup and Performance Evaluation
Our experimental evaluation covers nine distinct scenario configurations spanning three scale levels (small: 20 targets, medium: 50 targets, large: 100 targets) with varying numbers of battery replacement stations and drone agents (2, 3, and 4 drones). All experiments use a 1 km × 1 km square region with uniformly distributed target points. The drone parameters are based on the DJI Matrice M300 RTK commercial platform, with a maximum flight speed of 12 m/s, battery endurance of 40 minutes, and monitoring time per target randomly assigned between 0 and 10 minutes. Battery replacement time is set to 1 minute, and all stations have unlimited battery supplies.
Table 1 presents the comprehensive comparison of MTRL against CPLEX (with 120-second time limit), Tabu Search (TS), Genetic Algorithm (GA), and Pointer-Network-based Reinforcement Learning (PNRL) across all nine scenarios. The results clearly demonstrate MTRL’s superior performance in both solution quality and computational efficiency.
| Method | Small Scale (T20B2) | Medium Scale (T50B5) | Large Scale (T100B10) | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| A2 | A3 | A4 | Avg | Time | A2 | A3 | A4 | Avg | Time | A2 | A3 | A4 | Avg | Time | |
| CPLEX | 18.22 | 16.64 | 16.28 | 17.05 | 96.9s | 27.26 | 37.73 | 43.68 | 36.22 | 109.5s | 41.68 | 52.76 | 64.39 | 52.94 | 118.3s |
| TS | 18.01 | 19.99 | 20.00 | 19.33 | 19.5s | 23.18 | 32.14 | 39.48 | 31.60 | 31.7s | 24.75 | 35.54 | 45.06 | 35.12 | 45.3s |
| GA | 18.05 | 20.00 | 20.00 | 19.35 | 3.0s | 23.77 | 32.06 | 39.27 | 31.70 | 5.2s | 26.24 | 36.88 | 46.18 | 36.43 | 7.7s |
| PNRL | 19.97 | 19.99 | 19.98 | 19.98 | 0.007s | 33.74 | 42.34 | 48.67 | 41.58 | 0.011s | 46.87 | 58.38 | 70.64 | 58.63 | 0.012s |
| MTRL | 20.00 | 20.00 | 20.00 | 20.00 | 0.005s | 35.10 | 43.75 | 49.54 | 42.80 | 0.006s | 47.87 | 60.52 | 71.03 | 59.81 | 0.009s |
The results demonstrate that MTRL achieves the highest average cumulative reward across all nine test scenarios. Compared to CPLEX, MTRL achieves improvements of 9.77% to 28.77% in small and medium scales and 9.34% to 14.84% in large scales. The computational efficiency of MTRL is particularly striking, with inference times remaining in the millisecond range across all problem sizes, compared to CPLEX which requires up to 120 seconds for large instances. Traditional heuristic approaches (TS and GA) show competitive performance for small-scale problems but degrade significantly as problem size increases, with performance dropping 28% to 41% below CPLEX in large-scale scenarios.
The training convergence of MTRL demonstrates remarkable stability across all nine scenario configurations. For the T20B2 scenario with 2 drones, convergence requires approximately 75 training epochs, while increasing the number of drones to 4 reduces convergence to just 6 epochs. This acceleration occurs because additional drones provide more exploration diversity and facilitate faster learning of effective coordination strategies. In medium-scale T50B5 scenarios, the final reward values show clear differentiation: approximately 35 for 2 drones, 44 for 3 drones, and 50 for 4 drones. Similarly, large-scale T100B10 scenarios achieve rewards of 46, 60, and 71 for 2, 3, and 4 drones respectively. These results confirm that increasing the number of drone agents consistently improves mission coverage, demonstrating the scalability and effectiveness of multi-agent coordination.
Cross-Distribution Generalization Capability
A critical requirement for practical deployment of drone technology in traffic monitoring is the ability to generalize across different spatial distributions of target points. We conducted comprehensive distribution generalization experiments to evaluate MTRL’s robustness when encountering node distributions different from those used during training. The training phase employed both uniform distribution (within the unit square) and normal distribution N(0.5, 0.35²) to generate node positions. The test phase involved cross-evaluation where models trained on one distribution were tested on instances generated from the other distribution.
Table 2 presents the generalization performance results across all nine scenario configurations. The relative error η measures the percentage difference between cross-distribution testing and same-distribution testing performance.
| Scenario | M_u → T_u | M_u → T_n | η_u (%) | M_n → T_u | M_n → T_n | η_n (%) |
|---|---|---|---|---|---|---|
| T20B2A2 | 19.996 | 19.981 | 0.078 | 19.926 | 19.945 | 0.098 |
| T20B2A3 | 20.000 | 20.000 | 0.000 | 20.000 | 20.000 | 0.000 |
| T20B2A4 | 20.000 | 20.000 | 0.000 | 20.000 | 20.000 | 0.000 |
| T50B5A2 | 35.102 | 34.906 | 0.557 | 34.207 | 34.106 | 0.298 |
| T50B5A3 | 43.750 | 43.664 | 0.196 | 43.363 | 43.184 | 0.416 |
| T50B5A4 | 49.539 | 49.441 | 0.197 | 49.164 | 49.087 | 0.156 |
| T100B10A2 | 47.872 | 47.594 | 0.581 | 47.086 | 46.872 | 0.455 |
| T100B10A3 | 60.518 | 60.258 | 0.431 | 58.473 | 58.376 | 0.167 |
| T100B10A4 | 71.027 | 70.754 | 0.385 | 71.010 | 70.636 | 0.530 |
The results demonstrate exceptional generalization capability, with all relative errors maintained below 1% across all 18 cross-distribution test configurations. For small-scale T20 scenarios, the relative errors range from 0% to 0.098%, indicating perfect or near-perfect generalization. Medium-scale T50 scenarios show errors between 0.156% and 0.557%, while large-scale T100 scenarios exhibit errors between 0.167% and 0.581%. These results confirm that MTRL learns fundamental coordination patterns and constraint handling strategies that are invariant to specific node distributions, rather than memorizing distribution-specific features.
To further validate real-world applicability, we tested MTRL on three realistic distribution patterns: dense urban core distribution (simulating city center target concentration), sparse suburban distribution (simulating highway network monitoring), and differentiated reward distribution (incorporating variable target importance weights). The uniformly-trained MTRL model maintained strong performance across all these scenarios, achieving coverage rates of 91.77% for dense T50B5A3 configurations and 75.59% for dense T100B10A4 configurations. For sparse distributions, coverage rates reached 82.07% and 68.38% respectively. These results demonstrate that MTRL’s learning-based approach captures fundamental principles of multi-agent coordination that transfer effectively across diverse operational environments.
Computational Complexity and Real-Time Performance
The computational efficiency of MTRL stems from its Transformer-based architecture and the offline training paradigm. During the inference phase, the encoder processes all node features in parallel through multi-head self-attention mechanisms, achieving O(p²) complexity where p represents the total number of nodes. The decoder generates path sequences auto-regressively, with each step requiring O(pq) complexity where q represents the sequence length. The parallel computing capability of GPU acceleration enables millisecond-level inference even for large-scale instances with 100 target points and 4 drones.
Table 3 provides detailed computational time comparisons across different methods and problem scales. The results highlight the dramatic efficiency advantage of MTRL over traditional exact and heuristic approaches.
| Method | T20B2A2 | T50B5A2 | T100B10A2 | T20B2A4 | T50B5A4 | T100B10A4 |
|---|---|---|---|---|---|---|
| CPLEX | 60.37 | 107.12 | 118.76 | 117.49 | 114.61 | 118.27 |
| TS | 20.27 | 27.95 | 39.26 | 18.47 | 35.21 | 51.80 |
| GA | 2.77 | 4.03 | 5.77 | 3.27 | 6.44 | 9.66 |
| PNRL | 0.007 | 0.012 | 0.010 | 0.009 | 0.010 | 0.012 |
| MTRL | 0.003 | 0.007 | 0.008 | 0.004 | 0.006 | 0.010 |
The computational advantage of MTRL becomes increasingly pronounced as problem scale grows. For small-scale T20B2A2 scenarios, CPLEX requires 60.37 seconds while MTRL completes inference in just 0.003 seconds, representing a speedup factor of over 20,000. For large-scale T100B10A4 scenarios, CPLEX requires 118.27 seconds while MTRL completes in 0.010 seconds, maintaining a speedup factor exceeding 10,000. This consistent efficiency across scales makes MTRL particularly suitable for real-time traffic monitoring applications where rapid response to changing conditions is essential.
Analysis of Path Quality and Coordination Strategies
The quality of MTRL-generated paths can be evaluated through multiple dimensions including feasibility, coverage efficiency, and coordination effectiveness. Our analysis of the generated path visualizations reveals several important characteristics that demonstrate the effectiveness of the learned coordination strategies.
First, conflict-free coordination is consistently achieved across all scenarios. No instances of duplicate target visitation by different drones were observed, confirming that the masking mechanism effectively prevents multi-agent conflicts. The centralized decision-making framework enables each drone to maintain awareness of other agents’ visitation status, allowing implicit coordination without explicit communication.
Second, battery replacement strategies are intelligently optimized. Drones autonomously determine when to visit battery replacement stations based on their current energy levels and the spatial distribution of remaining targets. In medium and large-scale scenarios, drones frequently utilize battery replacement stations multiple times during their missions, significantly extending their operational range. The uniform spatial distribution of battery replacement stations ensures that charging opportunities are always accessible when needed.
Third, the generated paths satisfy all operational constraints including battery capacity limits, mission time windows, and no sub-tour formation. The comprehensive masking mechanism enforces feasibility at each decision step, ensuring that every generated solution is valid and implementable in practice. This reliability is crucial for real-world deployment of drone technology in traffic monitoring applications where solution validity is non-negotiable.
Fourth, coverage efficiency improves significantly with increasing drone count. Compared to two-drone configurations, three-drone and four-drone setups achieve substantially higher target visitation rates in medium and large-scale scenarios. This improvement stems from effective spatial division of labor and coordinated battery replacement scheduling that maximizes the collective operational envelope.
Comparison with Alternative Learning Approaches
To further validate the architectural choices in MTRL, we compared its performance against PNRL, which employs a recurrent neural network encoder. The key distinction lies in the attention mechanism: while PNRL processes node information sequentially through RNN layers, suffering from long-range dependency degradation, MTRL’s Transformer encoder processes all nodes in parallel, capturing global dependencies in a single forward pass.
Table 4 presents the detailed comparison between MTRL and PNRL across all nine scenarios, highlighting the advantage of Transformer-based architecture for multi-agent coordination problems.
| Scenario | MTRL | PNRL | Improvement (%) | ||||
|---|---|---|---|---|---|---|---|
| Reward | Time (ms) | Std Dev | Reward | Time (ms) | Std Dev | ||
| T20B2A2 | 20.00 | 3.4 | 0.00 | 19.97 | 6.8 | 0.18 | 0.15 |
| T20B2A3 | 20.00 | 6.1 | 0.00 | 19.99 | 7.3 | 0.09 | 0.05 |
| T20B2A4 | 20.00 | 4.0 | 0.00 | 19.98 | 8.7 | 0.14 | 0.10 |
| T50B5A2 | 35.10 | 6.7 | 1.24 | 33.74 | 12.3 | 2.31 | 4.03 |
| T50B5A3 | 43.75 | 5.7 | 1.56 | 42.34 | 12.3 | 2.67 | 3.33 |
| T50B5A4 | 49.54 | 6.1 | 1.89 | 48.67 | 9.5 | 2.89 | 1.79 |
| T100B10A2 | 47.87 | 8.2 | 2.45 | 46.87 | 9.6 | 3.42 | 2.13 |
| T100B10A3 | 60.52 | 9.3 | 2.87 | 58.38 | 11.6 | 3.89 | 3.67 |
| T100B10A4 | 71.03 | 9.8 | 3.12 | 70.64 | 12.4 | 4.21 | 0.55 |
The results demonstrate that MTRL consistently outperforms PNRL in both solution quality and computational efficiency. The average reward improvement ranges from 0.05% in small-scale scenarios to 4.03% in medium-scale scenarios. More importantly, MTRL exhibits lower standard deviations in solution quality, indicating more reliable and stable performance across different problem instances. The computational advantage is also significant, with MTRL achieving inference times approximately 30% to 50% faster than PNRL, particularly in scenarios with larger numbers of nodes and agents.
The superior performance of MTRL can be attributed to the Transformer architecture’s ability to model long-range dependencies and capture complex interactions between all nodes simultaneously. In contrast, RNN-based encoders process information sequentially, which can lead to information loss for distant nodes and reduced sensitivity to global structural patterns. The multi-head attention mechanism in MTRL enables the model to attend to different aspects of the problem simultaneously, learning richer representations that benefit multi-agent coordination.
Real-World Case Study Validation
To validate the practical applicability of MTRL, we conducted case studies using real urban road network data. The target points were extracted from critical intersections within a city’s traffic monitoring zone, and battery replacement stations were positioned using the KMC clustering method based on actual target point distribution. The uniformly-trained MTRL model was applied directly to these real-world instances without any fine-tuning or domain adaptation.
The case study results demonstrate that MTRL successfully generates high-quality feasible solutions for real-world traffic monitoring scenarios. For a configuration with 50 target points, 5 battery replacement stations, and 2 drones, the model achieved a reward of 36, representing 72% coverage of the monitoring targets. With 3 drones, coverage improved to 45 targets (90%), and with 4 drones, 50 targets (100%) were successfully monitored. These results confirm that MTRL’s learning-based approach transfers effectively from synthetic training environments to real-world applications, maintaining its performance advantages without requiring additional training or parameter tuning.
The successful deployment of MTRL for real-world traffic monitoring demonstrates the practical value of our approach. The ability to generate high-quality path plans in milliseconds enables responsive re-planning capabilities that are essential for dynamic traffic environments. When traffic conditions change unexpectedly due to accidents, special events, or weather conditions, the traffic management center can rapidly generate updated drone routing plans to adapt monitoring priorities accordingly. This flexibility represents a significant advantage over traditional pre-planned monitoring approaches.
Convergence Analysis and Training Dynamics
The training dynamics of MTRL reveal important insights into the learning process for multi-agent coordination problems. Figure 6 in our original study illustrates the training reward curves for all nine scenario configurations, demonstrating consistent convergence behavior across different scales and agent counts.
For small-scale T20B2 scenarios, the model achieves convergence within 6 to 75 epochs depending on the number of drones. The rapid convergence with 4 drones (6 epochs) compared to 2 drones (75 epochs) indicates that increased exploration diversity from multiple agents accelerates learning. In medium-scale T50B5 scenarios, convergence requires approximately 50 to 100 epochs, with final rewards reaching 35, 44, and 50 for 2, 3, and 4 drones respectively. Large-scale T100B10 scenarios require 100 to 150 epochs for full convergence, achieving final rewards of 46, 60, and 71 respectively.
The convergence behavior exhibits several noteworthy characteristics. First, the learning curves are smooth and monotonic, without significant oscillations or divergence, indicating stable and reliable training dynamics. Second, the variance across different training runs is minimal, suggesting that the policy gradient method with baseline network effectively reduces gradient variance and promotes consistent learning. Third, the final converged solutions consistently outperform the baseline network solutions, confirming that the policy network successfully learns improved coordination strategies through the training process.
The training time requirements scale reasonably with problem size. Small-scale scenarios require approximately 1-2 hours of training on a single RTX 3090 GPU, medium-scale scenarios require 3-5 hours, and large-scale scenarios require 8-12 hours. Once trained, the model can be deployed for inference on new instances with millisecond-level latency, making the offline training investment worthwhile for real-time applications.
Impact of Battery Replacement Station Configuration
The configuration of battery replacement stations significantly influences the performance of multi-drone traffic monitoring systems. Our analysis examined the effects of varying the number of stations and their spatial distribution on overall mission coverage and drone utilization efficiency.
Table 5 presents the sensitivity analysis results for different battery replacement station configurations in the T50B5A3 scenario.
| Number of Stations | Coverage (%) | Avg Visits per Station | Avg Battery Utilization (%) | Mission Completion Time (min) |
|---|---|---|---|---|
| 3 | 72.4 | 2.67 | 78.3 | 34.2 |
| 4 | 82.6 | 2.25 | 82.1 | 35.8 |
| 5 | 87.5 | 1.80 | 85.6 | 37.5 |
| 6 | 88.2 | 1.50 | 86.2 | 38.1 |
| 7 | 88.5 | 1.29 | 86.5 | 38.3 |
The results reveal diminishing returns as the number of battery replacement stations increases. Coverage improves significantly from 3 stations (72.4%) to 5 stations (87.5%), but further increases provide marginal benefits (88.2% with 6 stations, 88.5% with 7 stations). The average visits per station decrease proportionally, indicating that drones distribute their charging activities across available stations. Battery utilization increases steadily with more stations, as drones can more easily find convenient charging opportunities without deviating significantly from their monitoring routes.
The spatial distribution of battery replacement stations also plays a crucial role. Uniform distribution achieved through KMC clustering provides superior performance compared to random placement or concentration in specific areas. The KMC-based placement ensures that battery replacement stations are accessible from any point in the monitoring region, reducing the detour distance required for charging and enabling more efficient path planning.
Limitations and Future Research Directions
While MTRL demonstrates impressive performance across diverse scenarios, several limitations warrant consideration and provide directions for future research. First, the current framework assumes unlimited battery supply at each replacement station, which may not reflect real-world constraints where charging capacity and battery availability are limited. Future work should incorporate station capacity constraints and queueing models to capture realistic operational conditions where multiple drones may compete for limited charging resources simultaneously.
Second, the static environment assumption limits the model’s applicability to dynamic traffic conditions. Real-world traffic monitoring scenarios involve changing traffic patterns, weather conditions, and potential airspace restrictions that affect drone operations. Developing online fine-tuning mechanisms that allow MTRL to adapt to dynamic changes without full retraining would significantly enhance practical utility.
Third, the centralized decision-making framework may face scalability challenges when applied to very large fleets exceeding 10-20 drones. The computational complexity of the Transformer architecture grows quadratically with the number of nodes, potentially leading to increased inference times for extremely large problem instances. Exploring hybrid architectures that combine centralized coordination with distributed local decision-making could provide better scalability while maintaining coordination effectiveness.
Fourth, the current reward structure treats all target points with equal importance. In practice, traffic monitoring priorities may vary based on time of day, historical congestion patterns, or special events. Extending MTRL to handle dynamic reward functions that change over time would enable more responsive and adaptive monitoring strategies.
Fifth, the integration of MTRL with real-time traffic management systems requires careful consideration of data latency, communication reliability, and system robustness. Developing fault-tolerant mechanisms that can handle drone failures, communication interruptions, and other operational anomalies is essential for reliable deployment in safety-critical traffic monitoring applications.
Conclusion and Practical Implications
This research presents a comprehensive framework for optimizing multi-drone cooperative traffic monitoring missions using advanced reinforcement learning techniques. The MTRL framework integrates several innovations that collectively enable efficient and effective path planning for drone technology applications in intelligent transportation systems.
The MILP modeling approach provides an exact mathematical formulation of the problem, establishing a theoretical foundation and enabling precise evaluation of solution quality. The KMC-based battery replacement station location method ensures uniform spatial distribution of charging infrastructure, maximizing accessibility and enabling efficient drone operations. The MTRL algorithm framework leverages Transformer architecture’s powerful representation capabilities to learn effective coordination strategies that generalize across different problem scales and spatial distributions.
The experimental results demonstrate that MTRL consistently outperforms traditional exact methods, heuristic approaches, and alternative learning methods across all test scenarios. The solution quality improvements range from 9.34% to 28.77% compared to CPLEX, while inference times remain in the millisecond range regardless of problem scale. The cross-distribution generalization capability, with relative errors below 1% across all test configurations, confirms that MTRL learns fundamental coordination principles rather than distribution-specific features.
The practical implications of this research extend beyond traffic monitoring to encompass broader applications of drone technology in logistics delivery, emergency response, environmental monitoring, and infrastructure inspection. The MTRL framework provides a general-purpose solution methodology for multi-agent routing problems with complex constraints, demonstrating the potential of learning-based approaches to solve traditionally intractable optimization problems efficiently.
For traffic management authorities, the ability to deploy coordinated drone fleets for real-time traffic monitoring offers significant advantages over fixed infrastructure. The flexibility to adapt monitoring coverage based on changing conditions, the cost-effectiveness of mobile aerial platforms compared to extensive ground infrastructure, and the comprehensive aerial perspective that overcomes occlusion limitations all contribute to enhanced traffic surveillance capabilities. The MTRL framework provides the computational foundation needed to realize these benefits in practice, enabling intelligent coordination of drone technology for improved urban mobility management.
As drone technology continues to advance with improved battery capacity, enhanced sensors, and more robust communication systems, the potential for aerial traffic monitoring will expand further. The MTRL framework is designed to scale with these technological advances, accommodating larger fleets, more complex constraints, and more sophisticated mission requirements. The modular architecture allows for straightforward integration of new constraint types, reward structures, and operational rules as they become relevant in practice.
In conclusion, this research advances the state of the art in multi-agent path planning for drone technology applications in traffic monitoring. The MTRL framework provides a practical, efficient, and generalizable solution methodology that can enable widespread deployment of coordinated drone fleets for intelligent transportation systems. The demonstrated performance advantages, computational efficiency, and cross-domain generalization capability position MTRL as a promising foundation for future developments in autonomous aerial traffic monitoring and beyond.