The operational scope of unmanned aerial vehicles (UAVs) has expanded dramatically into sectors such as logistics, infrastructure inspection, and emergency response. Within these applications, the concept of coordinated drone formation flight is paramount, enabling complex tasks through synchronized behavior. A particularly challenging subclass is the passive or uncooperative drone swarm, which operates without emitting active positioning signals like GPS, relying instead on relative observations between members. This characteristic offers advantages in stealth and reduced electromagnetic signature but introduces significant hurdles for navigation, especially in complex urban or adversarial environments plagued by GPS-denial and severe multipath interference.
The core challenge for a passive drone formation is achieving and maintaining accurate relative positioning. Traditional methods, such as inertial navigation systems (INS), suffer from unbounded error drift. While cooperative schemes using Ultra-Wideband (UWB) or vision exist, they often struggle with the dynamic spatial-temporal relationships within a moving formation, the non-linear noise characteristics in harsh environments, and the computational and communication burdens of centralized processing. This research directly addresses these limitations by proposing a novel, integrated framework that synergizes dynamic graph modeling, federated deep reinforcement learning, and differentiable filtering to enable robust, efficient, and privacy-aware collaborative positioning for large-scale passive drone formations.
Our approach is grounded in a formal mathematical description of the cooperative positioning problem. Consider a formation of \(M\) drones. The state of drone \(m\) at time \(k\) is represented as \(\mathbf{s}_m^k = [\Delta x, \Delta y, v_x, v_y, a_x, a_y, \phi, \theta, \psi]^T\), encompassing relative 2D position, velocity, acceleration, and attitude (roll \(\phi\), pitch \(\theta\), yaw \(\psi\)). Each drone has access to multi-modal but noisy observations \(\mathbf{o}_m^k\), which may include Angle-of-Arrival (AOA) measurements from peers, visual odometry or feature points from an onboard camera (visual SLAM), and inertial readings from an IMU: \(\mathbf{o}_m^k = [\boldsymbol{\alpha}_{AOA}, \mathbf{f}_{SLAM}, \boldsymbol{\omega}_{IMU}]\). The primary objective is for each drone to estimate its own state \(\hat{\mathbf{s}}_m^k\) within the formation’s frame of reference.
The action space for control involves trajectory adjustments \(\mathbf{a}_{traj} = [d, \theta]\), where \(d\) is a movement distance and \(\theta\) is a heading change, subject to kinematic constraints \(d \leq d_{max}\). A critical component of our method is the dynamic allocation of trust to different sensor modalities, formalized as an action \(\mathbf{a}_{res} = [\beta_1, \beta_2, …]\), representing adaptive weighting coefficients for a multi-sensor fusion filter. The system’s performance is guided by a compound reward function \(R^k\):
$$
R^k = \lambda_1 \cdot (-\text{RMSE}^k) + \lambda_2 \cdot (-\|\mathbf{a}_{traj}^k\|) + \lambda_3 \cdot \mathbb{I}(R_{m,m’} < R_{min}) + \lambda_4 \cdot \mathbb{I}(|z_m – \bar{z}| > \Delta z_{max})
$$
This function penalizes positioning Root Mean Square Error (RMSE), excessive control effort (energy consumption), collisions (when distance \(R_{m,m’}\) falls below a minimum safe threshold \(R_{min}\)), and deviations from the formation’s intended altitude plane \(\bar{z}\). The coefficients \(\lambda_1\) to \(\lambda_4\) balance these objectives.
Core Algorithmic Framework for Dynamic Collaborative Positioning
The proposed solution is built upon three interconnected algorithmic pillars, each designed to tackle a specific aspect of the drone formation positioning problem. Their roles are summarized below:
| Algorithm Module | Primary Function |
|---|---|
| Dynamic Heterogeneous Graph Convolutional Network (DHGCN) | Spatio-temporal topology modeling of the drone formation. |
| Hierarchical Federated Deep Deterministic Policy Gradient (HF-DDPG) | Privacy-preserving, distributed decision-making and trajectory optimization. |
| Differentiable Cubature Kalman Filter (DCKF) | End-to-end trainable non-linear state estimation and sensor fusion. |
1. Dynamic Heterogeneous Graph Convolutional Network (DHGCN)
Traditional graph networks with static adjacency matrices fail to capture the evolving interactions in a mobile drone formation. Our DHGCN models the formation as a dynamic graph \(\mathcal{G}^k = (\mathcal{V}, \mathcal{E}^k, \mathbf{W}^k)\), where nodes \(\mathcal{V}\) represent drones and edges \(\mathcal{E}^k\) represent communication or sensing links at time \(k\). The key innovation is the learnable, time-varying adjacency weight matrix \(\mathbf{W}^k\), where each element \(w_{m,m’}^k\) (the weight between drone \(m\) and \(m’\)) is computed dynamically based on real-time features:
$$
w_{m,m’}^k = \sigma \left( \mathbf{W}_1 \cdot [\mathbf{v}_{m,m’}^k; \alpha_{UWB}^k] + b_1 \right)
$$
Here, \(\mathbf{v}_{m,m’}^k\) is the relative velocity vector, \(\alpha_{UWB}^k\) is the received signal strength indicator (RSSI) of UWB signals (a proxy for link quality and proximity), \(\mathbf{W}_1\) and \(b_1\) are learnable parameters, and \(\sigma\) is the sigmoid activation function. This allows the network to strengthen edges between drones moving coherently and with strong signals, while weakening spurious or noisy links. The DHGCN processes node features (e.g., state estimates, sensor embeddings) through these adaptive graph convolution layers to output a cohesive graph-level embedding \(\mathbf{h}_g^k\) that encapsulates the formation’s spatial configuration and temporal trend.
2. Hierarchical Federated Deep Deterministic Policy Gradient (HF-DDPG)
Centralized reinforcement learning for a drone formation raises privacy concerns and creates a single point of failure. Fully independent learning leads to sub-optimal, uncoordinated policies. Our HF-DDPG framework strikes a balance. A high-level meta-policy, trained via model-agnostic meta-learning (MAML), provides a robust initial policy \(\pi_{meta}(\theta_{init} | \mathbf{h}_g)\) parameterized by \(\theta_{init}\) conditioned on the formation context \(\mathbf{h}_g\). This meta-policy is distributed to all drones at the start of a mission, ensuring a baseline level of coordinated behavior.
During operation, each drone \(m\) runs a local DDPG agent. The actor network \(\mu_m(\mathbf{s}_m | \theta_m^\mu)\) outputs continuous control actions \(\mathbf{a}_m\), while a dueling critic network \(Q_m(\mathbf{s}_m, \mathbf{a}_m | \theta_m^Q)\) estimates the action-value function. To mitigate overestimation bias, we employ a Twin Delayed DDPG (TD3) architecture with two critic networks and target policy smoothing. The core federated learning step occurs periodically every \(T_{FL}\) global rounds. Drones upload their locally updated actor parameters \(\theta_m^\mu\) to a secure aggregator, which computes a new global model:
$$
\theta_{global}^\mu = \frac{1}{M} \sum_{m=1}^{M} \theta_m^\mu
$$
This aggregated global model is then broadcast back to the swarm, synchronizing and improving the collective intelligence without sharing raw trajectory or observation data. This process significantly reduces communication overhead and preserves operational privacy within the drone formation.
3. Differentiable Cubature Kalman Filter (DCKF)
Sensor fusion in non-linear systems is typically handled by algorithms like the Extended Kalman Filter (EKF) or Cubature Kalman Filter (CKF), which require pre-defined, often inaccurate, noise covariance matrices (\(\mathbf{Q}\) for process noise, \(\mathbf{R}\) for measurement noise). We integrate a CKF directly into the training pipeline by making it fully differentiable (DCKF). This allows the network to learn optimal noise models from data, adapting to complex interference like urban multipath.
The DCKF operates within each drone. It takes the prior state estimate \(\hat{\mathbf{s}}_m^{k-1}\) and covariance \(\mathbf{P}_m^{k-1}\) and the current multi-modal observations \(\mathbf{o}_m^k\). Instead of fixed matrices, it uses small neural network heads to generate context-aware noise covariances:
$$
\mathbf{Q}_m^k = \text{Softplus}(\mathbf{W}_Q \cdot \mathbf{h}_g^k + \mathbf{b}_Q) + \mathbf{Q}_{base}
$$
$$
\mathbf{R}_m^k = \text{Softplus}(\mathbf{W}_R \cdot [\mathbf{h}_g^k; \mathbf{o}_m^k] + \mathbf{b}_R) + \mathbf{R}_{base}
$$
The filter then executes the standard CKF steps—cubature point generation, propagation, and update—but since all operations (including Cholesky decomposition for point generation) are implemented with auto-differentiation, gradients can flow backward through the filter. The loss function for training the DCKF module combines state estimation error and regularization:
$$
\mathcal{L}_{DCKF} = \|\mathbf{s}_{gt}^k – \hat{\mathbf{s}}_m^{k}\|_2^2 + \alpha \cdot \text{tr}(\mathbf{P}_m^{k}) + \gamma \cdot \|\mathbf{Q}_m^k – \mathbf{Q}_{prior}\|_F^2
$$
Minimizing this loss teaches the network to produce accurate state estimates (\(\hat{\mathbf{s}}_m^{k}\)) with appropriate uncertainty (\(\mathbf{P}_m^{k}\)), while keeping the learned noise covariances (\(\mathbf{Q}_m^k\)) near a sensible prior.
Integrated System Architecture and Training
The three modules form a coherent “perception-decision-calibration” loop. The DHGCN consumes the raw observations and states to understand the formation’s topology. Its output, \(\mathbf{h}_g^k\), informs both the HF-DDPG actor for making navigation decisions and the DCKF for adjusting its fusion strategy. The DCKF produces a refined, low-noise state estimate \(\hat{\mathbf{s}}_m^{k}\), which is then fed back as input to the DHGCN for the next time step, closing the loop. This end-to-end differentiability allows the final positioning error signal to propagate backwards and optimize all components jointly, ensuring the feature representations learned by the DHGCN are those most useful for accurate state estimation by the DCKF.
A progressive three-stage training strategy was employed:
- DCKF Pre-training: The DCKF module is first trained independently on simulated sensor data to learn basic state estimation.
- Alternating DHGCN-HF-DDPG Training: With DCKF fixed, the DHGCN and HF-DDPG modules are trained together in a simulated environment, learning to navigate and model the formation.
- Federated Fine-tuning: The HF-DDPG policies are refined using the federated learning scheme across multiple independent simulation runs, enhancing generalization and robustness.
The simulation environment was built on ROS and Gazebo, incorporating high-fidelity sensor models (UWB with ray-traced multipath, IMU with bias drift) and communication models with packet loss. A conical drone formation of 30 UAVs was used as the primary test case. Key simulation and training parameters are listed below:
| Parameter | Symbol | Value |
|---|---|---|
| Formation Size | \(M\) | 30 drones |
| Max Speed | \(v_{max}\) | 5 m/s |
| Min Safe Distance | \(R_{min}\) | 10 m |
| Federated Aggregation Period | \(T_{FL}\) | 10 rounds |
| Replay Buffer Size | – | 50,000 samples |
| RL Learning Rate | \(\eta_{RL}\) | 1e-4 |
| Discount Factor | \(\gamma\) | 0.99 |
Experimental Validation and Performance Analysis
The proposed framework was rigorously evaluated against two established baseline methods: a traditional geometry-based iterative method and a Distributed Extended Kalman Filter (D-EKF) with fixed sensor weights. The mission involved constructing and maintaining a 30-drone conical formation under GPS-denied conditions with simulated multipath interference and packet loss. Four key metrics were measured, with results demonstrating clear superiority of our approach:
| Performance Metric | Geometry-Based Method | Distributed EKF (D-EKF) | Our STGRL Method |
|---|---|---|---|
| Formation Build Time (s) | 14.20 | 9.80 | 5.70 |
| Average Positioning RMSE (m) | 0.51 | 0.33 | 0.18 |
| Comm. Overhead per Node (MB) | 12.40 | 18.60 | 4.30 |
| RMSE under Strong Interference (m) | 0.89 | 0.47 | 0.23 |
The analysis reveals compelling advantages. The 45.5% improvement in positioning accuracy (0.18m vs. 0.33m for D-EKF) stems from the DCKF’s ability to learn and suppress context-specific noise like multipath, and the DHGCN’s provision of rich contextual features. The reduction in formation build time by 41.8% (5.7s vs. 9.8s for D-EKF) highlights the efficiency gains from coordinated reinforcement learning policies that understand the spatial-temporal constraints of the drone formation. Most notably, the federated learning mechanism achieved a 76.9% reduction in per-node communication overhead (4.3MB vs. 18.6MB for D-EKF) by transmitting only model parameters instead of raw sensor data. This is critical for scaling to large swarms and for security-sensitive applications. Finally, the superior performance under strong interference (0.23m RMSE) underscores the robustness granted by the multi-modal, adaptive fusion strategy.
Conclusion
This research presents a comprehensive deep reinforcement learning framework for solving the cooperative positioning problem in passive drone formations operating in denied and dynamic environments. By innovatively integrating Dynamic Heterogeneous Graph Convolutional Networks, Hierarchical Federated Deep Deterministic Policy Gradient, and a Differentiable Cubature Kalman Filter, we have developed a system that excels not only in positioning accuracy and formation agility but also in communication efficiency and privacy preservation. The framework’s ability to dynamically model formation topology, learn optimal control policies in a distributed yet coordinated fashion, and adaptively calibrate sensor fusion parameters in an end-to-end manner represents a significant advancement over traditional filtering and geometric methods. The results demonstrate clear pathways towards enabling intelligent, secure, and efficient large-scale autonomous drone formation operations for the future low-altitude economy.