We investigate the problem of jointly optimizing three-dimensional (3D) flight trajectories, user scheduling, and power allocation for a fleet of unmanned aerial vehicles (UAVs) that serve as mobile integrated sensing and communication (ISAC) platforms. The rapid evolution of drone technology has enabled unprecedented flexibility in deploying aerial base stations for emergency communication, disaster response, and remote sensing. However, the coupling between sensing and communication functions, together with the need for real-time coordination among multiple drones, poses a challenging non-convex optimization task. In this work, we propose a multi-agent deep reinforcement learning framework named IA-QMIX (ISAC-Aware QMIX) that embeds the physical constraints of sensing–communication trade-offs directly into the value decomposition architecture. Our method respects the individual-global-max (IGM) principle while leveraging a hierarchical state encoder, a constraint-guided feasibility network, and an adaptive hypernetwork-based mixer. Extensive simulations demonstrate that our approach significantly outperforms existing baselines such as QMIX, MAPPO, IQL, greedy search, and SCA-based convex optimization. The results confirm that advanced drone technology, when combined with intelligent multi-agent coordination, can dramatically enhance both communication throughput and sensing fairness.
1. Introduction
The integration of sensing and communication on a single radio platform is a key enabler for future wireless systems. Drone technology offers a natural flying infrastructure for ISAC because of its high mobility, line-of-sight propagation, and ability to reach remote areas. In a multi-UAV ISAC system, each drone simultaneously performs radar sensing to detect ground users and communicates data to those users. The limited onboard energy and the need to avoid mutual interference require careful joint optimization of trajectories, resource allocation, and scheduling.
Traditional optimization methods, such as alternating optimization and convex approximation, struggle with the high-dimensional mixed-integer non-convex nature of the problem. Moreover, they cannot adapt to dynamic environments in real time. Multi-agent reinforcement learning (MARL) has emerged as a promising alternative, enabling drones to learn cooperative policies through centralized training and decentralized execution. However, existing MARL algorithms like QMIX typically ignore the specific structure of ISAC systems, including the mutual information (MI) constraint for sensing and the collision avoidance among drones. To bridge this gap, we present the IA-QMIX algorithm, which explicitly incorporates ISAC constraints into the value function factorization.
2. System Model and Problem Formulation
We consider a set of \(M\) UAVs (denoted by \(\mathcal{M} = \{1,\dots,M\}\)) and a set of \(K\) ground users \(\mathcal{K} = \{1,\dots,K\})\). The flight horizon \(\Gamma\) is divided into \(T\) equal time slots of duration \(\tau\). Each UAV is equipped with an ISAC transceiver operating in a time-division multiplexing (TDM) mode: a fraction \(\beta\) of each slot is used for sensing, and the remaining \(1-\beta\) for communication, as depicted in the figure below.
Let \(\mathbf{U}_m^{(t)} = [x_m^{(t)}, y_m^{(t)}, z_m^{(t)}]^\top\) denote the 3D position of UAV \(m\) at slot \(t\), and let \(\mathbf{G}_k = [x_k, y_k, 0]^\top\) be the fixed location of ground user \(k\). The speed constraint is \(\|\mathbf{U}_m^{(t)} – \mathbf{U}_m^{(t-1)}\| = V\tau\), where \(V\) is the constant speed. Collision avoidance between any two drones requires \(\|\mathbf{U}_m^{(t)} – \mathbf{U}_{m’}^{(t)}\| \ge d_{\min}\). The flight altitude is bounded by \(H_{\min} \le z_m^{(t)} \le H_{\max}\).
The channel gain for the communication link between UAV \(m\) and user \(k\) follows the free-space path loss model:
$$ g_{m,k}^{\text{com},(t)} = \frac{\rho_{\text{com}}}{(l_{m,k}^{(t)})^2}, $$
where \(l_{m,k}^{(t)} = \|\mathbf{U}_m^{(t)} – \mathbf{G}_k\|\) and \(\rho_{\text{com}}\) is the reference gain at 1 m. Similarly, the radar sensing gain is:
$$ g_{m,k}^{\text{rad},(t)} = \frac{\rho_{\text{rad}}}{(l_{m,k}^{(t)})^4}. $$
A binary variable \(\alpha_{m,k}^{(t)} \in \{0,1\}\) indicates whether UAV \(m\) serves user \(k\) at slot \(t\). Each UAV serves at most one user per slot, and each user is served by at most one UAV:
$$ \sum_{k=1}^K \alpha_{m,k}^{(t)} \le 1,\quad \forall m,t, \qquad \sum_{m=1}^M \alpha_{m,k}^{(t)} \le 1,\quad \forall k,t. $$
The transmit power of UAV \(m\) at slot \(t\) is \(p_m^{(t)}\), with \(0 \le p_m^{(t)} \le P_{\max}\). The signal-to-interference-plus-noise ratio (SINR) for communication and radar are:
$$ \text{SINR}_{m,k}^{\text{com},(t)} = \frac{p_m^{(t)} g_{m,k}^{\text{com},(t)}}{\sum_{j \neq m} p_j^{(t)} g_{j,k}^{\text{com},(t)} + N_0}, $$
$$ \text{SINR}_{m,k}^{\text{rad},(t)} = \frac{p_m^{(t)} g_{m,k}^{\text{rad},(t)}}{\sum_{j \neq m} p_j^{(t)} g_{j,k}^{\text{rad},(t)} + N_0}. $$
The communication rate from UAV \(m\) to user \(k\) is \(R_{m,k}^{\text{com},(t)} = B \log_2(1 + \text{SINR}_{m,k}^{\text{com},(t)})\), and the radar mutual information is \(R_{m,k}^{\text{rad},(t)} = B \log_2(1 + \text{SINR}_{m,k}^{\text{rad},(t)})\). Over one flight cycle, the average communication rate for user \(k\) is:
$$ R_k^{\text{com}} = \frac{1}{T} \sum_{t=1}^T \sum_{m=1}^M \alpha_{m,k}^{(t)} (1-\beta) R_{m,k}^{\text{com},(t)}, $$
and the average radar MI is:
$$ R_k^{\text{rad}} = \frac{1}{T} \sum_{t=1}^T \sum_{m=1}^M \alpha_{m,k}^{(t)} \beta R_{m,k}^{\text{rad},(t)}. $$
Our objective is to maximize the maximum average communication rate among all users while satisfying a minimum sensing MI threshold \(\text{MI}_{\min}\) for every user. The joint optimization problem is:
$$ \begin{aligned}
\mathcal{P}: & \max_{\{\alpha,p,\mathbf{U}\}} \quad \max_k R_k^{\text{com}} \\
\text{s.t.} & \quad R_k^{\text{rad}} \ge \text{MI}_{\min}, \forall k, \\
& \quad \text{constraints on collision, speed, altitude, power, and scheduling.}
\end{aligned} $$
This is a mixed-integer non-convex problem that is notoriously difficult to solve with conventional methods. The rapid advancement of drone technology demands real-time adaptive solutions, which motivates our MARL approach.
3. IA-QMIX: ISAC-Aware Multi-Agent Reinforcement Learning
3.1 Overview of the Centralized Training Decentralized Execution Paradigm
We adopt the CTDE framework: during training, a central critic has access to the global state \(\mathbf{s}^{(t)}\) and all actions; during execution, each UAV acts based only on its local observation \(\mathbf{o}_m^{(t)}\). Our IA-QMIX algorithm builds upon the value decomposition philosophy of QMIX but introduces three novel components to handle the ISAC-specific constraints: (i) a hierarchical state encoder, (ii) a constraint feasibility guidance network, and (iii) an adaptive mixer with hypernetworks.
3.2 Hierarchical State Encoder
Instead of flattening the global state, we extract three structured representations:
- UAV-level encoder: processes the concatenated states of all drones \(\mathbf{S}^{\text{uav},(t)} = [\mathbf{s}_1^{(t)},\dots,\mathbf{s}_M^{(t)}]\) through an MLP to produce \(\mathbf{h}^{\text{uav},(t)}\).
- GU-level encoder: processes the fixed user positions \(\mathbf{S}^{\text{gu}} = [\mathbf{G}_1,\dots,\mathbf{G}_K]\) through another MLP to obtain \(\mathbf{h}^{\text{gu},(t)}\).
- Interaction layer: uses a bipartite attention mechanism to compute cross-attention weights between each UAV and all users:
$$ e_{m,k}^{(t)} = \frac{(\mathbf{h}_m^{\text{uav},(t)} W_Q) (\mathbf{h}_k^{\text{gu},(t)} W_K)^\top}{l_{m,k}^{(t)}}, \quad \alpha_{m,k}^{(t)} = \text{softmax}(e_{m,k}^{(t)}). $$
The interaction feature for UAV \(m\) is:
$$ \mathbf{h}_{m}^{\text{assoc},(t)} = \sum_{k=1}^K \alpha_{m,k}^{(t)} \mathbf{h}_k^{\text{gu},(t)} W_V, $$
which yields the overall interaction representation \(\mathbf{h}^{\text{assoc},(t)} = [\mathbf{h}_{1}^{\text{assoc},(t)},\dots,\mathbf{h}_{M}^{\text{assoc},(t)}]\). The final global encoding is \(\mathbf{h}^{(t)} = [\mathbf{h}^{\text{uav},(t)}, \mathbf{h}^{\text{gu},(t)}, \mathbf{h}^{\text{assoc},(t)}]\). This hierarchical encoding allows the agent to perceive the structure of the UAV–user graph, which is crucial for learning cooperative scheduling.
3.3 Constraint Feasibility Guidance
We compute two feasibility scalars for each UAV from \(\mathbf{h}^{(t)}\) using two small networks:
$$ c_{m,t}^{\text{coll}} = \sigma(\mathbf{W}_{\text{coll}} \cdot [\mathbf{h}^{(t)}, \mathbf{h}_m^{\text{uav},(t)}] + b_{\text{coll}}), $$
$$ c_{m,t}^{\text{sense}} = \sigma(\mathbf{W}_{\text{sense}} \cdot [\mathbf{h}^{(t)}, \mathbf{h}_m^{\text{assoc},(t)}] + b_{\text{sense}}). $$
These values lie in \([0,1]\) and indicate collision safety and sensing fairness contribution, respectively. The adjusted local Q-value for UAV \(m\) becomes:
$$ Q_{m,\text{adj}}^{(t)} = Q_m^{(t)} \cdot c_{m,t}^{\text{coll}} \cdot c_{m,t}^{\text{sense}}. $$
This multiplication effectively suppresses actions that would risk collision or neglect users with unsatisfied sensing requirements, guiding the policy toward constraint-satisfying solutions without modifying the reward function.
3.4 Adaptive Mixing Network
To satisfy the IGM principle (i.e., the joint action that maximizes \(\sum_m Q_{m,\text{adj}}\) is also the global optimal), we design a hypernetwork that generates the weights of a monotonic mixer. The hypernetwork takes \(\mathbf{h}^{(t)}\) as input and outputs the weight matrices \(\mathbf{W}_1(\mathbf{h}^{(t)})\) and \(\mathbf{W}_2(\mathbf{h}^{(t)})\), whose elements are taken absolute values to ensure monotonicity. The final joint value is:
$$ Q_{\text{tot}}^{\text{IA}} = \mathbf{W}_2(\mathbf{h}) \cdot \text{ReLU}\left( \mathbf{W}_1(\mathbf{h}) \begin{bmatrix} Q_{1,\text{adj}} \\ \vdots \\ Q_{M,\text{adj}} \end{bmatrix} + \mathbf{b}_1(\mathbf{h}) \right) + \mathbf{b}_2(\mathbf{h}). $$
This architecture allows the mixing weights to adapt to the current traffic distribution and interference pattern, which is a key advantage over the fixed-weight mixer in standard QMIX.
3.5 Training and Execution
We use a deep recurrent Q-network (DRQN) for each UAV to handle partial observability. The loss function is the temporal-difference error:
$$ \mathcal{L}(\theta) = \mathbb{E}\left[ \left( y^{(t)} – Q_{\text{tot}}^{\text{IA}}(\mathbf{s}^{(t)},\mathbf{a}^{(t)};\theta) \right)^2 \right], $$
where the target \(y^{(t)} = r^{(t)} + \gamma \max_{\mathbf{a}} Q_{\text{tot}}^{\text{IA}}(\mathbf{s}^{(t+1)},\mathbf{a};\theta^-)\). During execution, each UAV chooses action greedily: \(a_m^{(t)} = \arg\max_a Q_m^{(t)}(\mathbf{o}_m^{(t)}, a)\). The final algorithm is summarized in Table 1.
| Step | Description |
|---|---|
| 1 | Initialize networks \(\theta\) (online) and \(\theta^- = \theta\) (target), replay buffer \(\mathcal{D}\). |
| 2 | For each episode, reset environment, obtain initial state \(\mathbf{s}^{(0)}\) and observations \(\mathbf{o}_m^{(0)}\). |
| 3 | For \(t=1\) to \(T\):
|
| 4 | Return trained local DRQNs for decentralized execution. |
4. Simulation Results and Analysis
4.1 Simulation Setup
We simulate an area of 2 km × 2 km with \(M=4\) UAVs and \(K=8\) ground users unless specified otherwise. Key parameters are listed in Table 2. The flight horizon is 900 s divided into 90 slots of 10 s each. The TDM parameter \(\beta=0.35\) (favoring communication). The sensing MI threshold \(\text{MI}_{\min}=0.05\) bps/Hz. The drone speed \(V=20\) m/s, altitude range [100,150] m, and collision distance \(d_{\min}=15\) m. The reference gains are \(\rho_{\text{com}}=-111\) dB and \(\rho_{\text{rad}}=-131\) dB at 1 m, noise power \(N_0=-169\) dBm, max power \(P_{\max}=0.5\) W, bandwidth \(B=1\) MHz.
| Parameter | Value |
|---|---|
| Slot duration \(\tau\) | 10 s |
| Number of slots \(T\) | 90 |
| Drone speed \(V\) | 20 m/s |
| Collision distance \(d_{\min}\) | 15 m |
| Altitude range \([H_{\min},H_{\max}]\) | [100,150] m |
| Max power \(P_{\max}\) | 0.5 W |
| Noise power \(N_0\) | -169 dBm |
| Reference gain \(\rho_{\text{com}}\) | -111 dB |
| Reference gain \(\rho_{\text{rad}}\) | -131 dB |
| Bandwidth \(B\) | 1 MHz |
| Sensing MI threshold \(\text{MI}_{\min}\) | 0.05 bps/Hz |
| TDM parameter \(\beta\) | 0.35 |
| Number of UAVs \(M\) | 4 |
| Number of users \(K\) | 8 |
4.2 Convergence Behavior
Figure 1 shows the average communication rate over the training episodes for IA-QMIX and three baseline MARL algorithms: QMIX, MAPPO, and IQL. Our proposed IA-QMIX converges to the highest rate of approximately 0.8163 bps, outperforming QMIX (0.724 bps) by about 12.7%. MAPPO reaches 0.6032 bps and IQL only 0.4513 bps. The superior performance of IA-QMIX is attributed to the hierarchical encoding that captures the drone–user interplay and the constraint guidance that prevents collisions and ensures sensing fairness.
4.3 Scalability with Number of Users
We fix \(M=4\) and vary \(K\) from 4 to 64. Table 3 lists the achieved maximum average communication rate. IA-QMIX maintains the best performance across all user counts. The rate initially dips when moving from \(K=4\) to \(K=8\) because the system transitions from a quasi-static allocation to a more dynamic one. Thereafter, the rate increases due to multi-user diversity. The drone technology allows UAVs to exploit favorable channels among many users, but the coordination complexity also grows. IA-QMIX’s attention mechanism effectively handles the increased combinatorial space.
| \(K\) | IA-QMIX | QMIX | MAPPO | IQL |
|---|---|---|---|---|
| 4 | 0.712 | 0.650 | 0.534 | 0.402 |
| 8 | 0.816 | 0.724 | 0.603 | 0.451 |
| 16 | 0.921 | 0.805 | 0.672 | 0.510 |
| 32 | 1.045 | 0.911 | 0.757 | 0.581 |
| 64 | 1.163 | 1.002 | 0.834 | 0.639 |
4.4 Scalability with Number of Drones
We fix \(K=8\) and vary \(M\) from 1 to 16. As shown in Table 4, the average communication rate increases with more drones due to improved coverage and spatial reuse. IA-QMIX consistently outperforms other methods. The gap widens at larger \(M\), demonstrating that IA-QMIX effectively coordinates more agents without suffering from the curse of dimensionality. This is because the value factorization and constraint guidance prevent conflicting actions among drones.
| \(M\) | IA-QMIX | QMIX | MAPPO | IQL |
|---|---|---|---|---|
| 1 | 0.312 | 0.298 | 0.245 | 0.190 |
| 2 | 0.501 | 0.464 | 0.387 | 0.301 |
| 4 | 0.816 | 0.724 | 0.603 | 0.451 |
| 8 | 1.120 | 0.973 | 0.810 | 0.602 |
| 16 | 1.452 | 1.201 | 1.024 | 0.755 |
4.5 Impact of Maximum Transmit Power
We compare IA-QMIX with a greedy baseline and a traditional SCA-based convex optimization method (Table 5). As the maximum power \(P_{\max}\) increases from 0.1 W to 1 W, the communication rate initially rises but then saturates due to co-channel interference. IA-QMIX achieves the highest rate across the entire range, confirming that our learning-based approach can intelligently manage interference while satisfying the sensing MI constraint.
| \(P_{\max}\) (W) | IA-QMIX | Greedy | SCA-Convex |
|---|---|---|---|
| 0.1 | 0.412 | 0.338 | 0.371 |
| 0.3 | 0.651 | 0.530 | 0.582 |
| 0.5 | 0.816 | 0.667 | 0.714 |
| 0.7 | 0.852 | 0.701 | 0.743 |
| 1.0 | 0.861 | 0.712 | 0.751 |
4.6 Sensitivity to Time-Division Parameter \(\beta\)
Table 6 shows the average communication rate for different values of \(\beta\). The optimal trade-off occurs at \(\beta=0.35\), yielding the highest rate. Too small \(\beta\) (e.g., 0.2) reduces sensing quality, leading to poor decision making; too large \(\beta\) (e.g., 0.8) leaves insufficient time for communication. This analysis illustrates the importance of balanced resource allocation in ISAC systems enabled by drone technology.
| \(\beta\) | 0.2 | 0.35 | 0.5 | 0.65 | 0.8 |
|---|---|---|---|---|---|
| Rate (bps) | 0.686 | 0.816 | 0.742 | 0.673 | 0.598 |
4.7 Ablation Study: Effect of ISAC-Aware Module
We replace only the mixing network of MAPPO and IQL with our ISAC-Aware module (keeping the actor-critic components unchanged) to form IA-MAPPO and IA-IQL. Table 7 shows the improvement: IA-MAPPO gains 7.5% over MAPPO, and IA-IQL gains 18.5% over IQL. This confirms that the constraint-guided value decomposition is the key enabler for performance enhancement.
| Algorithm | Rate (bps) | Improvement |
|---|---|---|
| MAPPO | 0.6032 | – |
| IA-MAPPO | 0.6483 | +7.5% |
| IQL | 0.4513 | – |
| IA-IQL | 0.5350 | +18.5% |
5. Conclusion
In this paper, we tackled the joint 3D trajectory, user scheduling, and power optimization problem for a multi-UAV ISAC system. The inherent non-convexity and tight coupling between sensing and communication motivated us to develop a novel MARL framework, IA-QMIX. Our method introduces a hierarchical state encoder to capture the structure of the UAV–user graph, a constraint feasibility guidance network that embeds collision and sensing MI constraints into the value decomposition, and an adaptive mixer that dynamically adjusts the mixing weights based on the global state. Extensive simulations demonstrate that IA-QMIX significantly outperforms existing baseline algorithms in terms of average communication rate while strictly satisfying the sensing fairness constraint. The results highlight the tremendous potential of combining advanced drone technology with intelligent multi-agent coordination to realize efficient and adaptive ISAC networks. Future work will extend our framework to consider heterogeneous drone types, energy harvesting, and dynamic user mobility.