I have been investigating how multiple UAV drones can safely and orderly share low-altitude integrated airspace. The rapid increase in UAV drone operations has created a pressing need for cooperative path planning that simultaneously ensures flight safety and respects established traffic rules. In this work, I propose a method that embeds direction-dependent operational rules into the reward signal of a deep reinforcement learning algorithm through a geometric structure called the asymmetric conflict region (ACR). Unlike conventional distance-based penalties that are symmetric about the UAV drone centroid, the ACR introduces a direction-biased hexagonal region in the body frame of each UAV drone. The geometric asymmetry produces differentiated overlap penalties for different passing directions, thereby implicitly encoding right-side avoidance, right-side overtaking, and clockwise circulation rules into the learning process. I adopt a centralized training with decentralized execution (CTDE) framework based on the soft actor-critic (SAC) algorithm. Simulation results show that my method achieves task success rates above 99.83% in three typical conflict scenarios while reducing average conflict counts to nearly zero and inducing an orderly passing pattern among UAV drones. The proposed approach reduces the randomness of passing directions and enables UAV drones to form consistent, rule-compliant behaviors without requiring centralized scheduling or extensive inter-UAV communication.
Low-altitude airspace is becoming increasingly crowded as more UAV drones enter service for delivery, inspection, surveillance, and emergency response. Future operations will gradually shift from segregated to integrated airspace, where UAV drones of different types and speeds share the same volume. In such environments, inter-UAV interactions occur frequently, and communication links may be intermittent or unreliable. Each UAV drone must rely primarily on local observations and limited information exchange to make decentralized decisions. Path planning must therefore guarantee safe separation while also satisfying necessary rule constraints. Without rules, high-density scenarios can quickly degrade into disorder or conflict. Consequently, safety separation and traffic order have become parallel rigid requirements for multi-UAV drone cooperative planning.
Existing research on multi-UAV drone path planning generally follows two main directions. One direction focuses on maintaining safe separation by simplifying conflicts into distance-based penalties or potential field constraints. These signals are symmetric with respect to the UAV drone centroid and carry no directional semantics. As a result, they cannot uniformly represent passing directions or circulation rules. Although such methods can avoid collisions, they rarely produce an orderly traffic flow. The other direction attempts to explicitly introduce operational rules, but often uses rule encoding or hard constraints. Rules act on decisions as discrete judgments and cannot provide continuous feedback reflecting the degree of violation. Rule modules and policy learning remain independent, making it difficult to adaptively adjust according to dynamic interactions. From a decision architecture perspective, centralized methods can achieve global coordination but incur high computation and communication overhead and depend heavily on reliable backbone networks, which are not always available in low-altitude environments. Distributed methods scale well, but each UAV drone independently decides based on local observations, and under communication constraints, inconsistent group behaviors may emerge. A more fundamental reason is that airspace rules are specified as discrete clauses, whereas learning-based path planning relies on continuous interaction feedback to optimize policies. The two lack a unified representation. Therefore, naturally integrating discrete traffic rules into a continuous decision process is a key challenge for enabling ordered cooperative behavior among multiple UAV drones.
To address this challenge, I construct an asymmetric conflict region (ACR) for multi-UAV drone cooperative path planning. The ACR transforms operational rules from discrete clauses into continuous geometric feedback within a reinforcement learning process. Unlike methods that apply collision avoidance penalties based solely on inter-UAV distance, the ACR introduces a direction-biased hexagonal structure in the body coordinate frame of each UAV drone. Different approach directions and passing modes correspond to different region overlap states, thereby expressing right-side avoidance, right-side overtaking, and clockwise circulation rules as accumulative overlap penalty signals. On this basis, I employ the soft actor-critic (SAC) algorithm to optimize policies in continuous action spaces. A centralized training with decentralized execution (CTDE) framework enables multi-UAV policy learning and independent decision-making. Rule consistency is internalized in the shared policy during training. During execution, each UAV drone independently decides based only on local observations, without requiring a central scheduling node, global information broadcast, or inter-UAV negotiation. This feature allows my method to reduce dependence on centralized scheduling and reliable backbone links with low information exchange overhead, making it suitable for decentralized cooperative operations under communication-constrained low-altitude integrated airspace. Simulation results in three typical scenarios—head-on flight, same-direction overtaking, and multi-UAV convergence—demonstrate that my method can suppress inter-UAV conflicts while guiding UAV drones to form rule-compliant cooperative avoidance behaviors.
Related Work
Research on multi-UAV drone cooperative path planning in integrated airspace can be grouped into three areas: UAV drone path planning methods, multi-UAV cooperative decision frameworks, and integration of airspace operational rules.
Traditional path planning methods include A* search, artificial potential fields (APF), and various heuristic optimization algorithms. Some studies improve A* heuristic functions and APF repulsive-attractive mechanisms to achieve hybrid path planning in complex urban low-altitude environments. Others use piecewise potential fields to alleviate local minima in fixed-wing UAV drone formation planning. Spherical vector-based particle swarm optimization has been proposed to improve safety and trajectory smoothness in three-dimensional path planning. For multi-objective task scenarios, UAV drone data collection path planning has been modeled as an orienteering problem and solved with pointer networks. Other work combines spatiotemporal dependency graph convolutional networks with tabu search for dynamic path optimization in emergency rescue scenarios. Deep reinforcement learning (DRL) offers a new framework for autonomous navigation and path planning of UAV drones by optimizing policies through environmental interaction. DRL has been used to solve path planning problems in multi-UAV drone data collection tasks. For internet-of-vehicles service migration, a multi-agent group relative policy optimization method without an explicit critic network has been proposed. Deep deterministic policy gradient (DDPG) has enabled autonomous UAV drone navigation in urban environments. Combining hindsight experience replay (HER) with SAC has alleviated reward sparsity. Compliance-aware trajectory planning for UAV drones in low-altitude airspace has also been studied from a regulatory constraint perspective.
To improve cooperative decision-making among multiple UAV drones, multi-agent reinforcement learning (MARL) has been increasingly applied. Attention-based deep MARL frameworks have introduced centralized training with decentralized execution into multi-agent cooperative decision-making. Multi-agent proximal policy optimization (MAPPO) has shown good stability in cooperative multi-agent tasks and has been widely used in multi-UAV drone cooperative search and target tracking. Collision avoidance with deep reinforcement learning (CADRL) uses value networks to model neighbor interactions and achieves cooperative collision avoidance in multi-agent scenarios. GPU-based asynchronous advantage actor-critic collision avoidance with deep reinforcement learning (GA3C-CADRL) further improves collision avoidance in complex dynamic environments. SAC has been extended to partially observable multi-agent path planning tasks. Constrained Markov decision processes (MDP) and SAC-Lagrangian algorithms have been used for multi-UAV safe decision-making.
Regarding the integration of airspace operational constraints, some studies attempt to introduce rule constraints in conflict resolution. Early research on conflict detection and resolution has been systematically reviewed. Model predictive control has been used to build real-time conflict resolution algorithms that follow right-of-way rules. In low-altitude traffic management, the UAS traffic management (UTM) concept framework was proposed, and a finite-state UTM model based on fixed corridors was established. A unified cellular-native network architecture has been proposed to provide infrastructure support for low-altitude multi-UAV cooperative operations. In summary, existing multi-UAV drone path planning and cooperative collision avoidance research has made significant progress in safe separation, dynamic obstacle avoidance, and multi-agent cooperative decision-making. However, most methods still characterize conflict risk using distance thresholds, collision penalties, or safety constraints, and cannot directly express directional passing relationships in operational rules. Existing explicit rule methods usually rely on rule judgments, priority decisions, or hard constraint corrections, with rules acting as independent modules in the decision process. In contrast, the ACR encodes directional operational rules as overlap differences of an asymmetric geometric region, allowing rule constraints to enter the reinforcement learning reward function as continuous penalty signals, thereby guiding multiple UAV drones to form rule-consistent cooperative behaviors during policy learning.
Asymmetric Conflict Region
Conflict Region Modeling
Operational rules have clear directional constraints, whereas conventional distance thresholds or symmetric safety regions mainly describe the magnitude of inter-UAV separation and cannot distinguish passing relationships under different relative bearings. Therefore, I need to construct a conflict region with directional bias so that different approach directions and passing modes correspond to differentiated region overlap results.
The choice of region shape directly determines the expressive capability of directional bias. Circles and ellipses are symmetric about the centroid and cannot express direction-dependent passing differences. A quadrilateral can achieve asymmetry in a single direction but cannot simultaneously carry the multi-directional biases required by three typical scenarios. A hexagon, while keeping the number of vertices small, can provide three groups of boundary features that carry different directional biases: lateral bias for distinguishing left/right passing directions during head-on approach; rear bias for distinguishing the approach side during same-direction overtaking; and a mid-chord bias for distinguishing circulation directions during multi-UAV convergence. For each UAV drone \(i\), assuming it is circular in the plane with radius \(R\), I construct a hexagonal asymmetric conflict region \(\mathrm{Conflict}_i^A\) in its body local coordinate frame. The region is centered at the UAV drone centroid, and its shape is defined by six vertices as shown in the following geometric description.
Let the position of UAV drone \(i\) be \(p_i = (x_i, y_i)\) and its heading angle be \(\psi_i\). In the world coordinate system, the conflict region translates with the UAV drone position \(p_i\) and rotates with the heading angle \(\psi_i\). The position of the \(k\)-th vertex in the world coordinate system is:
$$ Z_k’ = R(\psi_i) Z_k + p_i $$
where \(R(\psi_i)\) is the two-dimensional rotation matrix:
$$ R(\psi_i) = \begin{bmatrix} \cos\psi_i & -\sin\psi_i \\ \sin\psi_i & \cos\psi_i \end{bmatrix} $$
This ensures that the conflict region always maintains a consistent relative position with respect to the UAV drone flight direction.
The six vertices in the body frame are defined as:
$$ Z_1 = (-1.5R, 0.5R), \quad Z_2 = (-0.75R, -3R), \quad Z_3 = (0.75R, -3R) $$
$$ Z_4 = (1.5R, 0.5R), \quad Z_5 = (R, 1.5R), \quad Z_6 = (-R, 1.5R) $$
Here, the lateral bias is reflected by the asymmetry between the left and right sides. The rear bias is reflected by the extension toward the negative y-direction. The mid-chord bias is reflected by the line connecting \(Z_1\) and \(Z_4\), which lies above the centroid.
Conflict Region Feature Analysis
The geometric features of the ACR must simultaneously satisfy two requirements: safe separation and direction discrimination. The core lies in changing the region overlap state through bias relationships between different directions. When the directional bias is weak, the difference in overlap states corresponding to different passing directions is limited, making it difficult to stably distinguish expected passing directions from undesired ones. When the directional bias is strong, the conflict region occupies too large an area and imposes additional penalties on traffic that already satisfies normal separation. I analyze the induced effects on policy behavior for three typical conflict scenarios.
Head-on Flight
In head-on flight, direction discrimination is mainly reflected by the lateral bias of the ACR on both sides of the heading centerline. Let the maximum extension on the compliant passing side be \(\delta_1\) and the maximum extension on the violating passing side be \(\delta_2\). When two UAV drones cross head-on with lateral separation \(d_{\mathrm{avoid}}\), the sum of extensions on the adjacent side determines the ACR overlap result. \(\delta_1\) and \(\delta_2\) must respectively satisfy:
$$ \delta_1 \le \frac{d_{\mathrm{avoid}}}{2}, \quad \delta_2 \ge \frac{d_{\mathrm{avoid}}}{2}, \quad \delta_1 < \delta_2 $$
The lower bound of \(\delta_1\) ensures that the ACR covers the UAV drone body, and the upper bound ensures that right-side passing does not trigger overlap. The lower bound of \(\delta_2\) ensures that left-side passing triggers overlap, and the upper bound is the normal cruise separation (typically twice the minimum avoidance distance). I set \(\delta_1 = 1.5R\) and \(\delta_2 = 3R\), meaning the y-coordinates of \(Z_5\) and \(Z_6\) are \(1.5R\), and those of \(Z_2\) and \(Z_3\) are \(-3R\). When passing on the right, the sum of adjacent extensions is \(2\delta_1 = 3R\), which exactly equals \(d_{\mathrm{avoid}} = 3R\), placing the ACR in a critical state with no overlap (safe). When passing on the left, the sum of extensions is \(2\delta_2 = 6R\), exceeding \(d_{\mathrm{avoid}}\) by a factor of two, producing an overlap depth of approximately \(3R\) in the lateral direction (invade). The two passing directions exhibit clear left-right asymmetry in the penalty signal.
Same-direction Overtaking
In same-direction overtaking, the protruding part of the ACR relative to the heading centerline exhibits an upper-lower asymmetric feature. The y-coordinates of vertices \(Z_1\) and \(Z_4\) are \(0.5R\), located above the heading centerline, making the protruding areas on the rear left and rear right sides of the conflict region different. This geometric feature produces differentiated overlap results when a following UAV drone overtakes from different sides. When a following UAV drone overtakes from the left side of a leading UAV drone with a \(15^\circ\) angle, its forward region intrudes into the larger protruding area on the rear left of the leading UAV drone, causing overlap (invade). When overtaking from the right side with the same angle, its forward region is adjacent to the smaller area on the rear right of the leading UAV drone, and no overlap occurs (safe). At a separation of \(d_{\mathrm{avoid}} = 3R\), the width of the ACR region traversed by the left overtaking path is about \(1.5R\), while the right overtaking path is tangent to the ACR boundary, with an overlap width approximately zero. This difference, guided by the cumulative penalty signal, gradually shapes a right-side overtaking preference during training.
Multi-UAV Convergence
In multi-UAV convergence, the widest boundary of the ACR is formed by the line connecting vertices \(Z_1 = (-1.5R, 0.5R)\) and \(Z_4 = (1.5R, 0.5R)\), located at a mid-chord bias of \(h = 0.5R\) above the centroid. This is the conflict edge that adjacent UAV drones first contact during dense interactions. This geometric feature gives different safety margins for clockwise and counterclockwise circulation directions. Taking points on this conflict edge as centers and drawing arcs with the same radius yields a limiting arc that just avoids region intrusion. A UAV drone circulating clockwise is located one \(h\) below this limiting arc, providing more adjustment space. A UAV drone circulating counterclockwise is located one \(h\) above the limiting arc, providing less adjustment margin. Under the same circulation radius, the radial separation difference between the two circulation trajectories is \(2h = R\), giving clockwise circulation a larger safety margin. The policy is more likely to maintain a conflict-free state during training, and the system spontaneously converges to a clockwise circulation pattern in multi-UAV convergence scenarios.
SAC-Based Multi-UAV Autonomous Flight Decision Algorithm
Problem Description and Modeling
In low-altitude airspace, multiple UAV drones need to perform flight tasks simultaneously in a shared space, with each UAV drone flying from a specified start point to a target point. During flight, UAV drones must autonomously complete path planning, dynamic obstacle avoidance, and cooperative decision-making that complies with airspace operational rules. I model the multi-UAV drone autonomous flight decision problem as an MDP and solve it using deep reinforcement learning.
Assume there are \(N\) UAV drones in the airspace, each cruising at a fixed altitude layer, simplifying the problem to two-dimensional plane operation. Under the MDP framework, I define a five-tuple \((S, A, P, R, \gamma)\), where \(S\) is the state space, including the UAV drone’s own position, heading, speed, and local observations of the surrounding environment and other aircraft; \(A\) is the action space, consisting of linear velocity \(v\) and angular velocity \(\omega\); \(P(s_{t+1} | s_t, a_t)\) is the state transition probability, describing the dynamic process of the UAV drone reaching the next state after executing an action in the current state; \(R\) is the reward function, comprehensively evaluating navigation efficiency, flight safety, and rule compliance; and \(\gamma \in (0,1)\) is the discount factor, balancing immediate rewards and long-term returns. Due to the complexity of the multi-UAV interaction environment, the state transition probability \(P\) is unknown and must be learned through interaction with the environment.
At the multi-agent architecture level, I adopt a parameter-sharing CTDE framework. During training, \(N\) UAV drones share the same policy network parameters and experience replay buffer. Each UAV drone generates an experience tuple \((s_t^i, a_t^i, r_t^i, s_{t+1}^i)\) at each time step, which is uniformly written into a common buffer. The same training process samples and updates network parameters. During execution, each UAV drone independently decides based on its own state and local observations of neighboring UAV drone positions, velocities, and headings. This framework ensures multi-UAV policy consistency while reducing communication and scheduling overhead at the execution end, making it suitable for cooperative operations where each UAV drone makes autonomous decisions in low-altitude airspace.
SAC Algorithm
Based on the above MDP modeling, I use the SAC algorithm to solve the multi-UAV drone autonomous flight decision problem. SAC introduces a maximum entropy regularization term on top of the actor-critic architecture. Its optimization objective is to maximize the expected “soft return”, and the objective function can be expressed as:
$$ J(\pi) = \sum_{t=0}^{T} \mathbb{E}_{(s_t, a_t) \sim \rho_\pi} \left[ r(s_t, a_t) + \alpha \mathcal{H}(\pi(\cdot | s_t)) \right] $$
where \(\mathbb{E}\) denotes the expectation over the trajectory distribution induced by policy \(\pi\), \(r(s_t, a_t)\) is the immediate environmental reward obtained by the agent executing action \(a_t\) in state \(s_t\), \(\mathcal{H}(\pi(\cdot | s_t)) = -\mathbb{E}[\ln \pi(a_t | s_t)]\) is the information entropy of the policy at that state, and the temperature coefficient \(\alpha\) balances the relative weight of the reward term and the entropy term. When \(\alpha\) is large, the policy favors exploration; when \(\alpha\) approaches zero, it degenerates to the standard reinforcement learning cumulative return maximization form.
To achieve the maximum entropy optimization objective, SAC decomposes it into five modules: environment interaction and experience replay, critic network, target network, actor network, and temperature coefficient. The implementation details are as follows.
Environment Interaction and Experience Replay
At each decision time step \(t\), the agent samples action \(a_t\) from the current policy \(\pi_\phi\) based on state \(s_t\) and outputs it to the environment. The environment returns the immediate reward \(r_t\) and the next state \(s_{t+1}\), forming a quadruple experience \((s_t, a_t, r_t, s_{t+1})\). The generated experience, along with the log probability of the policy \(\ln \pi_\phi(a_t | s_t)\), is written into the experience buffer. During training, mini-batches are uniformly sampled to update the critic and actor networks synchronously.
Critic Network
SAC uses two independently parameterized Q-networks \(Q_{\theta_1}\) and \(Q_{\theta_2}\) to estimate the state-action value function. The training objective is to minimize the soft Bellman error:
$$ J_Q(\theta_i) = \mathbb{E}_{(s_t, a_t, r_t, s_{t+1}) \sim \mathcal{D}} \left[ \frac{1}{2} \left( Q_{\theta_i}(s_t, a_t) – y_t \right)^2 \right] $$
where \(i = 1,2\). When computing the target value, the smaller output of the two networks is taken to suppress overestimation of the value function. In my reward function, navigation efficiency is dominant, and conflict rule penalties are auxiliary. This guides the policy to gradually converge to standard-compliant flight behavior after mastering basic navigation capabilities. If the Q-value is overestimated, the policy will tend to ignore the penalty signal of conflict rules, and rule-compliant behavior will be difficult to acquire.
Target Network
The target network has the same structure as the critic network, but its parameters slowly track the main network through soft updates, specifically for generating stable target Q-values:
$$ y_t = r_t + \gamma \min_{j=1,2} Q_{\theta_j’}(s_{t+1}, a_{t+1}) – \alpha \log \pi_\phi(a_{t+1} | s_{t+1}) $$
where \(a_{t+1} \sim \pi_\phi(\cdot | s_{t+1})\). The entropy regularization term \(-\alpha \log \pi_\phi\) imposes an implicit penalty on deterministic actions: if the policy output is highly concentrated in a certain state, \(\log \pi_\phi\) approaches 0, the entropy \(\mathcal{H}\) is small, and the target Q-value decreases accordingly, guiding the policy to maintain a certain degree of randomness. The target network parameters are iteratively updated according to the soft update rule:
$$ \theta’ \leftarrow \tau \theta + (1-\tau) \theta’ $$
Actor Network
The actor network parameterizes the policy \(\pi_\phi\) as a conditional Gaussian distribution, outputting the mean and variance of the linear velocity and angular velocity. Through the reparameterization trick, the policy gradient can backpropagate along the sampled actions. The optimization objective is:
$$ J_\pi(\phi) = \mathbb{E}_{s_t \sim \mathcal{D}, \epsilon \sim \mathcal{N}} \left[ \alpha \log \pi_\phi(a_t | s_t) – \min_{j=1,2} Q_{\theta_j}(s_t, a_t) \right] $$
This objective simultaneously drives the policy toward high Q-values and high entropy, with the two balanced by \(\alpha\).
Temperature Coefficient \(\alpha\)
\(\alpha\) is automatically adjusted by minimizing the following loss:
$$ J(\alpha) = \mathbb{E}_{a_t \sim \pi_\phi} \left[ -\alpha \left( \log \pi_\phi(a_t | s_t) + \mathcal{H}_{\mathrm{target}} \right) \right] $$
When the policy entropy is lower than the target entropy \(\mathcal{H}_{\mathrm{target}}\), the loss drives \(\alpha\) to increase and strengthen exploration. When the policy entropy is higher than the target entropy, \(\alpha\) decreases to focus on exploitation. I set the target entropy to \(\mathcal{H}_{\mathrm{target}} = -2\), corresponding to the dimension of the two-dimensional action space, ensuring that the policy maintains moderate randomness in both linear and angular velocity dimensions.
State Space
For \(N\) UAV drones in the airspace, I design a hybrid state space for each UAV drone \(i\) that combines a structured vector and a local grid map. The structured vector consists of three parts: target relative state, neighbor relative state, and group motion state, representing target navigation, conflict perception, and cooperative avoidance, respectively. The target direction deviation angle \(\Delta \psi_i^{\mathrm{goal}}\) and target distance \(d_i^{\mathrm{goal}}\), as well as the neighbor relative bearing angle \(\Delta \psi_{ij}\) and relative distance \(d_{ij}\), are expressed in the body coordinate system. The calculations are as follows:
$$ \Delta \psi_i^{\mathrm{goal}} = \arctan2(y_{\mathrm{goal}} – y_i, x_{\mathrm{goal}} – x_i) – \psi_i $$
$$ d_i^{\mathrm{goal}} = \| p_i^{\mathrm{goal}} – p_i \|_2 $$
$$ \Delta \psi_{ij} = \arctan2(y_j – y_i, x_j – x_i) – \psi_i $$
$$ d_{ij} = \| p_i – p_j \|_2 $$
The local grid map \(\mathrm{GridMap}_i\) is centered on the UAV drone itself, aligned with the current heading as the positive direction. The grid map size is \(11 \times 11\), with a resolution of 2 m per cell, covering a square area with a side length of 22 m. The grid is assigned values based on content: free space is 0, static obstacles are 1, and dynamic obstacles are 0.5. The flattened grid map is a 121-dimensional vector, which is concatenated with the structured vector to form the complete state vector \(s_t^i\). The components are shown in Table 1.
| Component | Meaning | Dimension |
|---|---|---|
| \(\Delta \psi_i^{\mathrm{goal}}, d_i^{\mathrm{goal}}\) | Target relative state | 2 |
| \(v_j, \omega_j\) | All linear and angular velocities | \(2N\) |
| \(\Delta \psi_{ij}, d_{ij}\) | Neighbor relative state | \(2(N-1)\) |
| \(\mathrm{active}_j\) | Active flag | \(N\) |
| \(\mathrm{GridMap}_i\) | Local grid map | 121 |
Action Space
The motion of a UAV drone in a two-dimensional plane is jointly determined by linear velocity \(v\) and angular velocity \(\omega\). Within each control period \(\Delta t = 0.1\) s, the UAV drone updates its position and heading according to the following kinematic model:
$$ v = v_{\min} + \frac{a_1 + 1}{2} (v_{\max} – v_{\min}) $$
$$ \omega = \omega_{\min} + \frac{a_2 + 1}{2} (\omega_{\max} – \omega_{\min}) $$
I take \(v \in [0, 15]\) m/s and \(\omega \in [-1, 1]\) rad/s. The lower bound of linear velocity is set to zero, allowing the UAV drone to hover and wait in severe conflict situations. The angular velocity is symmetric about zero, ensuring that the learning process of conflict rules is not affected by directional bias inherent in the action space itself.
Reward Function
The reward function determines the behavior learned by the agent. I consider three types of objectives: navigation efficiency, flight safety, and airspace conflict rules. I design a multi-component weighted reward function consisting of navigation reward \(r_{\mathrm{nav}}\), time penalty \(r_{\mathrm{time}}\), rule penalty \(r_{\mathrm{norm}}\), and terminal reward \(r_{\mathrm{terminal}}\).
$$ r_t^i = r_{\mathrm{nav}} + r_{\mathrm{time}} + r_{\mathrm{norm}} + r_{\mathrm{terminal}} $$
The rule penalty is determined by the geometric overlap state of the ACR between UAV drones. Let \(C_i^t\) and \(C_j^t\) be the asymmetric conflict regions of UAV drones \(i\) and \(j\) at time \(t\). Their vertices are transformed from the body coordinate system to the world coordinate system based on the current position and heading angle. The overlap area of the two ACRs is defined as:
$$ S_{ij}^t = \mathrm{Area}(C_i^t \cap C_j^t) $$
where \(\mathrm{Area}\) denotes the area of the intersection of two convex hexagons. When the overlap area is greater than a given tolerance \(\epsilon\), the two UAV drones are considered to have an ACR overlap, and the indicator function is:
$$ I_{\mathrm{overlap}}^t(i,j) = \begin{cases} 1, & S_{ij}^t > \epsilon \\ 0, & S_{ij}^t \le \epsilon \end{cases} $$
where \(\epsilon\) is a very small positive number used to avoid boundary misjudgment caused by numerical errors. Only when the boundaries are tangent or the overlap area does not exceed \(\epsilon\) is the ACR overlap penalty not counted. The expressions and coefficients of the continuous components are shown in Table 2.
| Component | Symbol | Expression | Coefficient |
|---|---|---|---|
| Navigation reward | \(r_{\mathrm{nav}}\) | \(c_{\mathrm{fwd}} (d_{t-1}^{\mathrm{goal}} – d_t^{\mathrm{goal}})\) | \(c_{\mathrm{fwd}} = 5\) |
| Time penalty | \(r_{\mathrm{time}}\) | \(c_{\mathrm{time}}\) | \(c_{\mathrm{time}} = -2\) |
| Rule penalty | \(r_{\mathrm{norm}}\) | \(c_{\mathrm{norm}} \sum_{j \neq i} I_{\mathrm{overlap}}^t(i,j)\) | \(c_{\mathrm{norm}} = -10\) |
The reward coefficients are determined jointly by the single-step motion scale, rule constraint strength, and episode termination results. The navigation reward uses the distance difference form \(c_{\mathrm{fwd}} (d_{t-1}^{\mathrm{goal}} – d_t^{\mathrm{goal}})\) to measure the immediate benefit obtained by the UAV drone moving toward the target. Under the control period \(\Delta t = 0.1\) s and maximum linear velocity \(v_{\max} = 15\) m/s, the maximum single-step distance reduction is about 1.5 m, corresponding to a maximum navigation reward of about 7.5. The time penalty is \(c_{\mathrm{time}} = -2\), whose absolute value is smaller than the navigation gain during effective forward movement. It is used to suppress hovering and ineffective detours without changing the basic optimization direction toward the target. The rule penalty is \(c_{\mathrm{norm}} = -10\), slightly larger than the maximum single-step navigation reward, ensuring that it is difficult for a UAV drone to obtain a positive net benefit by shortening its path when violating rules and triggering ACR overlap, thereby suppressing illegal shortcut strategies. At the same time, the penalty is not set too large to avoid rule constraints continuously dominating the reward signal and weakening the target guidance effect.
| Termination Event | Type | Reward Value |
|---|---|---|
| Reaching target | Success | 120 |
| Collision with static obstacle | Static collision | −40 |
| Reaching maximum steps | Timeout | −40 |
| Collision with other UAV drone | Dynamic collision | −15 |
| Long-term stagnation | Abnormal | −10 |
| Cumulative yaw deviation \(> 4\pi\) | Abnormal | −10 |
Once an episode triggers any termination condition, the above continuous accumulation stops immediately, and a one-time terminal reward \(r_{\mathrm{terminal}}\) is given instead. The terminal reward is divided into four categories—success, collision, timeout, and abnormal—according to the nature of the event, providing clear feedback on the episode result with relatively large values. The values are shown in Table 3.
Experiments and Results Analysis
To verify the effectiveness of the SAC-ACR method, I conducted simulation experiments in three typical conflict scenarios: head-on flight, same-direction overtaking, and multi-UAV convergence. I evaluated performance from three dimensions: training convergence, conflict resolution capability, and rule consistency. I compared my method with A*+APF, DRL, CADRL, and a SAC method without ACR constraints. All experiments were conducted under the same state space, action space, and network structure, repeated with three independent random seeds (12, 34, and 42) to ensure reliability and reproducibility.
Experimental Setup
The experiments were implemented based on PyTorch, and training was completed on a single NVIDIA RTX 4060 GPU. The simulation airspace was a \(300 \text{ m} \times 150 \text{ m}\) rectangular area. The environment was updated with a fixed step size of \(\Delta t = 0.1\) s, and crossing the boundary was treated as a static collision. The UAV drone physical parameters were set according to a large cargo model, with a maximum linear velocity of 15 m/s, a maximum angular velocity of 1.0 rad/s, and a collision radius of 1.5 m. The SAC network used two fully connected layers with 500 neurons each. The relevant training hyperparameters are shown in Table 4.
| Parameter | Symbol | Value |
|---|---|---|
| Learning rate | \(lr\) | \(1 \times 10^{-4}\) |
| Batch size | \(batch\) | 4096 |
| Discount factor | \(\gamma\) | 0.99 |
| Soft update coefficient | \(\tau\) | 0.01 |
| Target entropy | \(\mathcal{H}_{\mathrm{target}}\) | −2 |
For the three typical conflict scenarios common in low-altitude operations, I constructed corresponding simulation scenarios. The head-on flight scenario corresponds to two-way routes, corridor intersections, or opposing heading tasks. The same-direction overtaking scenario corresponds to overtaking conflicts caused by UAV drones with different speeds on the same route or similar trajectory. The multi-UAV convergence scenario corresponds to multi-directional convergence conflicts at low-altitude route nodes, task assembly areas, or entry/exit intersection areas. The head-on flight and same-direction overtaking scenarios were each trained for 2500 episodes, and the multi-UAV convergence scenario was trained for 1000 episodes.
Convergence Analysis
The training curves of SAC-ACR in the three scenarios are shown in the following analysis. Each scenario curve includes the episode cumulative reward \(R\), the number of ACR overlaps \(N_c\), and the task success rate \(P_s\), reflecting overall policy performance, rule compliance level, and task completion, respectively. Each curve uses a solid line for the mean across seeds and a shaded band for the standard deviation. The curves in the three scenarios exhibit a consistent three-stage evolution pattern. In the early training stage, the policy has not yet established an effective state-action mapping, and UAV drones hover near the starting point, with \(R\), \(N_c\), and \(P_s\) all at low levels. In the middle training stage, the policy acquires actions tending toward the target under the guidance of the distance reduction reward, and \(R\) rises steadily. However, because avoidance rules have not yet been mastered, the routes of different UAV drones frequently intersect, and \(N_c\) climbs to a training peak. In the late training stage, the navigation policy gradually stabilizes, and the ACR overlap penalty becomes the main factor affecting \(R\). The policy converges along the direction indicated by the geometric asymmetry to an interaction mode that complies with conflict rules. \(N_c\) decays from the peak and approaches zero, \(P_s\) stabilizes near 100%, and \(R\) converges simultaneously. The shaded bands of the three seeds narrow significantly at the end of training, indicating good reproducibility.
The training convergence results of the three scenarios are shown in Table 5. The mean of the last 200 training episodes for each seed was taken as the statistical sample. Training metrics are given as mean ± standard deviation. At the end of training, the conflict count \(N_c^f\) is close to zero, and the success rate \(P_s^f\) is stable above 99.83%. Among them, the same-direction overtaking scenario has a smaller relative speed between UAV drones, and the peak conflict count \(N_c^{\mathrm{peak}}\) is significantly higher than in the other scenarios. In the multi-UAV convergence scenario, the geometric asymmetry of the ACR indicates a consistent clockwise circulation direction for each UAV drone, and the group quickly forms a unified avoidance pattern. The convergence episode is the earliest, and the policy learning efficiency is the highest.
| Scenario | \(N_c^{\mathrm{peak}}\) | Peak Episode | Convergence Episode | \(R^f\) | \(N_c^f\) | \(P_s^f\) |
|---|---|---|---|---|---|---|
| Head-on flight | 291 | 23 | 857 | 3429.60 ± 1.76 | 0.077 ± 0.080 | 100.00 ± 0.00 |
| Same-direction overtaking | 201 | 42 | 1258 | 2495.63 ± 10.04 | 0.075 ± 0.117 | 99.83 ± 0.22 |
| Multi-UAV convergence | 59 | 37 | 415 | 2764.21 ± 6.97 | 0.008 ± 0.014 | 100.00 ± 0.00 |
Flight Trajectory and Rule Consistency Analysis
The converged flight trajectories of SAC-ACR in the three scenarios are analyzed as follows. In the head-on flight scenario, when two groups of UAV drones approach each other, both deflect to their respective right sides and occupy the right half of the airspace to complete the crossing. In the same-direction overtaking scenario, when a faster UAV drone group approaches a slower group, it forms a lateral offset to the right, completes the overtaking from the right side, and then returns to the route. In the multi-UAV convergence scenario, six UAV drones deflect to the same side in advance when approaching the central area, forming a clockwise circulation before flying to their respective targets. Overall, the behavior is consistent with the right-side avoidance, right-side overtaking, and clockwise circulation rules corresponding to the ACR geometric asymmetry.
To further analyze the induction effect of the ACR on operational rules, I introduce a rule compliance rate metric, which is the proportion of successful test episodes that satisfy the corresponding operational rule. I set up four models for comparison: no ACR, circular ACR, the proposed ACR, and reverse-bias ACR. The reverse-bias ACR is a control model obtained by flipping the asymmetric structure of the proposed ACR about the heading axis.
The rule induction trajectories under different conflict regions show that no ACR and circular ACR lack clear directional bias, and the passing direction after convergence is random. The proposed ACR and reverse-bias ACR induce opposite passing trends. The probabilities of rule-compliant behavior appearing in test episodes are shown in Table 6.
| Scenario | Operational Rule | Compliance Rate \(P_n\) (%) |
|---|---|---|
| Head-on flight | Right-side avoidance | 94.8 |
| Same-direction overtaking | Right-side overtaking | 90.4 |
| Multi-UAV convergence | Clockwise circulation | 91.7 |
Algorithm Comparison in Multi-UAV Convergence Scenario
To verify the effectiveness of the SAC-ACR method, I compared it with A*+APF, DRL, CADRL, and SAC without ACR constraints in the multi-UAV convergence scenario. In this scenario, multiple UAV drones approach from different directions simultaneously, and local interaction relationships change continuously during motion. Without unified rule guidance, problems such as random circulation directions, local conflict propagation, and disordered group behavior easily occur. This scenario can fully reflect the group cooperation effect after combining ACR constraints with SAC policy learning. A*+APF is a traditional planning method, DRL and CADRL are multi-agent reinforcement learning methods based on policy gradients, and SAC-noACR is completely consistent with the proposed method in network structure and hyperparameters, only masking the ACR penalty term. A*+APF does not rely on learning mechanisms, and the other deep reinforcement learning methods were trained for 1000 episodes under the same environment and random seed conditions. The training curve comparison is shown in the following analysis.
The training process shows that different methods have significant differences in convergence characteristics. SAC-ACR achieves stable convergence in fewer training episodes, with a relatively smooth increase in cumulative reward and small overall fluctuations. Although SAC-noACR can gradually approach the cumulative reward level of SAC-ACR, the conflict count in the middle training stage is significantly higher than that of SAC-ACR, and the safety improvement lags. DRL converges slowly and still does not reach a stable state at the end of training. CADRL shows some improvement in the early stage but exhibits noticeable fluctuations in the middle and late stages, making it difficult to maintain a stable cooperative obstacle avoidance policy. In terms of conflict control, SAC-ACR can gradually suppress conflicts during training and eventually approach zero, while other methods have overall higher conflict levels or still exhibit fluctuations at the end of training. In terms of task success rate, SAC-ACR also shows faster improvement and a more stable convergence trend.
To evaluate the actual flight decision-making capability of each algorithm after training, I conducted 30 independent evaluation episodes in the multi-UAV convergence scenario, with different random initialization conditions for each episode. I statistically analyzed the task success rate \(P_s\), average cumulative reward \(R\), average conflict count \(N_c\), and clockwise compliance rate \(P_n\). The results are shown in Table 7. A*+APF corrects the trajectory according to a predetermined path and potential field repulsion. When multiple UAV drones converge, it lacks a unified yield rule and results in mutual deadlock. Among the 30 evaluations, there was no episode where all UAV drones succeeded, and the compliance rate could not be calculated. DRL is an algorithm for single-UAV avoidance. Each UAV drone independently decides based only on its own observations, and there is no group-level cooperation mechanism. The passing directions are unrelated, and a stable circulation pattern is not formed, so the compliance rate also cannot be calculated. CADRL has a compliance rate of 86.7%, but the direction selection still fluctuates. Both SAC-noACR and SAC-ACR achieve zero conflicts and 100% success rate, and there is no difference at the collision avoidance level. SAC-noACR lacks directional constraints, and the circulation direction after policy convergence is determined by training randomness. The passing direction is inconsistent across episodes, and the compliance rate is only 43.3%, close to the random level. SAC-ACR achieves a compliance rate of 93.3% with the help of ACR.
| Algorithm | Conflict \(N_c\) | Cumulative Reward \(R\) | Success Rate \(P_s\) (%) | Compliance Rate \(P_n\) (%) |
|---|---|---|---|---|
| A*+APF | 54.90 ± 14.86 | −479.77 ± 459.75 | 25.56 ± 9.52 | — |
| DRL | 0.03 ± 0.18 | 1603.33 ± 519.69 | 93.89 ± 10.25 | — |
| CADRL | 0.77 ± 1.38 | 2586.96 ± 99.35 | 98.33 ± 5.09 | 86.7 |
| SAC-noACR | 0.00 ± 0.00 | 2760.33 ± 6.81 | 100.00 ± 0.00 | 36.7 |
| SAC-ACR | 0.00 ± 0.00 | 2760.13 ± 6.57 | 100.00 ± 0.00 | 93.3 |
Conclusion
I have addressed the problem that rule constraints and autonomous decision-making are difficult to unify in multi-UAV drone cooperative operations in low-altitude integrated airspace. I constructed a cooperative path planning method based on an asymmetric conflict region. Through direction-dependent geometric region overlap relationships, I transformed traditional operational rules described by discrete clauses into continuous penalty signals perceivable by reinforcement learning. This allows agents to gradually form consistent cooperative avoidance behavior during interaction. Simulation results show that in the set typical conflict scenarios, my method maintains near-zero conflicts and a high task success rate while enabling group behavior to exhibit an orderly passing pattern of right-side avoidance, right-side overtaking, and clockwise circulation, reducing the randomness of passing directions. The proposed approach provides a promising solution for enabling multiple UAV drones to operate safely and orderly in shared low-altitude airspace without relying on centralized scheduling or extensive communication.
Future work could extend the ACR framework to three-dimensional operations, incorporate heterogeneous UAV drone performance characteristics, and investigate adaptive bias parameters that respond to changing traffic density. Additionally, integrating the method with real-time communication constraints and dynamic airspace configurations would further enhance its practical applicability for large-scale UAV drone operations.