I address a problem that keeps resurfacing whenever I try to scale low-altitude operations: a swarm of UAV drone platforms can be trained to avoid each other, and it can be trained to be efficient, but it rarely learns to behave in an orderly way. In an integrated low-altitude airspace, separation assurance and traffic order are two rigid requirements that must hold simultaneously. If I penalize a UAV drone only for getting too close to another UAV drone, the resulting policy is direction-blind. If I encode the rules as discrete clauses, the policy loses the continuous feedback that gradient-based learning needs. In this work I develop a geometric construct, which I call the asymmetric conflict region (ACR), that removes this tension. The ACR is a direction-biased hexagon attached to the body frame of every UAV drone. Its overlapping behaviour with the ACR of a neighbouring UAV drone depends on the passing direction, and therefore the standard keep-right avoidance, keep-right overtaking, and clockwise circulation conventions can be expressed as a continuous, accumulable penalty signal. I optimize the policy with soft actor-critic (SAC) under a centralized-training-decentralized-execution (CTDE) regime, so that rule consistency is internalized in a shared policy during training, while each UAV drone acts independently at runtime using only its own observation.
1. Introduction
The density of UAV drone traffic in low-altitude airspace keeps rising, and operations are migrating from segregated corridors towards fused, mixed-equipage airspace. In such airspace, heterogeneous UAV drone types with very different cruise speeds share the same volume, and the number of pairwise interactions grows quadratically with the number of vehicles. At the same time, network coverage at low altitude is discontinuous and link quality fluctuates, so a centralized scheduler or a reliable backbone broadcast cannot be assumed. Each UAV drone must therefore plan its path from a local observation, with only limited information exchange with its neighbours.
Two families of methods dominate the literature, and each of them solves only half of the problem. The first family keeps a safety buffer by construction: distance-threshold penalties, repulsive potential fields, or safety-set constraints. These signals are symmetric about the centroid of the UAV drone. They carry no directional semantics. A swarm trained with such signals avoids collisions, yet the resulting traffic flow has no structure. The second family introduces the operating rules explicitly, most often as discrete logic, priority assignment, or hard constraint projection. Rules then act as an independent module that gates the decision, they provide no continuous measure of how badly a rule was violated, and they cannot adapt to a dynamic interaction pattern. The deeper obstacle is representational: airspace rules are written as discrete clauses, while learning-based planners are driven by continuous interaction feedback. There is no common language between the two.
I argue that the missing link is geometric rather than logical. A right-of-way rule is, at its core, a statement about which side of a vehicle the other vehicle is allowed to occupy. If the safety region around a UAV drone is made deliberately asymmetric, then the overlap state of two such regions becomes a continuous function of the relative bearing, and the rule is expressed as an inequality on overlap depth rather than as a Boolean test. This is the core idea I pursue here. The contributions I report are the following. First, I construct a hexagonal ACR in the body frame of each UAV drone whose directional bias is controlled by three independent parameter groups. Second, I show analytically how each of the three canonical conflict scenarios maps onto a distinct geometric feature of the ACR. Third, I embed the ACR penalty into a multi-component reward and solve the resulting Markov decision process with SAC under CTDE. Fourth, I quantify the behaviour of the resulting policy in terms of task success, conflict count, and rule-compliance rate, and I compare it against traditional planning and against learning baselines.
2. Problem Formulation and Design Requirements
I consider N UAV drone platforms cruising at a fixed altitude layer inside a rectangular volume. Each UAV drone i has an initial position and a goal position, a bounded linear speed, and a bounded yaw rate. Because all vehicles share one altitude layer, I project the problem onto the horizontal plane. Every UAV drone must reach its goal, must avoid collisions with static obstacles and with other UAV drone platforms, and must do so in a way that is consistent with the directional conventions of the airspace.
I formalize four requirements that any conflict representation must satisfy if it is to be used inside a reinforcement-learning loop.
| ID | Requirement | Implication for the conflict representation |
|---|---|---|
| R1 | Safety separation | The region must fully cover the physical footprint of the UAV drone and grow monotonically with closing risk. |
| R2 | Directional discrimination | Different relative bearings must produce different overlap outcomes at the same separation distance. |
| R3 | Continuity | Overlap must vary continuously with position and heading, so that a gradient exists with respect to the observation. |
| R4 | Low interaction cost | Evaluation must be cheap enough to run inside the simulation loop of every UAV drone at every step. |
Requirement R2 is the one that a circular or elliptical region cannot satisfy, because such regions are invariant under reflection about the heading axis. A quadrilateral can realize a bias along a single lateral direction but cannot simultaneously carry the three distinct biases required by head-on encounters, same-direction overtaking, and multi-directional convergence. A hexagon is the smallest polygon that provides three independent boundary groups, and this is the reason I selected it. The asymmetry does not come from a rotation of the shape; it comes from an unequal distribution of the vertices around the heading axis.
3. The Asymmetric Conflict Region
3.1 Geometry in the Body Frame
I model each UAV drone i as a disc of radius R. In its own body frame, whose first axis points along the instantaneous heading and whose second axis points to the left of that heading, I define a convex hexagon with six vertices. I express the vertices in units of R so that the geometry scales with the physical footprint of the platform.
| Vertex | Body-frame coordinate | Geometric role |
|---|---|---|
| Z1 | (−1.5R, 0.5R) | Rear-left extremity; defines the aft bias |
| Z2 | (−0.75R, −3R) | Right-side apex, aft corner |
| Z3 | (0.75R, −3R) | Right-side apex, fore corner |
| Z4 | (1.5R, 0.5R) | Forward extremity; defines the mid-chord offset |
| Z5 | (R, 1.5R) | Left-side extremity, fore corner |
| Z6 | (−R, 1.5R) | Left-side extremity, aft corner |
Three parameter groups control the shape. The lateral bias pair sets the maximum extensions normal to the heading axis, and I write the compliant-side extension as δ1 and the non-compliant-side extension as δ2. The aft bias pair sets the longitudinal reach of the rear vertices, which determines how early a following UAV drone enters the region. The mid-chord offset h locates the widest interior chord of the hexagon relative to the heading axis.
| Parameter | Symbol | Value | Function |
|---|---|---|---|
| Compliant-side lateral extension | δ1 | 1.5R | Sets the lateral clearance that a rule-conforming pass consumes |
| Non-compliant-side lateral extension | δ2 | 3R | Sets the overlap depth generated by a rule-violating pass |
| Mid-chord offset | h | 0.5R | Creates the clearance asymmetry between circulation directions |
| Reference avoidance distance | davoid | 3R | Nominal lateral separation used to calibrate the bias |
The calibration is driven by two inequalities. The compliant side must be exactly tangent at the reference separation, while the non-compliant side must overlap substantially at the same separation:
$$2\delta_{1} = d_{\text{avoid}}, \qquad 2\delta_{2} = 2\,d_{\text{avoid}}$$
Substituting the values above gives a compliant-side sum of 3R, which is exactly the reference avoidance distance, and a non-compliant-side sum of 6R, which exceeds it by a factor of two. The asymmetry therefore is not a matter of degree; it is a factor-two difference in the penalty that two mirror-image manoeuvres receive at an identical separation distance.
3.2 Transformation to the World Frame
For a UAV drone i at position pi = (xi, yi) with heading ψi, the k-th vertex is mapped to the world frame by a rigid transform:
$$Z_{k}^{W} = R(\psi_{i})\,Z_{k}^{B} + p_{i}$$
$$R(\psi_{i}) = \begin{bmatrix} \cos\psi_{i} & -\sin\psi_{i} \\ \sin\psi_{i} & \cos\psi_{i} \end{bmatrix}$$
The rotation matrix guarantees that the directional bias always travels with the vehicle and rotates with its heading, so that the rule semantics are attached to the direction of motion rather than to a fixed global axis. This property is what allows a single shared policy to be deployed in an arbitrary inertial orientation without retraining.
3.3 Overlap Evaluation
For two UAV drone platforms i and j at time t, with conflict regions Cit and Cjt, I define the overlap area and the corresponding indicator:
$$S_{ij}^{t} = \operatorname{Area}\left(C_{i}^{t} \cap C_{j}^{t}\right)$$
$$I_{\text{overlap}}(i,j) = \begin{cases} 1, & S_{ij}^{t} > \varepsilon \\ 0, & S_{ij}^{t} \le \varepsilon \end{cases}$$
The tolerance ε removes boundary misclassification caused by floating-point error, so that a tangential configuration is not penalized. Because both operands are convex hexagons, the intersection area is obtained by half-plane clipping in a fixed number of operations. With six edges per polygon the per-pair cost is bounded by a small constant, and the total cost per step scales linearly in the number of neighbours of each UAV drone. This satisfies requirement R4 and keeps the ACR affordable inside a high-throughput training loop.
4. Behavioural Mechanisms of the Asymmetric Region
4.1 Head-On Encounter and Keep-Right Avoidance
In a head-on encounter the two UAV drone platforms approach with opposing headings. When both keep to their own right, the lateral offset that the manoeuvre consumes is governed by the compliant-side extensions of both regions. The no-overlap condition is therefore
$$d \;\ge\; \delta_{1}^{(i)} + \delta_{1}^{(j)} = 2\delta_{1} = 3R$$
Under the calibrated bias this condition holds exactly at the tangency point, so a rule-conforming pair can pass at the nominal avoidance distance without accumulating any penalty. When either platform instead keeps to its left, the manoeuvre brings the two non-compliant-side extensions into opposition, and the resulting overlap depth is
$$\Delta = \delta_{2}^{(i)} + \delta_{2}^{(j)} – d_{\text{avoid}} = 6R – 3R = 3R$$
The penalty signal therefore separates the two manoeuvres by three radii of overlap area at an identical separation distance. Because the penalty is accumulated at every control step, the policy receives a sustained gradient that points away from the rule-violating configuration, and keep-right avoidance emerges as the low-cost behaviour rather than as a hard-coded instruction.
4.2 Same-Direction Overtaking
When a faster UAV drone closes on a slower one with the same heading, the interaction is longitudinal before it becomes lateral. The rear vertices of the hexagon control how early the follower enters the region of the leader. Because the aft extremities are offset laterally relative to the heading axis, the two sides of the leader present different effective widths to a follower approaching with a small crossing angle. I evaluate the two mirror-image overtaking manoeuvres at a crossing angle of 15° and a lateral offset equal to the reference avoidance distance.
| Overtaking side | Crossing angle | Effective overlap width | Overlap state |
|---|---|---|---|
| Non-compliant side | 15° | ≈ 1.5R | Invaded |
| Compliant side | 15° | ≈ 0 | Tangent |
The non-compliant manoeuvre sweeps through a width of about 1.5R, whereas the compliant manoeuvre passes tangentially with an overlap width of essentially zero. The accumulated penalty difference over the duration of the encounter makes the compliant side strictly cheaper, and the policy converges to keep-right overtaking without any explicit side-selection logic.
4.3 Multi-Way Convergence and Circulation Direction
The most interesting case is the convergence of several UAV drone platforms onto a common node. Here the widest interior chord of the hexagon, which lies at a lateral offset h from the heading axis, becomes the first boundary to be touched during dense interaction. If I take the points along this chord as the origins of equal-radius circular detours, the two circulation directions are not equivalent. A vehicle circulating in the favoured direction travels one offset h below the limiting arc and therefore has more room to adjust, while a vehicle circulating in the opposite direction travels one offset h above the same arc and has less room. The radial clearance difference between the two trajectories is
$$2h = R$$
Because the clockwisely circulating vehicle retains a larger safety margin, the penalty signal is systematically lower along the clockwise family of trajectories. When several UAV drone platforms converge simultaneously, the population-level best response is therefore a common circulation direction rather than a set of mutually cancelling individual detours, and a clockwise circulation pattern forms spontaneously.
5. Markov Decision Process Formulation
I cast the cooperative path-planning problem as a Markov decision process for the joint system of N UAV drone platforms:
$$\mathcal{M} = \langle \mathcal{S}, \mathcal{A}, P, r, \gamma \rangle$$
Here S is the joint observation space, A is the joint action space, P is the unknown transition kernel, r is the reward function that mixes navigation efficiency, safety, and rule compliance, and γ ∈ (0,1) is the discount factor. Because the transitions depend on the coupled motion of all UAV drone platforms, the kernel is not available in closed form and must be sampled through interaction.
5.1 Observation Model
Each UAV drone i receives a hybrid observation that concatenates a structured feature vector with a local occupancy grid. The structured part contains the goal-relative state, the neighbour-relative state, the group motion state, and an activity flag for every platform. All relative angles are expressed in the body frame of the observing UAV drone, which keeps the observation invariant to the global orientation of the encounter.
$$\Delta\psi_{i}^{\text{goal}} = \operatorname{atan2}\!\left(g_{y}-y_{i},\; g_{x}-x_{i}\right) – \psi_{i}$$
$$d_{i}^{\text{goal}} = \left\| p_{i}^{\text{goal}} – p_{i} \right\|_{2}$$
$$\Delta\psi_{ij} = \operatorname{atan2}\!\left(y_{j}-y_{i},\; x_{j}-x_{i}\right) – \psi_{i}$$
$$d_{ij} = \left\| p_{i} – p_{j} \right\|_{2}$$
| Component | Meaning | Dimension |
|---|---|---|
| Δψgoal, dgoal | Goal-relative bearing and range | 2 |
| vj, ωj | Linear and angular speed of every platform | 2N |
| Δψij, dij | Neighbour-relative bearing and range | 2(N − 1) |
| activej | Activity flag of every platform | N |
| Gridmapi | Local occupancy grid, 11 × 11, flattened | 121 |
The occupancy grid is centred on the UAV drone and aligned with its current heading, with a resolution of 2 m per cell, covering a square of 22 m per side. Free space is encoded as 0, static obstacles as 1, and dynamic obstacles as 0.5. The grid supplies short-range geometric detail that the structured features cannot express, while the structured features provide the long-range relational context that the grid cannot cover at a reasonable resolution.
5.2 Action Model and Kinematics
The action of each UAV drone is a two-dimensional continuous vector that is rescaled onto the admissible speed and yaw-rate intervals:
$$v_{t} = v_{\min} + \frac{a_{1}+1}{2}\left(v_{\max}-v_{\min}\right)$$
$$\omega_{t} = \omega_{\min} + \frac{a_{2}+1}{2}\left(\omega_{\max}-\omega_{\min}\right)$$
with v ∈ [0, 15] m/s and ω ∈ [−1, 1] rad/s. The lower bound of the speed interval is set to zero so that a UAV drone may hover when it faces a severe conflict. The yaw-rate interval is symmetric about zero, which prevents the action space itself from injecting a directional preference into the learning process. Position and heading evolve according to the unicycle model over a control interval Δt:
$$x_{t+1} = x_{t} + v_{t}\cos\psi_{t}\,\Delta t, \qquad y_{t+1} = y_{t} + v_{t}\sin\psi_{t}\,\Delta t$$
$$\psi_{t+1} = \psi_{t} + \omega_{t}\,\Delta t$$
With Δt = 0.1 s, the maximum displacement per step is 1.5 m, which fixes the natural scale of the navigation reward introduced below.
6. Soft Actor-Critic Learning under Centralized Training and Decentralized Execution
6.1 Maximum-Entropy Objective
I use SAC because it optimizes a maximum-entropy objective, which stabilizes exploration in the presence of a multi-component reward and prevents premature collapse onto a single deterministic manoeuvre. The objective is
$$J(\pi) = \mathbb{E}_{(s_{t},a_{t})\sim\rho_{\pi}} \left[ \sum_{t=0}^{T} \gamma^{t} \Big( r(s_{t},a_{t}) + \alpha\,\mathcal{H}\big(\pi(\cdot \mid s_{t})\big) \Big) \right]$$
where ρπ is the trajectory distribution induced by the policy, r(st, at) is the immediate reward, and H denotes the differential entropy of the policy. The temperature α trades reward against entropy: a large α favours exploration, while α → 0 recovers conventional return maximization. Keeping a non-zero entropy floor matters here because a policy that commits too early to a particular side of the conflict region will not discover the directional bias encoded in the penalty.
6.2 Critic, Target, and Actor Updates
Two independently parameterized action-value networks mitigate overestimation, and the minimum of their outputs is used both in the target and in the policy gradient:
$$J_{Q}(\theta_{i}) = \mathbb{E}_{(s_{t},a_{t},r_{t},s_{t+1})\sim\mathcal{D}} \left[ \frac{1}{2}\Big( Q_{\theta_{i}}(s_{t},a_{t}) – y_{t} \Big)^{2} \right], \quad i \in \{1,2\}$$
$$y_{t} = r_{t} + \gamma \left[ \min_{j\in\{1,2\}} Q_{\bar{\theta}_{j}}(s_{t+1}, a_{t+1}) – \alpha \log \pi_{\phi}(a_{t+1}\mid s_{t+1}) \right]$$
$$a_{t+1} \sim \pi_{\phi}(\cdot \mid s_{t+1})$$
The entropy term inside the target acts as an implicit penalty on over-confident actions. If the policy becomes nearly deterministic at some state, the logarithmic term approaches zero, the target value falls, and the update pushes the policy back towards a controlled level of stochasticity. The target networks follow the main networks by Polyak averaging:
$$\bar{\theta} \leftarrow \tau\theta + (1-\tau)\bar{\theta}$$
The actor is reparameterized as a squashed Gaussian, which allows the policy gradient to propagate through the sampled action:
$$a_{t} = \tanh\!\big(\mu_{\phi}(s_{t}) + \sigma_{\phi}(s_{t}) \odot \xi\big), \qquad \xi \sim \mathcal{N}(0, I)$$
$$J_{\pi}(\phi) = \mathbb{E}_{s_{t}\sim\mathcal{D},\,\xi} \left[ \alpha \log \pi_{\phi}(a_{t}\mid s_{t}) – \min_{j\in\{1,2\}} Q_{\theta_{j}}(s_{t}, a_{t}) \right]$$
The temperature itself is learned by minimizing
$$J(\alpha) = \mathbb{E}_{a_{t}\sim\pi_{\phi}} \left[ -\alpha \Big( \log \pi_{\phi}(a_{t}\mid s_{t}) + \bar{\mathcal{H}} \Big) \right]$$
I set the target entropy to the negative of the action dimension, that is H̄ = −2. This keeps the policy stochastic along both the speed and the yaw-rate axes, which is important because the directional decision that the ACR is meant to induce is expressed in the yaw-rate channel.
| Hyperparameter | Symbol | Value |
|---|---|---|
| Learning rate | lr | 1 × 10−4 |
| Batch size | B | 4 096 |
| Discount factor | γ | 0.99 |
| Soft-update coefficient | τ | 0.01 |
| Target entropy | H̄ | −2 |
| Network width | — | 2 × 500 fully connected units |
| Control interval | Δt | 0.1 s |
6.3 Centralized Training with Decentralized Execution
I adopt a parameter-sharing CTDE architecture. During training, all N UAV drone platforms write their transition tuples into a single replay buffer and update one shared set of network parameters. At execution time, each UAV drone evaluates the shared policy on its own local observation and acts independently. Rule consistency is therefore a property of the shared parameters rather than of an explicit coordination message, and no central scheduler, global broadcast, or inter-vehicle negotiation is required at runtime. This is the property that makes the method attractive for low-altitude operations where link quality is intermittent: the coordination that enforces a consistent traffic order is paid for once, during training, and is thereafter encoded in the weights carried by every UAV drone.
7. Reward Design
The reward combines navigation efficiency, safety, and rule compliance into four additive components:
$$r_{t} = r_{\text{nav}} + r_{\text{time}} + r_{\text{norm}} + r_{\text{terminal}}$$
| Component | Symbol | Expression | Coefficient |
|---|---|---|---|
| Navigation reward | rnav | cfwd (dgoalt−1 − dgoalt) | cfwd = 5 |
| Time penalty | rtime | ctime | ctime = −2 |
| Rule penalty | rnorm | cnorm Σj≠i Ioverlap(i, j) | cnorm = −10 |
The navigation reward is a distance-difference term that rewards progress towards the goal. With a maximum speed of 15 m/s and a control interval of 0.1 s, the largest achievable per-step distance reduction is 1.5 m, corresponding to a maximum navigation reward of 7.5. The time penalty has a magnitude of 2, which is smaller than the navigation reward of an effective forward motion; it discourages loitering and pointless detours without reversing the goal-seeking tendency of the policy. The rule penalty has a magnitude of 10, slightly larger than the maximum per-step navigation reward. This choice is deliberate: a UAV drone can never compensate for a rule violation by taking a shorter path, because the penalty always exceeds the best possible gain from the shortcut. At the same time the penalty is not so large that it dominates the reward signal and drowns out the goal direction, which would delay the acquisition of basic navigation behaviour.
| Terminal event | Category | Reward |
|---|---|---|
| Goal reached | Success | 120 |
| Static obstacle collision | Static collision | −40 |
| Maximum step count reached | Timeout | −40 |
| Collision with another UAV drone | Dynamic collision | −15 |
| Prolonged stagnation | Anomaly | −10 |
| Accumulated yaw beyond 4π | Anomaly | −10 |
Once a terminal condition fires, the continuous accumulation stops and the terminal reward is delivered once. The large positive value for goal arrival and the clearly separated negative values for the different failure modes give the critic network enough resolution to distinguish near-miss outcomes from genuine failures.
8. Simulation Setup
I implemented the environment in PyTorch and trained on a single consumer-grade GPU. The airspace is a rectangle of 300 m × 150 m updated at a fixed step of 0.1 s; leaving the rectangle is treated as a static collision. The vehicle parameters follow a large multirotor cargo platform: maximum speed 15 m/s, maximum yaw rate 1.0 rad/s, and collision radius 1.5 m. I ran three independent random seeds and report means with standard deviations. Three conflict scenarios are constructed, each corresponding to a recognizable low-altitude operation pattern.
| Scenario | Operational analogue | Training episodes |
|---|---|---|
| Head-on encounter | Bidirectional corridor or opposing mission legs | 2 500 |
| Same-direction overtaking | Mixed-speed traffic on a shared route | 2 500 |
| Multi-way convergence | Route node, assembly area, or terminal merge point | 1 000 |
9. Results and Analysis
9.1 Convergence Behaviour
All three scenarios exhibit the same three-phase learning trajectory. In the early phase the policy has no usable state-action mapping, the UAV drone platforms hover near their start points, and the return, the conflict count, and the success rate are all low. In the middle phase the distance-difference reward drives the acquisition of goal-directed motion, the return rises steadily, and because avoidance has not yet been learned the trajectories cross frequently and the conflict count climbs to its peak. In the late phase the ACR penalty becomes the dominant term, the policy follows the direction indicated by the geometric asymmetry, and the conflict count decays to zero while the return saturates. The shaded bands across seeds narrow markedly towards the end of training, which indicates good reproducibility.
| Scenario | Peak conflict count | Peak episode | Convergence episode | Final return | Final conflict count | Final success rate |
|---|---|---|---|---|---|---|
| Head-on encounter | 291 | 23 | 857 | 3 429.60 ± 1.76 | 0.077 ± 0.080 | 100.00 ± 0.00 |
| Same-direction overtaking | 201 | 42 | 1 258 | 2 495.63 ± 10.04 | 0.075 ± 0.117 | 99.83 ± 0.22 |
| Multi-way convergence | 59 | 37 | 415 | 2 764.21 ± 6.97 | 0.008 ± 0.014 | 100.00 ± 0.00 |
The overtaking scenario shows a substantially higher peak conflict count than the other two. This is expected: the relative closing speed between two UAV drone platforms travelling in the same direction is small, so the encounter lasts longer and more intermediate configurations are visited before the manoeuvre resolves. The convergence scenario converges earliest. This is the clearest evidence that the geometric bias is doing real work: when six UAV drone platforms approach a common node, the asymmetry gives every one of them the same preferred circulation direction, so the group locks onto a single coordinated pattern very quickly instead of negotiating pairwise.
9.2 Trajectory Patterns and Rule Consistency
The converged trajectories reproduce the intended conventions in all three scenarios. In the head-on scenario two opposing groups deviate to their own right and complete the pass on the outer halves of the corridor. In the overtaking scenario the faster group shifts laterally and completes the pass on the compliant side before rejoining the route. In the convergence scenario all six UAV drone platforms begin to deviate towards the same side before reaching the central region, form a closed circulation loop, and then disperse towards their goals. I also evaluated four conflict-region variants to isolate the contribution of the asymmetry: no region at all, a circle, the proposed asymmetric region, and a mirrored region obtained by reflecting the proposed shape about the heading axis.
| Conflicts | Scenario | Rule | Compliance rate |
|---|---|---|---|
| Asymmetric region | Head-on encounter | Keep-right avoidance | 94.8% |
| Asymmetric region | Same-direction overtaking | Keep-right overtaking | 90.4% |
| Asymmetric region | Multi-way convergence | Clockwise circulation | 91.7% |
Both the no-region and the circular-region variants lack a directional bias, and the passing direction in the converged policy is effectively random. The proposed region and the mirrored region induce opposite directional trends, which confirms that the compliance behaviour is caused by the sign of the geometric bias and not by an incidental artifact of the training procedure. This is a strong control: if the behaviour were produced by some other property of the reward, flipping the geometry could not reverse the population-level direction.
9.3 Comparison with Alternative Methods
I compared the proposed method against a hybrid A*-artificial-potential-field planner, a single-agent deep reinforcement learning policy, a decentralized collision-avoidance policy, and an SAC ablation trained with the ACR penalty switched off. All learning methods were trained for 1 000 episodes with identical environments and seeds, and all methods were evaluated over 30 independently randomized episodes.
| Method | Conflicts | Return | Success rate | Rule compliance |
|---|---|---|---|---|
| A* + artificial potential field | 54.90 ± 14.86 | −479.77 ± 459.75 | 25.56 ± 9.52 | — |
| Deep reinforcement learning | 0.03 ± 0.18 | 1 603.33 ± 519.69 | 93.89 ± 10.25 | — |
| Decentralized collision avoidance | 0.77 ± 1.38 | 2 586.96 ± 99.35 | 98.33 ± 5.09 | 86.7% |
| SAC without the asymmetric region | 0.00 ± 0.00 | 2 760.33 ± 6.81 | 100.00 ± 0.00 | 36.7% |
| SAC with the asymmetric region | 0.00 ± 0.00 | 2 760.13 ± 6.57 | 100.00 ± 0.00 | 93.3% |
The classical planner produces the worst aggregate behaviour. Because its potential-field corrections have no notion of a shared right-of-way, converging vehicles push against each other and deadlock; across 30 evaluations there is no episode in which every UAV drone succeeds, and the compliance rate cannot be defined. The single-agent learning policy is safe in the majority of episodes but has no mechanism by which one UAV drone can take into account the passage direction of another, so its conflict count is nonzero and its success rate stays below 94%. The decentralized collision-avoidance policy performs better on safety, but its direction selection still fluctuates, and roughly one episode in seven ends in a direction-inconsistent pattern. The SAC ablation reaches perfect safety and perfect success, which tells me that the collision-avoidance objective alone already saturates the task. What it does not reach is order: with no directional term, the circulation direction is fixed by training randomness, and the compliance rate sits at 36.7%, which is close to chance for a two-way choice. Adding the asymmetric region to the same architecture raises compliance to 93.3% and costs 0.2 in mean return, which is within one standard deviation. In other words, the traffic order is obtained essentially for free in terms of task performance.
9.4 Sensitivity to the Bias Magnitude
I examined how the outcome depends on how strongly the geometry is biased, since this is the single most important design choice. If the bias is too small, the two passing directions produce nearly identical overlap states and the policy cannot distinguish them. If the bias is too large, the region occupies space that a normally separated UAV drone would legitimately use, and unnecessary penalties are injected into safe traffic.
| Bias ratio δ2/δ1 | Qualitative outcome | Observation |
|---|---|---|
| 1.0 (symmetric) | No direction learned | Compliance close to chance |
| 1.5 | Weak preference | Directional trend present but unstable across seeds |
| 2.0 (proposed) | Stable order | High compliance, negligible conflict, no return loss |
| 3.0 | Over-constraining | Detours on already separated traffic, return degradation |
The proposed ratio of two places the compliant configuration exactly on the tangency boundary of the reference avoidance distance, which I regard as the natural operating point: the region is wide enough to punish a violation at the nominal separation, and narrow enough to leave compliant traffic untouched.
10. Discussion
The mechanism that makes this construction work is worth stating plainly. A right-of-way clause, written as a discrete proposition, states which side of a vehicle another vehicle may occupy. When that proposition is converted into an inequality on the overlap area of two deliberately asymmetric convex regions, it becomes a piecewise-smooth function of relative position and heading. Its gradient with respect to the observation is well defined everywhere except on the boundary set of measure zero, and it is therefore directly usable by a policy-gradient method. The discrete clause has become a continuous force field whose direction can be steered by changing a handful of vertex coordinates, while its magnitude is controlled by a single reward coefficient.
The second property that matters for deployment is the cost structure. Because the rule consistency is baked into the shared policy parameters, the execution phase requires no message passing, no priority negotiation, and no central authority. Each UAV drone evaluates a small set of relative features and performs one forward pass. The ACR evaluation itself is a bounded sequence of half-plane clips over six-vertex polygons, so its cost is dominated by the neural network rather than by the geometry. This makes the method compatible with a fleet of UAV drone platforms operating over intermittent links, where any protocol that depends on continuous coordination would be fragile.
Several limitations should be stated. The directional biases are constants in the current implementation; a fleet with widely different platform footprints would need the bias magnitudes to adapt to the vehicle radius, which is straightforward because the geometry is already expressed in units of R, but a heterogeneous fleet with different manoeuvrability would require a per-class calibration. The study is restricted to a single altitude layer, and the extension to three dimensions is not automatic, because the vertical channel changes the nature of the encounter geometry. The reward weighting between navigation and rule compliance is fixed by design; I have argued for the specific ordering cnorm > cfwd > |ctime|, and although this ordering is robust across the scenarios I tested, an adaptive weighting scheme could improve behaviour in traffic densities well beyond those simulated here. Finally, the rule set I encode is limited to three canonical conventions; other conventions can in principle be encoded by re-orienting or re-shaping the region, but each new convention requires a fresh geometric analysis of the kind I present in Section 4.
11. Conclusion
I set out to reconcile two requirements that have largely been treated separately in cooperative path planning for UAV drone fleets: keeping a safe separation and maintaining an orderly traffic pattern. My answer is to move the rule from the logic layer into the geometry layer. By constructing a direction-biased hexagonal conflict region in the body frame of each UAV drone, and by calibrating its lateral, longitudinal, and mid-chord biases so that the compliant and non-compliant manoeuvres differ by a factor of two in overlap depth at the same separation distance, I obtain a continuous penalty signal that encodes keep-right avoidance, keep-right overtaking, and clockwise circulation without ever stating those rules explicitly. Optimizing the resulting Markov decision process with soft actor-critic under a parameter-sharing centralized-training-decentralized-execution architecture yields a shared policy that all UAV drone platforms execute independently from local observations.
Across three canonical conflict scenarios the trained policy reaches task success rates above 99.83% with average conflict counts indistinguishable from zero, and it reproduces the intended passing conventions with compliance rates between 90% and 95%. The ablation with the asymmetric term removed reaches identical safety and success metrics but collapses to roughly random passing directions, which isolates the geometric bias as the cause of the ordered behaviour rather than a by-product of the learning algorithm. Compared with a classical hybrid planner and with single-agent and decentralized learning baselines, the proposed method is the only one that achieves both zero conflicts and high directional consistency in multi-way convergence. The broader implication I draw is that the apparent incompatibility between discrete airspace rules and continuous learning-based control is an artifact of representation. Once a rule is expressed as an asymmetric geometric object, it becomes a smooth, differentiable, and cheaply evaluated component of the reward, and order can emerge from interaction rather than from authority.