UAV Path Planning and Computation Offloading via Deep Reinforcement Learning

In the era of 5G and beyond, the proliferation of mobile devices has led to an explosion of latency-sensitive applications such as intelligent transportation systems, augmented reality, and real-time traffic monitoring. These applications demand massive computational resources and low-latency responses, which often exceed the limited capabilities of user equipments (UEs). Mobile edge computing (MEC) has emerged as a promising architecture to overcome these limitations by bringing computation resources closer to the network edge. This paradigm shift effectively reduces transmission latency and alleviates the energy consumption issues inherent in traditional cloud computing. However, in the event of natural disasters or extreme traffic congestion, ground-based communication infrastructure may collapse, leaving UEs without network connectivity and computational services. This is where an unmanned aerial vehicle (UAV) can play a pivotal role. Deployed as a mobile aerial base station or a flying edge computing node, the unmanned aerial vehicle offers significant advantages in terms of flexibility, mobility, and the ability to establish line-of-sight (LoS) communication links, making it an ideal candidate for providing both communication coverage and computation offloading services in emergency scenarios.

The integration of an unmanned aerial vehicle into MEC systems, often referred to as UAV-assisted MEC, has drawn increasing attention from both academia and industry. The unique attributes of the unmanned aerial vehicle—including its rapid deployment and adaptive altitude adjustment—enable it to serve as a dynamic edge node, enhancing coverage, improving spectral efficiency, and facilitating task processing for ground users. Nevertheless, key challenges remain in optimizing the UAV’s three-dimensional trajectory while dynamically managing computation offloading decisions, all under the stringent resource constraints of the onboard battery. The interaction of these tasks gives rise to non-convex optimization problems that are computationally intractable with traditional approaches. Deep reinforcement learning (DRL) has proven effective in such settings, enabling the unmanned aerial vehicle to learn near-optimal policies through interaction with the environment, even in continuous action spaces.

In this research, I systematically investigate the problems of path planning and computation offloading for an unmanned aerial vehicle in emergency communication scenarios. My work is built upon a comprehensive framework that integrates both communication and computation concerns. First, I examine the path planning challenge in a traffic monitoring context, where the unmanned aerial vehicle is used to collect periodic data from widely distributed sensor nodes. I propose a DRL-based scheduling and navigation algorithm that leverages the Twin Delayed Deep Deterministic Policy Gradient (TD3) method to optimize the flight route of the unmanned aerial vehicle, ensuring timely data collection. Second, I extend the analysis to a more general MEC network where the unmanned aerial vehicle acts as an edge server, providing computation offloading services to multiple ground users. I formulate a joint optimization problem of task offloading and trajectory planning, aiming at minimizing the total system energy consumption and transmission delay. To tackle the non-convex optimization, I develop a hybrid algorithm called PDPSO-TD3, which combines a diversity-enhanced particle swarm optimizer for offloading decision-making with a TD3-based controller for the unmanned aerial vehicle’s flight trajectory optimization. Extensive simulations demonstrate that my proposed schemes achieve notable performance gains over existing benchmark algorithms, underlining the significant potential of the unmanned aerial vehicle in resilient and efficient edge computing systems.

System Model and Problem Formulation

I consider a communication-denied zone where a single unmanned aerial vehicle is dispatched to provide temporary communication and computation services. The area of interest is defined as a three-dimensional region, with ground UEs randomly distributed. The set of UEs is denoted by \(\mathcal{K} = \{1, 2, …, K\}\). Each UE generates a computation-intensive task \(L_k(t)\) at time slot \(t\). The horizontal positions of the UEs and the unmanned aerial vehicle are written as \(S_k(t) = (x_k(t), y_k(t), 0)\) and \(S_{uav}(t) = (X(t), Y(t), H)\), respectively, with a fixed operating altitude \(H\) for the unmanned aerial vehicle. The time horizon is divided into \(T\) equal-length slots, each of duration \(\chi\). At each slot, the unmanned aerial vehicle adjusts its heading and speed, denoted as \(o_t\) and \(V_t\), to navigate optimally among the users. The dynamics of the unmanned aerial vehicle’s movement are given by the following update rules in continuous space:

$$
\begin{aligned}
X(t+1) &= X(t) + V_t \chi \sin(o_t), \\
Y(t+1) &= Y(t) + V_t \chi \cos(o_t),
\end{aligned}
$$

with the velocity constraint \(V_t \leq V_{max}\) ensuring physical feasibility. This formulation reflects the realistic operating conditions of an unmanned aerial vehicle, which must balance travel time and energy consumption while moving toward optimal service positions.

Communication and Offloading Model

To support the offloading of computation tasks, the uplink transmission from the UEs to the unmanned aerial vehicle plays a critical role. The channel gain between a UE \(k\) and the unmanned aerial vehicle can be modeled using a free-space path-loss model, considering LoS propagation as a dominant characteristic of the air-to-ground link. Let \(\beta_0\) denote the channel power gain at a reference distance of 1 meter, and \(\delta_0^2\) denote the noise power at the unmanned aerial vehicle receiver. The channel gain \(h_k(t)\) and the achievable uplink data rate \(r_k(t)\) are then expressed as:

$$
h_k(t) = \frac{\beta_0}{H^2 + \|S_{uav}(t) – S_k(t)\|^2},
$$

$$
r_k(t) = \frac{B_u}{K} \log_2\left(1 + \frac{P_{user} h_k(t)}{\delta_0^2}\right),
$$

where \(B_u\) is the total uplink bandwidth and \(P_{user}\) is the transmit power of each UE. Since the unmanned aerial vehicle communicates with only one UE per slot under my assumption, the bandwidth is equally divided among users to ensure fairness. To ensure reliable communication, a threshold on the minimum data rate is set, which directly influences whether an effective communication link \(I_k(t)\) can be established between the UE \(k\) and the unmanned aerial vehicle at time slot \(t\). A binary offloading strategy is adopted, where \(\phi_k(t) = 1\) indicates that the UE \(k\) offloads a portion of its task to the unmanned aerial vehicle, whereas \(\phi_k(t) = 0\) means that all computations are performed locally by the UE. When a task is partially offloaded, the UE transmits a fraction \(\alpha_k(t)\) of its total task data \(L_k(t)\) to the unmanned aerial vehicle for remote execution. The transmission delay \(T_k^{Tra}(t)\) is given by:

$$
T_k^{Tra}(t) = \frac{\alpha_k(t) L_k(t) \phi_k(t)}{r_k(t)}.
$$

Accordingly, the communication energy consumption \(E_k^{Tra}(t)\) incurred during this offloading process is computed as:

$$
E_k^{Tra}(t) = P_{uav} \cdot T_k^{Tra}(t),
$$

where \(P_{uav}\) denotes the transmit power of the unmanned aerial vehicle. It should be emphasized that transmission delay and energy consumption are tightly coupled, as longer transmission times inevitably lead to elevated energy expenditure, both for the UE and the unmanned aerial vehicle.

Computation Model and Energy Consumption

Each UE is equipped with a local CPU, characterized by its processing capability \(f_{user}\) (in cycles per second) and the number of CPU cycles required to process one bit of data \(C_{user}\). Similarly, the unmanned aerial vehicle has a more powerful computing capability \(f_{uav}\) and a higher processing cost per bit \(C_{uav}\). The local computation delay for a UE that processes all tasks locally or handles the remaining tasks after partial offloading is expressed as follows. For the all-local case,

$$
T_{k,loc}^{Com}(t) = \frac{(1 – \phi_k(t)) L_k(t) C_{user}}{f_{user}},
$$

and for the partial local computation case,

$$
T_{k,par}^{Com}(t) = \frac{(1 – \alpha_k(t)) (1 – \phi_k(t)) L_k(t) C_{user}}{f_{user}}.
$$

When a portion of the task is offloaded to the unmanned aerial vehicle, the processing delay at the UAV side is:

$$
T_{k,uav}^{Com}(t) = \frac{\alpha_k(t) \phi_k(t) L_k(t) C_{uav}}{f_{uav}}.
$$

The corresponding energy consumptions are derived by incorporating the effective switched capacitance factors \(\kappa_{user}\) and \(\kappa_{uav}\) for the UE and the unmanned aerial vehicle, respectively:

$$
E_{k,loc}^{Com}(t) = \kappa_{user} f_{user}^2 (1 – \phi_k(t)) L_k(t) C_{user},
$$

$$
E_{k,par}^{Com}(t) = \kappa_{user} f_{user}^2 (1 – \alpha_k(t)) (1 – \phi_k(t)) L_k(t) C_{user},
$$

$$
E_{k,uav}^{Com}(t) = \kappa_{uav} f_{uav}^2 \alpha_k(t) \phi_k(t) L_k(t) C_{uav}.
$$

For the partial offloading mode, the combined delay perceived by a user involves both the computation time on the unmanned aerial vehicle and the local processing time, since the two paths execute in parallel. Consequently, the total delay and energy consumption for the partial offloading mode are given by:

$$
T_{par}(t) = \max\left( T_{k,par}^{Com}(t), T_{k,uav}^{Com}(t) \right) + T_k^{Tra}(t),
$$

$$
E_{par}(t) = E_k^{Tra}(t) + E_{k,par}^{Com}(t) + E_{k,uav}^{Com}(t).
$$

Finally, the cumulative delay and energy consumption across all UEs in the MEC system are:

$$
T(t) = \sum_{k=1}^{K} \left[ (1 – \phi_k(t)) T_{k,loc}^{Com}(t) + \phi_k(t) T_{par}(t) \right],
$$

$$
E(t) = \sum_{k=1}^{K} \left[ (1 – \phi_k(t)) E_{k,loc}^{Com}(t) + \phi_k(t) E_{par}(t) \right].
$$

UAV Flight Energy Model

Beyond communication and computation energy, the hovering and translational motion of the unmanned aerial vehicle also constitute substantial components of the total system energy budget, which is often neglected in prior work. Modeling the energy consumption of the unmanned aerial vehicle is of paramount importance, especially because the unmanned aerial vehicle has a limited onboard battery capacity. The propulsion power of a rotary-wing unmanned aerial vehicle at a constant speed \(V\) can be expressed as:

$$
P(V) = P_0 \left(1 + \frac{3V^2}{U_{tip}^2}\right) + P_i \left( \sqrt{1 + \frac{V^4}{4V_0^4}} – \frac{V^2}{2V_0^2} \right)^{1/2} + \frac{1}{2} d_0 \rho s A V^3,
$$

where \(P_0\) and \(P_i\) are the blade profile power and induced power during hover, \(U_{tip}\) the rotor blade tip speed, \(V_0\) the mean rotor induced velocity, \(d_0\) the fuselage drag ratio, \(\rho\) the air density, \(s\) the rotor solidity, and \(A\) the rotor disc area. Using the average speed \(V_t\) at each time slot, the energy consumed by the unmanned aerial vehicle in slot \(t\) is \(E_{uav}(t) = P(V_t) \chi\). The total flight energy consumption of the unmanned aerial vehicle throughout its mission is then

$$
E_{uav}^{total} = \sum_{t=1}^{T} E_{uav}(t).
$$

This modeling provides a bridge between the dynamics of the unmanned aerial vehicle and the energy efficiency of the overall MEC system, which is essential for realistic trajectory optimization.

Optimization Problem Formulation

My research objective is to jointly minimize the weighted sum of communication/computation energy and the propulsion energy of the unmanned aerial vehicle, together with the system delay. The overall optimization problem, summarizing all the aforementioned models, is presented as follows. It accounts for two key aspects of the system design: task offloading decision \(\phi_k(t)\) and the trajectory of the unmanned aerial vehicle characterized by its direction \(o_t\) and speed \(V_t\).

$$
\begin{aligned}
P1: \quad & \min_{\{S_{uav}(t)\}, \{\phi_k(t)\}, \{\alpha_k(t)\}} \quad \omega \sum_{t=1}^{T} \left[ E(t) + (1 – \omega) T(t) \right] + \vartheta \sum_{t=1}^{T} E_{uav}(t) \\
\text{s.t.} \quad & C1: S_{uav}(t) \in [0, X_{size}] \times [0, Y_{size}] \times \{H\}, \forall t, \\
& C2: \phi_k(t) \in \{0, 1\}, \forall k, t, \\
& C3: \alpha_k(t) \in [0, 1], \forall k, t, \\
& C4: \|S_{uav}(t+1) – S_{uav}(t)\| \leq V_{max} \chi, \forall t, \\
& C5: I_k(t) = 1, \forall k, t, \\
& C6: V_t \leq V_{max}, \forall t.
\end{aligned}
$$

Here, \(\omega\) and \(\vartheta\) are weighting factors to adjust the relative importance of the energy-delay cost and the unmanned aerial vehicle’s flight energy. Constraint (C1) confines the unmanned aerial vehicle’s movement within the designated service area. Constraints (C2) and (C3) impose the binary offloading decision and the allowed range of the offloading ratio, respectively. Constraint (C4) establishes the mobility relationship between consecutive time slots of the unmanned aerial vehicle, while (C5) guarantees communication reliability, ensuring the unmanned aerial vehicle remains connected with the target UE at each time slot. Finally, (C6) enforces the speed limit of the unmanned aerial vehicle.

The above joint optimization is a mixed-integer non-convex problem, which is intrinsically challenging to solve with conventional optimization tools due to its combinatorial nature and the tightly coupled variables. This motivates the use of DRL, particularly the TD3 algorithm, which excels in continuous control tasks and can handle the uncertainties of the environment by learning an end-to-end mapping from system states to actions of the unmanned aerial vehicle.

DRL-Based Path Planning for Data Collection

To address the data collection challenge for the unmanned aerial vehicle in a traffic monitoring context, I model the problem as a Markov Decision Process (MDP). In this environment, the state space includes the current position of the unmanned aerial vehicle, the distances to all sensor nodes, and the time remaining until data generation at each node. This rich set of environment observations allows the unmanned aerial vehicle to make informed scheduling decisions without requiring centralized coordination among the distributed sensor nodes. The action space is defined by the desired heading angle of the unmanned aerial vehicle, which is normalized to a continuous range of \([-1, 1]\), corresponding to a full rotation of \(0\) to \(2\pi\) radians. The reward function is carefully crafted to guide the training process of the unmanned aerial vehicle toward optimal performance. Positive rewards are conferred when the unmanned aerial vehicle successfully collects data from a node. Penalties are incurred when the unmanned aerial vehicle moves out of the predefined boundary or fails to collect data within the expiration time.

For this setup, the TD3 algorithm provides a robust learning framework. As an actor-critic method, TD3 uses a pair of critic networks to mitigate the overestimation bias that commonly plagues standard DDPG, thereby stabilizing the training of the unmanned aerial vehicle. Algorithm 1 presents the pseudocode for my path-planning solution based on TD3.

Algorithm 1: TD3-Based UAV Path Planning

Input: Distances to sensor nodes; time-to-data-generation for each node

Output: Optimal flight path and control actions for the unmanned aerial vehicle

  1. Initialize critic networks \(Q_{\theta_1}\), \(Q_{\theta_2}\), and actor network \(\pi_\phi\) with random parameters; initialize target networks with the same weights.
  2. Initialize the experience replay buffer \(\mathcal{B}\).
  3. For each training episode do:
  4. a. Reset the environment and obtain the initial state \(s_1\) of the unmanned aerial vehicle.
  5. b. For each time step \(t\):
  6. i. Select an action based on the current policy and exploration noise.
  7. ii. Execute the action, observe the immediate reward \(r_t\) and the next state \(s_{t+1}\).
  8. iii. Store the transition \((s_t, a_t, r_t, s_{t+1}, done)\) in \(\mathcal{B}\).
  9. iv. If the unmanned aerial vehicle collects a data packet, accumulate the reward.
  10. v. If the buffer is sufficiently large, sample a mini-batch of transitions.
  11. vi. Compute the target Q-value using the target networks with clipped noise addition.
  12. vii. Update the critic networks by minimizing the Mean Squared Error (MSE) loss.
  13. viii. Every \(d\) iterations, update the actor network using the deterministic policy gradient.
  14. ix. Soft-update the target networks of the critic and the actor.

To further improve the timeliness of the data-gathering approach, I introduce the concept of predictive waiting. Instead of having the unmanned aerial vehicle visit sensor nodes blindly, my model predicts the exact time slot at which each node’s next data packet will be generated. This knowledge allows the unmanned aerial vehicle to accelerate its transit to the geographic neighborhood of the upcoming data generator, wait in a controlled loiter pattern, and then collect the fresh data the moment it becomes available, thereby minimizing idle waiting time and avoiding stale data at the collection point. The predictive mechanism is tightly integrated into the state design for the TD3 agent. In the simulation, the unmanned aerial vehicle successfully learns to balance between visiting high-priority nodes and traveling to distant nodes early enough for predictive waiting. The reward curves demonstrate clear convergence, with the unmanned aerial vehicle attaining a stable average reward after around 700 training episodes.

PDPSO-TD3 for Joint Computation Offloading and Trajectory Optimization

For the extended scenario where the unmanned aerial vehicle serves as a mobile edge computing server, I propose the PDPSO-TD3 algorithm. This hybrid algorithm is designed to handle the joint complexity of binary task offloading decisions and the continuous trajectory optimization of the unmanned aerial vehicle. The overall architecture is shown in the figure above, highlighting the interplay between the two optimization modules. The responsibilities of the algorithm are divided into two functional stages: the first stage uses a particle swarm optimizer to generate an optimal offloading ratio, and the second stage relies on the TD3 agent to refine the unmanned aerial vehicle’s trajectory in real time.

Stage 1: PDPSO for Optimal Offloading Decision

In the first stage, I use an enhanced particle swarm optimization algorithm that considers the population diversity of the particles to adaptively adjust the inertia weight. Each particle in the swarm represents a candidate offloading policy for the UEs. The velocity \(v_i\) and position \(x_i\) of the \(i\)-th particle are updated iteratively according to the following rules, incorporating its own best position \(p_{best}\) and the global best position \(g_{best}\):

$$
v_i^{(t+1)} = \varpi v_i^{(t)} + c_1 r_1 (p_{best} – x_i^{(t)}) + c_2 r_2 (g_{best} – x_i^{(t)}),
$$

$$
x_i^{(t+1)} = x_i^{(t)} + v_i^{(t+1)}.
$$

The influence of the adaptive inertia weight \(\varpi\) is pivotal in balancing the global exploration and local exploitation of the particle swarm. By monitoring the diversity index \(D\) of the population, the inertia weight is adjusted dynamically, which helps the swarm escape from premature convergence. The diversity index is computed as follows:

$$
D(t+1) = \frac{1}{K – 1} \sum_{i=1}^{K} \sqrt{\left( d_i^{Ave}(t) – d_i^{Min}(t) \right)^2},
$$

where \(d_i^{Ave}\) and \(d_i^{Min}\) are the average and minimum distances between the \(i\)-th particle and its neighboring particles, respectively. This diversity-aware mechanism enhances the global search capability of the optimizer, which is critical for finding the optimal offloading ratio affecting the energy-delay trade-off in the highly dynamic network. Once the optimal offloading policy \(\phi_i(t)\) is determined and the offloading proportion is extracted, it feeds directly into the TD3 interaction loop.

Stage 2: TD3 for UAV Trajectory Optimization

The task offloading strategy obtained from the first stage is fed into the TD3 module, which then optimizes the trajectory of the unmanned aerial vehicle accordingly. The state space for the TD3 agent at time slot \(t\) is defined as the position of the unmanned aerial vehicle, the positions of all the UEs, and the current offloading decisions of the UEs:

$$
State(t) = \big\{ S_{uav}(t), S_1(t), …, S_K(t), \phi_1(t), …, \phi_K(t) \big\}.
$$

The action space is composed of the heading angle \(o_t\) and the speed \(v_t\) of the unmanned aerial vehicle, giving the agent continuous control over its movement. The reward function for the TD3 agent is designed to align the objectives of energy efficiency and delay reduction. Concretely, the agent receives a reward \(R_{step}\) that is the negative of the instantaneous system energy consumption and delay, plus a penalty term for moving outside the predefined boundary. This drives the unmanned aerial vehicle to dynamically adjust its trajectory toward areas with higher user density and more favorable channel conditions, while keeping its energy consumption low. A critical enhancement in my algorithm is the inclusion of the flight energy model into the reward computation, which encourages the unmanned aerial vehicle to avoid unnecessary movement and hover when appropriate, thus significantly prolonging the operational lifetime of the unmanned aerial vehicle.

The integration of PDPSO and TD3 is the key novelty of my approach. The deterministic offloading decisions from the first stage provide a stable environment for the TD3 agent, enabling it to focus primarily on the continuous control problem without being overwhelmed by the combinatorial aspects of task assignment. Conversely, the TD3 agent’s trajectory updates alter the channel conditions and thereby influence the offloading decision landscape, which is continuously re-optimized by the PDPSO module. This two-way coupling forms a powerful optimization loop, as illustrated in the system framework diagram.

Algorithm Complexity Analysis

I provide a comprehensive analysis of the computational complexity of my proposed PDPSO-TD3 algorithm. The time complexity of the PDPSO part, which addresses the discrete task-offloading decisions, is primarily determined by the number of particles and the number of iterations of the swarm. Let \(K\) be the number of UEs (particles) and \(I\) the number of iterations of PSPSO. The complexity of the PDPSO module is \(O(I \cdot K)\). The TD3 component, which is responsible for optimizing the continuous flight trajectory of the unmanned aerial vehicle over \(T\) time slots, interacts with a neural network that has constant depth and width in terms of parameter count. This results in a complexity of \(O(I \cdot K \cdot T)\). When considering the full training process consisting of \(M\) episodes, the total time complexity of the algorithm is approximately \(O(M \cdot T \cdot I \cdot K)\).

The time complexity of my algorithm is of the same order of magnitude as many existing DRL-based approaches, such as DDPG and PPO, but with a better convergence behavior due to the effectiveness of the hybrid structure. More importantly, the space complexity of the algorithm is constant, as it only depends on the neural network architecture, not on the problem size. This guarantees that my algorithm is highly scalable and can be used in real-time applications where both the UEs and the unmanned aerial vehicle are moving.

Simulation Results and Discussion

I conduct a series of comprehensive simulations to validate the effectiveness of the proposed PDPSO-TD3 algorithm. The simulation environment is set up to mimic a realistic post-disaster scenario: a \(1000 \times 1000\) m² area with a communication outage, where a single unmanned aerial vehicle is deployed to provide edge computing services. The unmanned aerial vehicle is assumed to fly at a fixed altitude of 250 m, which is the optimal trade-off between coverage and signal attenuation. The key simulation parameters are summarized in the following table.

Parameter Symbol Value
Number of UEs \(K\) 10 – 80
Task size of UEs \(L_k(t)\) [1, 10] Mbits
UAV altitude \(H\) 250 m
Area dimension \(X_{size} \times Y_{size}\) 1000 × 1000 m²
Channel gain at 1 m \(\beta_0\) -50 dB
Uplink bandwidth \(B_u\) 100 MHz
Transmission power \(P_{user}\) 0.5 W
UE local CPU resource \(f_{user}\) 1 GHz
UAV CPU resource \(f_{uav}\) 3 GHz
Max UAV speed \(V_{max}\) 15 m/s
Learning rate \(lr\) 0.0003
Discount factor \(\gamma\) 0.99
Training episodes \(M\) 6000

I compare my algorithm against several baselines, including DQN+PDPSO, DDPG+PDPSO, and PPO. Figure 4-6 showcases the performance of these methods across four key indicators: average transmission delay, energy loss, unmanned aerial vehicle flight energy, and user access rate. The numerical comparison is provided in the table below, where it is clear that my approach is superior on all metrics.

Performance Metric DQN+PDPSO DDPG+PDPSO PPO PDPSO-TD3 (Ours)
Average Transmission Delay 1.23 s 0.91 s 1.31 s 0.73 s
Average Energy Loss 5061 W 4342 W 5954 W 3906 W
Average UAV Flight Energy 5261 W 4210 W 5754 W 3955 W
UEs Access Rate 0.74 0.92 0.69 0.95

The improved performance of the unmanned aerial vehicle with my PDPSO-TD3 algorithm can be attributed to several factors. First, the adaptive inertia weight mechanism of the particle swarm optimizer helps to avoid local optima in the offloading decision space, thereby finding more energy-efficient task assignments. Second, the double critic network design and delayed policy updates in TD3 significantly reduce the overestimation issue, leading to more stable and accurate value estimates, which in turn guides the unmanned aerial vehicle to explore a broader set of trajectories and converge to the optimal policy faster. Third, the integration of flight energy into the reward function provides the agent with a clear and immediate negative feedback whenever the unmanned aerial vehicle strays from the energy-efficient path, prompting rapid corrections.

Parameter Sensitivity Analysis

To further analyze the robustness of my algorithm, I evaluate the influence of key hyperparameters on its performance. Figure 4-7 compares the training reward curves under different combinations of learning rate and discount factor. The results indicate that my algorithm achieves stable convergence with a learning rate of \(3e-4\) and a discount factor of \(0.99\). Using a much larger learning rate, \(1e-2\), leads to early convergence to a suboptimal solution, which demonstrates the importance of choosing a moderate learning rate for the unmanned aerial vehicle’s training. Similarly, Figure 4-8 shows the impact of varying batch sizes. A batch size of 128 offers the best performance in terms of both convergence speed and final reward value, compared to smaller batch sizes (which introduce high variance and instability) and larger batch sizes (which reduce the exploration capabilities of the agent).

Convergence and Scalability Analysis

Figure 4-9 illustrates the convergence curves of the discussed algorithms over 6000 training episodes. While all algorithms show a downward trend in system cost, PDPSO-TD3 achieves the fastest convergence and the lowest stable cost value. Notably, the PPO algorithm exhibits significant variance after the 3000-th episode, which suggests that its clipped surrogate objective might lead to oscillations in such a high-dimensional continuous control problem. In contrast, the TD3-based methods, including my proposed approach, show smoother convergence, highlighting the benefits of the deterministic policy with target policy smoothing. The user access rate of my algorithm remains around 95% even when the number of UEs increases to 80, confirming its highly scalable nature. This is especially important for practical deployment of the unmanned aerial vehicle in dense urban areas, where the number of users demanding connectivity can be quite large.

Trajectory Visualization

To provide an intuitive understanding of the optimized flight route, Figure 4-14 depicts the simulated flight path of the unmanned aerial vehicle from its start point at \((0, 0, 250)\) meters to the target endpoint. The green line indicates the route chosen by the TD3 controller. It can be seen that the unmanned aerial vehicle slightly adjusts its path to approach the locations where the UEs with partial offloading tasks are concentrated, reducing the transmission distance and improving the channel quality in a dynamic manner. This trajectory demonstrates that the unmanned aerial vehicle effectively trades off between traveling quickly to its final destination and lingering near high-priority users to provide them with more communication resources. The flight path is smooth and obstacle-free, confirming the practical feasibility of the generated actions in the real-world operation of the unmanned aerial vehicle.

Practical Deployment Discussion

To translate my research findings into real-world practice, I investigate the deployment of the proposed framework in a realistic traffic monitoring system. The primary application scenario involves aerial support in a multi-cell road network where some base stations have malfunctioned or are experiencing congestion. In such situations, the unmanned aerial vehicle can be quickly dispatched to the affected area to restore temporary connectivity. However, the onboard battery of a commercial unmanned aerial vehicle typically supports only about 20–30 minutes of flight time. During this period, training the DRL agent from scratch is not feasible given the environmental uncertainty and the complexity of the task.

To address this challenge, I propose using a digital twin system as a training ground for the unmanned aerial vehicle’s control model. The digital twin continuously collects data from traffic sensors, weather stations, and live road cameras, recreating the real environment in a virtual space. The PDPSO-TD3 algorithm is pre-trained in this simulated scenario, enabling the unmanned aerial vehicle to learn the approximate optimal policy offline before its mission. Upon deployment, the pre-trained model can be fine-tuned with real-time data, speeding up the convergence process substantially and ensuring that the unmanned aerial vehicle is ready to provide communication services immediately upon arrival at the target zone. This workflow significantly improves the practicality of the unmanned aerial vehicle-assisted MEC system, making it a viable solution for emergency and public safety networks.

Conclusion and Future Work

In this paper, I have presented a comprehensive investigation of path planning and computation offloading for an unmanned aerial vehicle-assisted MEC system. The key contributions of my work are threefold: (i) I constructed an energy-aware system model that jointly considers communication, computation, and propulsion energy consumption of the unmanned aerial vehicle; (ii) I developed a predictor-based data collection scheme that enables the unmanned aerial vehicle to pre-schedule its route and wait for the data, thereby ensuring fresh data collection; and (iii) I proposed a hybrid PDPSO-TD3 algorithm to solve the non-convex optimization of the joint offloading decision and trajectory of the unmanned aerial vehicle. Extensive simulation results demonstrate that my algorithm outperforms several existing DRL baselines, achieving improvements of 10.1%, 14.3%, 6.1%, and 3.3% in terms of system energy consumption, transmission delay, UAV flight energy, and user access rate, respectively.

Looking forward, there are several promising directions for extending my work. First, the current model considers a single unmanned aerial vehicle. Deploying a swarm of UAVs to collaboratively cover a large area is an interesting and challenging extension, which involves joint trajectory design, inter-UAV coordination, and task partitioning. Second, the incorporation of energy harvesting technologies, such as solar-powered unmanned aerial vehicle platforms, could further increase the endurance and sustainability of the system. Third, the integration of emerging 6G communication technologies may open up new opportunities for ultra-reliable, low-latency aerial edge computing. Finally, I believe that the digital twin-based training methodology will play a critical role in the practical deployment of unmanned aerial vehicle networks, and I intend to further develop this approach toward more robust and adaptive control policies that can handle a wide range of unforeseen operational conditions. The findings of this study firmly establish the unmanned aerial vehicle as a key enabler of resilient, intelligent, and energy-efficient edge computing in future wireless networks.

Scroll to Top