In recent years, the rapid advancement of intelligent unmanned systems has revolutionized various high-risk and complex operational scenarios, including industrial inspection, emergency rescue, and military reconnaissance. Among these, the quadrotor drone, with its autonomous target search capabilities, has demonstrated remarkable performance in fields such as aerial remote sensing and disaster detection. However, the inherent limitations of existing flight control systems in adapting to dynamic and uncertain external environments remain a critical bottleneck. Traditional control methods, such as PID, LQR, and MPC, often rely on precise system models or extensive parameter tuning, which hampers robustness and generalization in real-world applications. To address these challenges, I explore the potential of deep reinforcement learning (DRL) as a model-free optimization paradigm, which enables autonomous exploration and adaptation through interaction with the environment. In this paper, I propose an improved algorithm named KPPO, which integrates a composite regularization mechanism—combining a threshold-triggered KL divergence penalty and an L2 regularization term—into the proximal policy optimization (PPO) framework. This approach aims to enhance the convergence speed and environmental adaptability of quadrotor drone flight control. Through extensive physical simulations across multiple tasks, I validate the effectiveness of the KPPO algorithm, demonstrating its superior performance compared to baseline PPO and DDPG algorithms in terms of convergence stability and task completion efficiency.
The quadrotor drone operates by adjusting the rotational speeds of its four rotors to control flight attitude and position. Its flight principles are based on Newton-Euler equations, where lift and torque generated by the rotors enable attitude adjustment and movement. Each rotor produces an upward lift force given by \( F_i = C_T \cdot \omega_i^2 \), where \( C_T \) is the lift coefficient and \( \omega_i \) is the angular velocity. The total lift force is the sum of all rotor forces: \( F = F_1 + F_2 + F_3 + F_4 \). For hover, this total lift must balance the gravitational force: \( F = mg \), where \( m \) is the mass and \( g \) is gravitational acceleration. Attitude control—pitch, roll, and yaw—is achieved by differential rotor speeds. For instance, pitch control involves increasing front rotor speeds and decreasing rear rotor speeds, while roll control adjusts left and right rotor speeds, and yaw control manipulates diagonal rotor pairs. The dynamics and kinematics of the quadrotor drone are described by the following equations. The position dynamics are:
$$
\begin{align*}
a_x &= -\frac{F}{m} (\cos\psi \sin\theta \cos\phi + \sin\psi \sin\phi) \\
a_y &= -\frac{F}{m} (\sin\psi \sin\theta \cos\phi – \cos\psi \sin\phi) \\
a_z &= g – \frac{F}{m} \cos\phi \cos\theta
\end{align*}
$$
Here, \( \phi \), \( \theta \), and \( \psi \) represent roll, pitch, and yaw angles, respectively, and \( a_x \), \( a_y \), \( a_z \) are accelerations in the x, y, z directions. The kinematic model for attitude, under small-angle perturbations, is:
$$
\begin{align*}
\dot{\phi} &= \frac{1}{I_{xx}} (\tau_x + qr (I_{yy} – I_{zz}) – J_1 q \Omega) \\
\dot{\theta} &= \frac{1}{I_{yy}} (\tau_y + pr (I_{zz} – I_{xx}) + J_1 p \Omega) \\
\dot{\psi} &= \frac{1}{I_{zz}} (\tau_z + qp (I_{xx} – I_{yy}))
\end{align*}
$$
In these equations, \( I_{xx} \), \( I_{yy} \), \( I_{zz} \) are moments of inertia, \( \tau_x \), \( \tau_y \), \( \tau_z \) are external torques, \( p \), \( q \), \( r \) are angular velocities in the body frame, \( J_1 \) is rotor inertia, and \( \Omega \) is rotor angular velocity. The control system of a quadrotor drone typically includes sensors (e.g., IMU, barometer, GPS), a control unit running algorithms like KPPO, and actuators (motors and drivers). The motor speeds are computed as:
$$
\begin{align*}
\text{Motor}_{\text{right1}} &= \text{Thrust}_{\text{cmd}} + \text{Yaw}_{\text{cmd}} + \text{Pitch}_{\text{cmd}} + \text{Roll}_{\text{cmd}} \\
\text{Motor}_{\text{left1}} &= \text{Thrust}_{\text{cmd}} – \text{Yaw}_{\text{cmd}} + \text{Pitch}_{\text{cmd}} – \text{Roll}_{\text{cmd}} \\
\text{Motor}_{\text{right2}} &= \text{Thrust}_{\text{cmd}} – \text{Yaw}_{\text{cmd}} – \text{Pitch}_{\text{cmd}} + \text{Roll}_{\text{cmd}} \\
\text{Motor}_{\text{left2}} &= \text{Thrust}_{\text{cmd}} + \text{Yaw}_{\text{cmd}} – \text{Pitch}_{\text{cmd}} – \text{Roll}_{\text{cmd}}
\end{align*}
$$
Deep reinforcement learning combines the representational power of deep neural networks with reinforcement learning to handle high-dimensional state and action spaces. In DRL, an agent learns optimal policies by interacting with an environment to maximize cumulative reward. For quadrotor drone control, DRL algorithms can be categorized into value-based and policy-based methods. Among policy-based algorithms, proximal policy optimization (PPO) and deep deterministic policy gradient (DDPG) are widely used. PPO employs a clipped objective function to ensure stable policy updates, while DDPG uses deterministic policy gradients with actor-critic networks and experience replay. However, PPO can suffer from slow convergence and difficulty in adapting to complex environments, which motivates the development of improved algorithms like KPPO for quadrotor drone applications.
The KPPO algorithm builds upon PPO by incorporating a composite regularization mechanism. Traditional PPO uses a clipped surrogate objective to limit policy updates, but KPPO introduces a threshold-triggered KL divergence penalty combined with L2 regularization. This design aims to dynamically constrain the trust region during policy iteration, preventing excessive deviation and enhancing robustness. Specifically, the KL divergence between old and new policies is computed as:
$$
D_{\text{KL}} = \mathbb{E} \left[ \log \pi_{\theta_{\text{old}}}(a_t | s_t) – \log \pi_{\theta}(a_t | s_t) \right]
$$
A penalty is applied only when \( D_{\text{KL}} \) exceeds a threshold \( \delta = 0.01 \), ensuring that updates are constrained without overly restricting exploration. The KL penalty term is defined as:
$$
L_{\text{KL}} = \partial \cdot \max(D_{\text{KL}} – \delta, 0)
$$
where \( \partial \) is a hyperparameter controlling penalty strength. Additionally, L2 regularization is applied to policy network parameters to prevent overfitting and promote generalization:
$$
L_{\text{L2}} = \sum_i \| \theta_i \|^2
$$
The policy loss in KPPO uses a minimum constraint instead of clipping, formulated as:
$$
L_{\text{policy}} = \mathbb{E} \left[ -\min(r_t A_t, A_t) \right]
$$
Here, \( r_t = \frac{\pi_{\theta}(a_t | s_t)}{\pi_{\theta_{\text{old}}}(a_t | s_t)} \) is the probability ratio, and \( A_t = R_t – V(s_t) \) is the advantage function, with \( R_t \) being the discounted return and \( V(s_t) \) the state value estimate. The value loss is:
$$
L_{\text{value}} = \mathbb{E} \left[ (V(s_t) – R_t)^2 \right]
$$
The total loss function combines these components:
$$
L = L_{\text{policy}} + \beta L_{\text{L2}} + L_{\text{KL}} + L_{\text{value}}
$$
where \( \beta \) is the weight for L2 regularization. This composite loss enables the quadrotor drone to learn stable and adaptive control policies through gradient-based optimization.
To evaluate the KPPO algorithm, I conducted physical simulations across five distinct tasks for quadrotor drone flight control. The simulation environment models a quadrotor drone using the QuadDynamics class, which implements rigid-body dynamics. The state vector is 12-dimensional: \( s = [x, y, z, \phi, \theta, \psi, v_x, v_y, v_z, \omega_x, \omega_y, \omega_z]^T \), including position, attitude angles, linear velocities, and angular velocities. The action space consists of four continuous control inputs for rotor thrusts. Physical parameters are summarized in the table below.
| Parameter | Value | Unit |
|---|---|---|
| Gravity acceleration | 9.81 | m/s² |
| Time step | 0.02 | s |
| Quadrotor mass | 0.665 | kg |
| Rotor arm length | 0.105 | m |
| Moment of inertia | [0.0023, 0.0025, 0.0037] | kg·m² |
The five simulation tasks are designed to test various aspects of quadrotor drone control:
- Velocity Control Task: The quadrotor drone must achieve and maintain a target velocity. Reward is based on speed error: \( \text{reward} = -w_1 \cdot \text{speed\_err} \).
- Height Control Task: The quadrotor drone aims to reach and hold a target altitude. Reward is: \( \text{reward} = -w_1 \cdot z_{\text{err}} \).
- Hover Task: The quadrotor drone must stabilize at a target position and attitude. Reward combines position and angle errors: \( \text{reward} = -(w_1 \cdot \text{pos\_err} + w_2 \cdot \text{angle\_err}) \).
- Single-Target Tracking Task: The quadrotor drone tracks a moving target. Reward is: \( \text{reward} = -(w_1 \cdot \text{dist\_err} + w_2 \cdot \text{v\_err}) \).
- Multi-Target Tracking Task: The quadrotor drone simultaneously tracks multiple targets. Reward is: \( \text{reward} = -w_1 \cdot \sum_i \text{dist\_err}_i \).
In all tasks, additional rewards are given for task completion, and penalties for failures. The quadrotor drone operates in a continuous action space, with control inputs generated by the KPPO policy network. For comparison, I implemented baseline PPO and DDPG algorithms. Training parameters for KPPO are consistent across tasks, as shown below.
| Parameter | Value |
|---|---|
| Mini-batch size | 64 |
| Training epochs | 20 |
| Discount factor | 0.99 |
| Learning rate | 0.0001 |
| Optimizer coefficients | (0.9, 0.999) |
| Task completion reward | 300 |
| Additional reward | 1000 |
| Penalty value | -1 |
| Max training episodes | 1000 |
| Max steps per episode | 400 |
| Model update interval | 400 steps |
| KL penalty coefficient | 0.2 |
| L2 regularization coefficient | 1e-5 |
| Random seed | 10 |
| State dimension | 12 |
| Action dimension | 4 |
| KL divergence threshold | 0.01 |
DDPG parameters include a learning rate of 0.0001, discount factor 0.99, target network soft update coefficient 0.001, experience replay batch size 64, and action exploration noise strength 0.1. All simulations were run on a hardware platform with an AMD Ryzen 5 PRO 4650U processor and 16 GB RAM, ensuring consistent evaluation.
The results demonstrate that KPPO significantly outperforms PPO and DDPG in most tasks. For the velocity control task, KPPO achieves higher reward values and faster convergence, as shown in the average reward per episode. The hover task reveals that KPPO enables the quadrotor drone to stabilize more quickly, with reward curves indicating rapid policy improvement. In the height control task, KPPO initially lags but surpasses other algorithms after around 600 episodes, showcasing its adaptive learning capability. The multi-target tracking task poses greater complexity, but KPPO maintains stable performance, whereas PPO and DDPG exhibit higher volatility. To quantify these observations, I present the total average rewards and standard deviations across all tasks in the following tables.
| Task | KPPO Average Reward | PPO Average Reward | DDPG Average Reward |
|---|---|---|---|
| Height Control | -1322.185 | -3472.665 | -364.295 |
| Multi-Target Tracking | -3542.908 | -3577.797 | -432.059 |
| Velocity Control | 8879.512 | 3242.066 | 1504.431 |
| Hover Task | -90.670 | -255.520 | -320.119 |
| Single-Target Tracking | -478.866 | -481.655 | -497.700 |
| Task | KPPO Standard Deviation | PPO Standard Deviation | DDPG Standard Deviation |
|---|---|---|---|
| Height Control | 4031.546 | 11.052 | 0.647 |
| Multi-Target Tracking | 11.771 | 143.894 | 6.944 |
| Velocity Control | 11845.580 | 8519.012 | 3671.278 |
| Hover Task | 1764.684 | 686.389 | 27.569 |
| Single-Target Tracking | 220.065 | 218.798 | 8.875 |
These tables highlight that KPPO achieves higher average rewards in key tasks like velocity control and hover, indicating better task performance. The standard deviations reflect that KPPO, as a stochastic policy algorithm, exhibits more exploration-driven volatility, which contributes to its ability to escape local optima in complex environments. In contrast, DDPG’s deterministic policy leads to lower variance but slower convergence and poorer adaptation for the quadrotor drone. The trajectory analysis further supports these findings. For instance, in the hover task, the quadrotor drone under KPPO control converges to the target position within fewer steps, while in tracking tasks, KPPO shows more exploratory behavior initially but stabilizes faster than PPO. The composite regularization in KPPO effectively balances exploration and exploitation, enabling the quadrotor drone to handle dynamic disturbances and multi-objective scenarios.
The mathematical formulation of KPPO’s advantage estimation plays a crucial role in its success. The advantage function \( A_t \) is computed using generalized advantage estimation (GAE), which reduces variance in policy updates. For a quadrotor drone, this is expressed as:
$$
A_t = \delta_t + (\gamma \lambda) \delta_{t+1} + (\gamma \lambda)^2 \delta_{t+2} + \dots
$$
where \( \delta_t = r_t + \gamma V(s_{t+1}) – V(s_t) \), \( \gamma \) is the discount factor, and \( \lambda \) is a smoothing parameter. This allows the quadrotor drone to make informed decisions based on long-term rewards. Additionally, the policy network architecture in KPPO consists of two hidden layers with 256 and 128 units, using ReLU activations, which provides sufficient capacity to model the nonlinear dynamics of quadrotor drone flight. The value network mirrors this structure to estimate state values accurately. The integration of L2 regularization prevents overfitting by penalizing large weights, as shown in the loss function. For a quadrotor drone operating in noisy environments, this enhances generalization to unseen states.
In the single-target tracking task, the reward function is designed to minimize both distance and velocity errors. Let \( p_{\text{target}} \) be the target position and \( p_{\text{drone}} \) the quadrotor drone’s position. The distance error is \( \text{dist\_err} = \| p_{\text{target}} – p_{\text{drone}} \| \), and the velocity error is \( \text{v\_err} = \| v_{\text{target}} – v_{\text{drone}} \| \). The reward is then:
$$
\text{reward} = – (w_1 \cdot \text{dist\_err} + w_2 \cdot \text{v\_err})
$$
With KPPO, the quadrotor drone learns to reduce these errors efficiently by adjusting rotor thrusts. The threshold-triggered KL penalty ensures that policy updates do not deviate excessively, maintaining stability during learning. For example, if the KL divergence between consecutive policies exceeds 0.01, the penalty term \( L_{\text{KL}} \) activates, curbing large changes that could destabilize the quadrotor drone’s flight. This mechanism is particularly beneficial in tasks like multi-target tracking, where the quadrotor drone must coordinate multiple objectives. The L2 regularization further strengthens parameter robustness, as evidenced by the quadrotor drone’s consistent performance across simulation episodes.
To delve deeper into the simulation results, I analyze the convergence behavior using the average reward over multiple episodes. For the velocity control task, KPPO’s reward curve rises steeply, indicating rapid learning. The hover task shows that KPPO achieves near-optimal performance within 200 episodes, while PPO and DDPG require more episodes. In the height control task, KPPO’s reward initially fluctuates but surpasses others after 600 episodes, demonstrating its ability to refine policies over time. The multi-target tracking task presents the greatest challenge, but KPPO maintains a higher average reward than PPO, with DDPG performing poorly due to its limited exploration. These trends underscore the importance of the composite regularization in KPPO for enhancing quadrotor drone control in diverse scenarios.
The physical simulation of quadrotor drone dynamics involves numerical integration of the equations of motion. Given control inputs \( T = [T_1, T_2, T_3, T_4]^T \), the linear acceleration in the world frame is:
$$
a = \frac{1}{m} (R [0, 0, \sum_i T_i]^T + m g)
$$
where \( R \) is the rotation matrix from body to world frame. The angular acceleration is \( \alpha = I^{-1} \tau \), with torque \( \tau = [(T_4 – T_3)l, (T_2 – T_1)l, 0]^T \). Position and attitude are updated via Euler integration:
$$
\begin{align*}
p_{t+1} &= p_t + v_t \Delta t + \frac{1}{2} a_t \Delta t^2 \\
v_{t+1} &= v_t + a_t \Delta t \\
\phi_{t+1} &= \phi_t + \omega_t \Delta t + \frac{1}{2} \alpha_t \Delta t^2 \\
\omega_{t+1} &= \omega_t + \alpha_t \Delta t
\end{align*}
$$
This simulation framework ensures realistic quadrotor drone behavior, allowing the KPPO algorithm to learn from accurate dynamics. The state vector \( s \) is normalized before input to the neural networks to improve training stability. For each task, the quadrotor drone starts from a random initial state, and episodes terminate upon task completion or after 400 steps. The reward functions are tailored to guide the quadrotor drone toward task objectives, with the KPPO policy network outputting continuous actions in the range [-1, 1], which are scaled to actual rotor thrusts.
In comparison to DDPG, KPPO’s on-policy nature means it uses fresh data for each update, which can be less sample-efficient but more stable for quadrotor drone control. DDPG, as an off-policy algorithm, reuses experience replay but may suffer from delayed policy improvements. The results show that KPPO’s stochastic policy facilitates better exploration, leading to higher rewards in complex tasks. For instance, in the hover task, the quadrotor drone under KPPO quickly learns to minimize position and angle errors, while DDPG often gets stuck in suboptimal hover states. The tables above quantitatively confirm that KPPO outperforms in average reward across multiple tasks, with the quadrotor drone achieving faster convergence and better adaptation.
The KPPO algorithm’s design also incorporates a value function estimator that reduces variance in advantage calculations. The value loss \( L_{\text{value}} \) is minimized using gradient descent, ensuring that the quadrotor drone’s state value predictions align with actual returns. This is critical for tasks like tracking, where the quadrotor drone must anticipate future rewards. The policy loss \( L_{\text{policy}} \) uses the minimum constraint to prevent overly large updates, which could harm the quadrotor drone’s stability. Combined with the KL penalty and L2 regularization, KPPO strikes a balance between aggressive optimization and conservative learning, making it suitable for real-world quadrotor drone applications where safety and reliability are paramount.
Future work could focus on enhancing the KPPO algorithm for quadrotor drone control by integrating prioritized experience replay to improve sample efficiency, or employing temporal difference methods like TD(λ) for more stable value estimation. Additionally, extending the simulation to include more realistic environmental factors, such as wind gusts or sensor noise, would further test the robustness of the quadrotor drone under KPPO control. The composite regularization mechanism could also be adapted to other DRL algorithms for broader unmanned system applications.
In conclusion, the KPPO algorithm represents a significant advancement in deep reinforcement learning for quadrotor drone flight control. By fusing threshold-triggered KL divergence penalties with L2 regularization, it addresses the limitations of traditional PPO in convergence speed and environmental adaptability. Through comprehensive physical simulations, I have demonstrated that KPPO enables the quadrotor drone to achieve faster policy convergence, superior task performance, and enhanced robustness in complex scenarios. The results validate the effectiveness of the composite regularization approach, paving the way for more intelligent and autonomous quadrotor drone systems in practical applications.