In recent years, the application of quadrotor drones has expanded dramatically across various fields such as agricultural monitoring, industrial inspection, aerial transportation, and military operations. Their lightweight design, compact size, and maneuverability make them ideal for tasks that are hazardous or inaccessible to humans, such as explosive mine detection. However, achieving precise and stable flight control remains a critical challenge. Traditional control methods, like Proportional-Integral-Derivative (PID) controllers, often require meticulous model-specific tuning and expert knowledge, which can be time-consuming and suboptimal in dynamic environments. To address these limitations, we explore the use of reinforcement learning (RL), specifically the Proximal Policy Optimization (PPO) algorithm, for position control of quadrotor drones. This approach enables model-free learning, where the control policy is directly mapped from states to actuator commands through neural networks, eliminating the need for manual parameter adjustment and system identification. In this paper, we present a comprehensive framework for quadrotor drone hover control and trajectory tracking using PPO, with a novel reward function design that enhances convergence speed and flight stability. We demonstrate through simulations that our method outperforms conventional PID control in terms of accuracy, speed, and robustness, achieving over 97.2% precision in three-dimensional position control tasks.
The core of our work lies in leveraging deep reinforcement learning to autonomously learn control policies for quadrotor drones. Reinforcement learning allows an agent—here, the quadrotor drone—to interact with an environment, receive rewards based on its actions, and optimize its policy to maximize cumulative rewards. Unlike traditional control, which relies on linearized models and heuristic tuning, RL can handle nonlinear dynamics and adapt to uncertainties. We focus on PPO, a state-of-the-art RL algorithm known for its stability and sample efficiency. PPO uses an actor-critic architecture to iteratively improve policies while preventing large updates that could lead to instability. By applying PPO to quadrotor drone control, we aim to develop a controller capable of precise hover and trajectory tracking without prior knowledge of system dynamics. This paper details our methodology, including the quadrotor drone dynamics modeling, state-action representation, reward shaping, and experimental validation. We also discuss the generalization and robustness of our controller, showcasing its performance in various scenarios.

To understand the control problem, we first establish the mathematical model of a quadrotor drone. A quadrotor drone is a rigid body with four rotors arranged in a cross configuration. Each rotor produces thrust and torque, and by varying the rotational speeds, we can control the drone’s position and attitude. We define two coordinate systems: the body-fixed frame (attached to the drone) and the earth-fixed frame (inertial frame). Let the position vector be $\mathbf{P} = [x, y, z]^T$, the linear velocity $\mathbf{v} = [v_x, v_y, v_z]^T$, the attitude Euler angles $\boldsymbol{\eta} = [\phi, \theta, \psi]^T$ (roll, pitch, yaw), and the angular velocity $\boldsymbol{\omega} = [\omega_x, \omega_y, \omega_z]^T$. The dynamics of the quadrotor drone are derived from Newton-Euler equations, considering forces and moments. The thrust from each rotor is $F_i = k_F \omega_i^2$, and the drag moment is $M_i = k_M \omega_i^2$, where $k_F$ and $k_M$ are lift and drag coefficients, and $\omega_i$ is the angular speed of rotor $i$ ($i=1,2,3,4$). The total thrust $U_1$ and moments $U_2, U_3, U_4$ are given by:
$$
\begin{bmatrix} U_1 \\ U_2 \\ U_3 \\ U_4 \end{bmatrix} =
\begin{bmatrix}
k_F & k_F & k_F & k_F \\
0 & k_F l & 0 & -k_F l \\
-k_F l & 0 & k_F l & 0 \\
k_M & -k_M & k_M & -k_M
\end{bmatrix}
\begin{bmatrix} \omega_1^2 \\ \omega_2^2 \\ \omega_3^2 \\ \omega_4^2 \end{bmatrix},
$$
where $l$ is the arm length from the center to each rotor. The full nonlinear dynamics can be expressed as:
$$
\begin{aligned}
\dot{\mathbf{P}} &= \mathbf{R} \mathbf{v}, \\
m \dot{\mathbf{v}} &= \mathbf{G} – \mathbf{R} \mathbf{F} + \mathbf{F}_d, \\
\dot{\boldsymbol{\eta}} &= \mathbf{T} \boldsymbol{\omega}, \\
\mathbf{I} \dot{\boldsymbol{\omega}} &= -\boldsymbol{\omega} \times \mathbf{I} \boldsymbol{\omega} + \boldsymbol{\tau} + \mathbf{M}_d,
\end{aligned}
$$
where $m$ is the mass, $\mathbf{G} = [0, 0, mg]^T$ is gravity, $\mathbf{F} = [0, 0, U_1]^T$ is the thrust vector, $\mathbf{F}_d$ and $\mathbf{M}_d$ are disturbance forces and moments, $\mathbf{I}$ is the inertia matrix, $\boldsymbol{\tau} = [U_2, U_3, U_4]^T$ is the control moment, $\mathbf{R}$ is the rotation matrix from body to earth frame, and $\mathbf{T}$ is the transformation matrix for angular velocities. The rotation matrix $\mathbf{R}$ is:
$$
\mathbf{R} = \begin{bmatrix}
c\theta c\psi & s\phi s\theta c\psi – c\phi s\psi & c\phi s\theta c\psi + s\phi s\psi \\
c\theta s\psi & s\phi s\theta s\psi + c\phi c\psi & c\phi s\theta s\psi – s\phi c\psi \\
-s\theta & s\phi c\theta & c\phi c\theta
\end{bmatrix},
$$
where $s$ and $c$ denote sine and cosine. The matrix $\mathbf{T}$ is:
$$
\mathbf{T} = \begin{bmatrix}
1 & s\phi t\theta & c\phi t\theta \\
0 & c\phi & -s\phi \\
0 & s\phi/c\theta & c\phi/c\theta
\end{bmatrix},
$$
with $t$ denoting tangent. These equations form the basis for simulating the quadrotor drone environment in our RL framework.
We now describe the PPO algorithm and its application to quadrotor drone control. PPO is a policy gradient method that optimizes a stochastic policy by maximizing a surrogate objective function. It addresses the instability issues of traditional policy gradients by clipping the probability ratio between old and new policies, ensuring conservative updates. The algorithm uses an actor-critic architecture: the actor network $\mu(\mathbf{s} | \theta_\mu)$ outputs a probability distribution over actions given state $\mathbf{s}$, and the critic network $V(\mathbf{s} | \theta_V)$ estimates the value function. The objective for the actor is to maximize:
$$
J(\theta_\mu) = \mathbb{E}_{\mathbf{s}, \mathbf{a} \sim \pi_{\theta_{\mu,old}}} \left[ \min\left( r(\theta_\mu) A^{\pi_{\theta_{\mu,old}}}(\mathbf{s}, \mathbf{a}), \text{clip}(r(\theta_\mu), 1-\epsilon, 1+\epsilon) A^{\pi_{\theta_{\mu,old}}}(\mathbf{s}, \mathbf{a}) \right) \right],
$$
where $r(\theta_\mu) = \frac{\mu(\mathbf{a} | \mathbf{s}, \theta_\mu)}{\mu(\mathbf{a} | \mathbf{s}, \theta_{\mu,old})}$ is the probability ratio, $A$ is the advantage function estimated by the critic, and $\epsilon$ is a clipping parameter (typically 0.1 to 0.2). The critic is updated by minimizing the mean-squared error between predicted and actual returns:
$$
L(\theta_V) = \mathbb{E}_{\mathbf{s}} \left[ (G_t – V(\mathbf{s} | \theta_V))^2 \right],
$$
where $G_t$ is the discounted cumulative reward. For the quadrotor drone control task, we define the state space, action space, and reward function as follows.
The state space $\mathbf{s}$ encapsulates all relevant information for position control. We design it to include position error, velocity error, acceleration error, attitude, and angular velocity, ensuring the agent can perceive deviations from the desired trajectory. Specifically:
$$
\mathbf{s} = [\mathbf{e}_{pos}, \mathbf{e}_{vel}, \mathbf{e}_{acc}, \mathbf{q}, \boldsymbol{\omega}],
$$
where $\mathbf{e}_{pos} = \mathbf{P}_{des} – \mathbf{P}$ is the position error vector, $\mathbf{e}_{vel} = \dot{\mathbf{P}}_{des} – \dot{\mathbf{P}}$ is the velocity error, $\mathbf{e}_{acc} = \ddot{\mathbf{P}}_{des} – \ddot{\mathbf{P}}$ is the acceleration error, $\mathbf{q}$ is the quaternion representation of attitude (for numerical stability), and $\boldsymbol{\omega}$ is the angular velocity. This comprehensive state representation allows the quadrotor drone to react dynamically to changes in its environment.
The action space $\mathbf{a}$ consists of the thrust forces for each rotor, directly controlling the quadrotor drone’s motion. We define:
$$
\mathbf{a} = [f_1, f_2, f_3, f_4],
$$
where $f_i$ is the thrust force of rotor $i$, bounded between 0 and 10 Newtons to reflect physical limits. By outputting continuous values, the PPO agent can finely adjust the rotor speeds to achieve precise control. This direct mapping from states to rotor forces eliminates the need for intermediate controllers, simplifying the architecture.
The reward function is crucial for guiding the quadrotor drone toward desired behaviors. We propose a novel reward function based on reward shaping, which combines multiple components to encourage accurate hover and trajectory tracking. The reward $r$ at each time step is:
$$
r = \mathbf{k}^T \cdot [r_p, r_v, r_a, r_y]^T,
$$
where $\mathbf{k} = [k_1, k_2, k_3, k_4]$ are weighting coefficients, and the components are defined as:
$$
\begin{aligned}
r_p &= \exp(-1 / \|\mathbf{e}_{pos}\|), \\
r_v &= \exp(-1 / \|\mathbf{e}_{vel}\|), \\
r_a &= \exp(-1 / \|\mathbf{e}_{acc}\|), \\
r_y &= \exp(-1 / |\psi_{error}|),
\end{aligned}
$$
with $\psi_{error}$ being the yaw angle error. These exponential terms provide high rewards for small errors and penalize large deviations smoothly. Additionally, we impose a penalty of -1 if the position error or yaw angle exceeds a safety threshold, ensuring safe operation. Compared to traditional reward functions, such as linear penalties, our design promotes faster convergence and smoother flight, as demonstrated in experiments. The coefficients are tuned empirically; we use $k_1=0.6, k_2=0.1, k_3=0.1, k_4=0.2$ to prioritize position accuracy while maintaining stability.
To validate our approach, we implemented a simulation environment based on the quadrotor drone dynamics. The environment is built in Python, using numerical integration to update states at a time step of 0.01 seconds. We trained the PPO agent using the Stable Baselines3 library, with network architectures comprising two hidden layers of 64 neurons each for both actor and critic. The training parameters are summarized in the table below.
| Parameter | Value |
|---|---|
| Discount factor ($\gamma$) | 0.99 |
| Learning rate (actor) | 0.0001 |
| Learning rate (critic) | 0.0001 |
| Clipping parameter ($\epsilon$) | 0.2 |
| Entropy coefficient | 0.4 |
| Batch size | 256 |
| Number of episodes | 10,000 |
| Steps per episode | 500 |
The quadrotor drone parameters used in simulation are based on typical models, as shown in the following table.
| Parameter | Value | Unit |
|---|---|---|
| Mass ($m$) | 0.961 | kg |
| Arm length ($l$) | 0.263 | m |
| Moment of inertia ($I_x$) | 0.001152 | kg·m² |
| Moment of inertia ($I_y$) | 0.001152 | kg·m² |
| Moment of inertia ($I_z$) | 0.0218 | kg·m² |
| Gravity ($g$) | 9.807 | m/s² |
We conducted two main tasks: hover control at a fixed point and trajectory tracking. For hover control, the quadrotor drone starts from a random initial state and must reach and stabilize at a target position in 3D space. For trajectory tracking, the drone follows predefined paths, such as an “8”-shaped curve and an Immelman turn (a complex maneuver). We compare our PPO-based controller with a well-tuned PID controller to evaluate performance. The PID controller uses separate loops for position and attitude, with gains manually optimized for fair comparison.
The training process for the quadrotor drone controller shows that our reward function leads to faster convergence than alternative designs. For instance, compared to a reward function from prior work (e.g., $r = 1.5 – \|\mathbf{e}_{pos}\| – 0.05|\dot{\phi}| – 0.05|\dot{\theta}| – |\dot{\psi}|$), our exponential-based reward achieves stable policy convergence within 2,000 episodes, whereas the other requires over 7,000 episodes. This highlights the effectiveness of our shaping approach in guiding the quadrotor drone toward optimal behavior. During training, we observed that the agent quickly learns to reduce position errors while maintaining smooth attitude changes, resulting in stable hover.
For hover control, we tested the quadrotor drone at 12 random target positions in 3D space. The steady-state error, defined as the average position deviation after stabilization, was consistently below 0.03 meters, demonstrating high accuracy. The results are summarized in the table below.
| Target Position (x, y, z) in meters | Steady-State Error (m) |
|---|---|
| (1, 1, 1) | 0.0282 |
| (2, 2, 2) | 0.0282 |
| (3, 4, 5) | 0.0258 |
| (1, 3, 5) | 0.0310 |
| (3, 3, 3) | 0.0271 |
| (1, 3, 0) | 0.0284 |
| (1, 2, 0) | 0.0286 |
| (1, 1, 3) | 0.0271 |
| (1, 2, 1) | 0.0256 |
| (1, 3, 2) | 0.0286 |
| (1, 5, 1) | 0.0302 |
| (2, 1, 4) | 0.0284 |
In a detailed comparison with PID control, we tasked the quadrotor drone to move from the origin $(0,0,0)$ to a target $(3,4,5)$ meters. The PPO controller achieved the target in 8.37 seconds with no overshoot and minimal oscillation, whereas the PID controller took 12 seconds with a 19% overshoot in altitude, causing significant vibrations that required 20 seconds to fully stabilize. This demonstrates that our RL-based controller offers superior speed and stability for quadrotor drone position control. The trajectory plots show that the PPO agent follows a smoother path, adapting dynamically to errors, while the PID controller exhibits jerky corrections due to its fixed gains.
For trajectory tracking, the quadrotor drone successfully follows complex paths. The “8”-shaped trajectory is defined parametrically as $x = \sin(t), y = \sin(t)\cos(t), z = 0.2$ (with scaling), and the Immelman turn is a 3D maneuver combining a loop and a roll. The quadrotor drone tracks these paths with an average position error of less than 0.05 meters, as shown in the figure above. The control commands generated by the PPO agent are continuous and smooth, ensuring the quadrotor drone maintains stable flight even during rapid maneuvers. This capability is essential for real-world applications where precise path following is required, such as surveillance or delivery missions.
The robustness of our quadrotor drone controller is tested under disturbances. We introduced random wind gusts as force disturbances $\mathbf{F}_d$ up to 10% of the thrust magnitude. The PPO controller maintains position errors within 0.1 meters, while the PID controller shows larger deviations and occasional instability. This highlights the adaptability of RL-based control to uncertain environments. Furthermore, we evaluated generalization by varying the quadrotor drone’s mass and inertia within ±20% of nominal values. Without retraining, the PPO controller still achieves hover with errors below 0.04 meters, indicating that the learned policy captures underlying dynamics rather than overfitting to specific parameters.
Our work advances the state of the art in quadrotor drone control by demonstrating the efficacy of PPO for position control. Unlike prior studies that focus only on height control or simple maneuvers, we address full 3D position control and complex trajectory tracking. The key innovations include a comprehensive state representation, a shaped reward function for fast convergence, and direct output of rotor forces. The quadrotor drone benefits from this approach through improved accuracy, speed, and robustness compared to traditional methods. We believe that RL-based controllers like ours will enable more autonomous and reliable quadrotor drones for diverse applications.
However, there are limitations to consider. The training process is computationally intensive, requiring thousands of episodes in simulation. While we used a lightweight simulator, transferring the policy to a real quadrotor drone would need additional steps like domain randomization or real-world fine-tuning. Future work could explore sample-efficient RL algorithms, such as off-policy methods or meta-learning, to reduce training time. Additionally, incorporating safety constraints explicitly into the RL framework could enhance the quadrotor drone’s reliability in critical tasks. Extending the controller to multi-quadrotor drone scenarios or dynamic obstacle avoidance would also be valuable.
In conclusion, we have presented a novel position control system for quadrotor drones using the PPO algorithm. Our method eliminates the need for manual tuning and model knowledge, learning a control policy directly from interactions with the environment. Through extensive simulations, we show that the quadrotor drone achieves precise hover and trajectory tracking with over 97.2% accuracy, outperforming PID control in both speed and stability. The controller generalizes well to different targets and disturbances, proving its robustness. This research underscores the potential of reinforcement learning for autonomous drone control, paving the way for more intelligent and adaptive quadrotor drone systems. As RL techniques evolve, we expect quadrotor drones to become even more capable in complex, real-world environments.
