In the realm of unmanned aerial vehicles (UAVs), the quadrotor drone stands out due to its simple structure, agility, and stability, making it ideal for applications like aerial photography, surveillance, and exploration. Precise trajectory tracking is fundamental for executing complex tasks, yet achieving high accuracy in unknown or noisy environments remains a significant challenge. Traditional feedback control methods, such as PID, often suffer from latency and sensitivity to disturbances, while model-free intelligent approaches demand extensive computation, hindering real-time deployment. In this work, I propose a novel hybrid control framework that integrates iterative learning control (ILC) as a feedforward component with a conventional feedback controller, enhanced by reinforcement learning (RL) for parameter optimization. This method, termed Reinforcement Learning-Iterative Learning Control (RL-ILC), aims to boost tracking accuracy and robustness for quadrotor drones by leveraging historical flight data and adapting to environmental uncertainties. The core innovation lies in using RL to tune the learning parameters of ILC, overcoming the manual tuning difficulties and ensuring rapid convergence. Through simulations with random noise, I demonstrate that RL-ILC outperforms standard ILC, sliding mode control (SMC), and PID in terms of error reduction and stability. This article delves into the system modeling, control design, and experimental validation, providing a comprehensive guide for advancing quadrotor drone trajectory control.
The quadrotor drone operates through four rotors that generate lift and torque, enabling movement in six degrees of freedom. To develop an effective control strategy, a accurate dynamic model is essential. The quadrotor drone’s motion can be described in body and inertial frames, with assumptions such as rigid body dynamics and symmetric structure. The lift force from each rotor is given by:
$$
f_i = C_T \cdot \omega_i^2
$$
where \( C_T \) is the thrust coefficient and \( \omega_i \) is the rotor speed. The total thrust in the body frame is:
$$
F_T = \sum_{i=1}^{4} f_i
$$
Ignoring aerodynamic drag, the translational accelerations in the inertial frame are derived using Euler angles—yaw (\( \phi \)), pitch (\( \theta \)), and roll (\( \gamma \)):
$$
\begin{bmatrix} \ddot{x} \\ \ddot{y} \\ \ddot{z} \end{bmatrix} = \begin{bmatrix} (\sin \phi \sin \gamma + \cos \phi \sin \theta \cos \gamma) F_T / m \\ (-\cos \phi \sin \gamma + \sin \phi \sin \theta \cos \gamma) F_T / m \\ (\cos \gamma \cos \theta) F_T / m + g \end{bmatrix}
$$
Here, \( m \) is the mass, \( g \) is gravity, and \( x, y, z \) are position coordinates. The rotational dynamics involve moments of inertia \( J_{xx}, J_{yy}, J_{zz} \) and rotor distances \( d \):
$$
\begin{bmatrix} \ddot{\phi} \\ \ddot{\theta} \\ \ddot{\gamma} \end{bmatrix} = \begin{bmatrix} \dot{\theta} \dot{\gamma} \Delta J_{xy} / J_{zz} + C_m (F_{2,4} – F_{1,3}) / J_{zz} \\ \dot{\phi} \dot{\gamma} \Delta J_{zx} / J_{yy} + d (f_1 – f_3) / J_{yy} \\ \dot{\theta} \dot{\phi} \Delta J_{yz} / J_{xx} + d (f_2 – f_4) / J_{xx} \end{bmatrix}
$$
with \( \Delta J_{ij} = J_{ii} – J_{jj} \) and \( F_{i,j} = f_i + f_j \). Defining control inputs \( U_1 = F_T \), \( U_2 = f_2 – f_4 \), \( U_3 = f_1 – f_3 \), and \( U_4 = F_{2,4} – F_{1,3} \), the complete dynamic model for the quadrotor drone is:
$$
\begin{aligned}
\ddot{x} &= (\sin \phi \sin \gamma + \cos \phi \sin \theta \cos \gamma) U_1 / m \\
\ddot{y} &= (-\cos \phi \sin \gamma + \sin \phi \sin \theta \cos \gamma) U_1 / m \\
\ddot{z} &= (\cos \gamma \cos \theta) U_1 / m + g \\
\ddot{\phi} &= \dot{\theta} \dot{\gamma} \Delta J_{xy} / J_{zz} + C_m U_4 / J_{zz} \\
\ddot{\theta} &= \dot{\phi} \dot{\gamma} \Delta J_{zx} / J_{yy} + d U_3 / J_{yy} \\
\ddot{\gamma} &= \dot{\theta} \dot{\phi} \Delta J_{yz} / J_{xx} + d U_2 / J_{xx}
\end{aligned}
$$
Key parameters for a typical quadrotor drone, such as the DJI 450 mm model, are summarized in Table 1. This model forms the basis for control system design, highlighting the nonlinear and underactuated nature of the quadrotor drone, where only four inputs control six outputs.
| Parameter | Symbol | Value |
|---|---|---|
| Mass | \( m \) | 1.5 kg |
| X-axis inertia | \( J_{xx} \) | \( 1.745 \times 10^{-2} \, \text{kg} \cdot \text{m}^2 \) |
| Y-axis inertia | \( J_{yy} \) | \( 1.745 \times 10^{-2} \, \text{kg} \cdot \text{m}^2 \) |
| Z-axis inertia | \( J_{zz} \) | \( 3.175 \times 10^{-2} \, \text{kg} \cdot \text{m}^2 \) |
| Arm length | \( d \) | 0.225 m |
| Thrust coefficient | \( C_T \) | \( 1.105 \times 10^{-5} \, \text{N} \cdot (\text{rad} \cdot \text{s}^{-1})^{-2} \) |
| Moment coefficient | \( C_m \) | \( 1.489 \times 10^{-7} \, \text{m} \) |
The control system architecture for the quadrotor drone combines feedback and feedforward elements. A traditional PID feedback controller handles real-time errors, but it introduces delay and struggles with persistent disturbances. To address this, I incorporate an ILC-based feedforward controller that learns from repeated trajectories. The overall control input \( U(t, k) \) at time \( t \) and iteration \( k \) is:
$$
U(t, k) = U_{\text{ff}}(t, k) + U_{\text{fb}}(t)
$$
where \( U_{\text{ff}} \) is the feedforward input from ILC and \( U_{\text{fb}} \) is the feedback input from PID. The error \( E(t) \) between desired trajectory \( Y_d(t) \) and actual output \( Y(t) \) is:
$$
E(t) = Y_d(t) – Y(t) = \begin{bmatrix} e_x(t) \\ e_y(t) \\ e_z(t) \end{bmatrix}
$$
The feedback control for each axis uses PID gains:
$$
u_{\text{fb}, i}(t) = \begin{bmatrix} k_P & k_I & k_D \end{bmatrix} \cdot \begin{bmatrix} e_i(t-1) \\ \int_0^{t-1} e_i(\tau) d\tau \\ \dot{e}_i(t-1) \end{bmatrix}
$$
This hybrid structure enhances responsiveness and accuracy for the quadrotor drone, leveraging ILC’s ability to refine inputs over iterations.
Iterative learning control is a powerful technique for systems that repeat tasks, such as a quadrotor drone following a predefined path. ILC updates the control input based on past errors, aiming to achieve perfect tracking after finite iterations. For the quadrotor drone, I design a PD-type ILC with learning laws for altitude and attitude. Assuming small angles and decoupling, the simplified dynamics for trajectory tracking are:
$$
\begin{aligned}
\ddot{x} &= \theta \cdot U_1 / m + D_x \\
\ddot{y} &= -\gamma \cdot U_1 / m + D_y \\
\ddot{z}’ &= U_1 / m + D_z
\end{aligned}
$$
where \( \ddot{z}’ = \ddot{z} – g \), and \( D_x, D_y, D_z \) are unknown disturbances. Desired pitch and roll angles are derived from acceleration profiles:
$$
\theta_d = \frac{\ddot{x}_d – D_x}{\ddot{z}’_d – D_z}, \quad \gamma_d = -\frac{\ddot{y}_d – D_y}{\ddot{z}’_d – D_z}
$$
Since disturbances are unknown, ILC iteratively estimates these. The state vector is \( X = [\dot{z}, \ddot{z}’, \theta, \dot{\theta}, \gamma, \dot{\gamma}]^T \), with inputs \( U = [U_1, U_2, U_3]^T \) and outputs \( Y = [z, \theta, \gamma]^T \). The PD-ILC laws for the quadrotor drone are:
$$
\begin{aligned}
U_1(t, k+1) &= U_1(t, k) + [K_{zP}, K_{zD}] \cdot \begin{bmatrix} e_z(t+2, k) \\ \dot{e}_z(t+1, k) \end{bmatrix} \\
\theta_d(t, k+1) &= \theta_d(t, k) + [K_{xP}, K_{xD}] \cdot \begin{bmatrix} e_x(t+2, k) \\ \dot{e}_x(t+1, k) \end{bmatrix} \\
\gamma_d(t, k+1) &= \gamma_d(t, k) + [K_{yP}, K_{yD}] \cdot \begin{bmatrix} e_y(t+2, k) \\ \dot{e}_y(t+1, k) \end{bmatrix}
\end{aligned}
$$
Here, \( K_{P} \) and \( K_{D} \) are learning gains, and error derivatives are shifted to compensate for system delays. The control inputs \( U_2 \) and \( U_3 \) are computed from desired angles:
$$
U_2 = \frac{J_{xx} \ddot{\gamma}_d}{d}, \quad U_3 = \frac{J_{yy} \ddot{\theta}_d}{d}
$$
This ILC framework allows the quadrotor drone to learn from previous flights, but its performance heavily depends on the learning parameters. Poor choices can lead to divergence or slow convergence, especially in noisy environments.
To optimize the ILC parameters, I employ reinforcement learning, specifically Q-learning, which is model-free and suitable for parameter tuning. The agent is the ILC controller, the environment is the quadrotor drone system, states are learning parameter pairs \( s = [K_P, K_D]^T \), and actions are adjustments \( a = [\Delta K_P, \Delta K_D]^T \). The state transition is:
$$
s’ = s + a
$$
An ε-greedy policy balances exploration and exploitation:
$$
\pi_\varepsilon(s) = \begin{cases} \arg\max_a Q(s, a) & \text{with probability } 1-\varepsilon \\ \text{random action} & \text{with probability } \varepsilon \end{cases}
$$
The Q-value is updated using:
$$
Q_{k+1}^\pi(s, a) = Q_k^\pi + \alpha \left( R_{s \to s’}^a + \gamma Q_k^\pi(s’, a’) – Q_k^\pi \right)
$$
where \( \alpha \) is the learning rate, \( \gamma \) is the discount factor, and \( R_{s \to s’}^a \) is the reward. The reward function combines precision and stability metrics for the quadrotor drone. Precision reward encourages error reduction across iterations:
$$
R_{\text{precision}} = \sum_{k=1}^{\text{Iter}-1} \lambda \cdot \frac{\sum_{t=0}^{S_t} |e_{k+1}(t)|}{\sum_{t=0}^{S_t} |e_k(t)|}
$$
with \( \lambda = \alpha \cdot \text{sgn}(\eta) \cdot (1 – \eta) \) if convergence improves, else \( \lambda = \beta \cdot \text{sgn}(\eta) \cdot \frac{1}{1-\eta} \), where \( \eta = 1 – \frac{\sum |e_{k+1}|}{\sum |e_k|} \). Stability reward promotes faster settling time:
$$
R_{\text{stable}} = \sum_{k=1}^{\text{Iter}-1} \mu \cdot \frac{\tau(k+1)}{\tau(k)}
$$
with \( \mu = \varepsilon \cdot \text{sgn}(\xi) \cdot (1 – \xi) \) or \( \mu = \chi \cdot \text{sgn}(\xi) \cdot \frac{1}{1-\xi} \), where \( \xi = 1 – \frac{\tau(k+1)}{\tau(k)} \), and \( \tau(k) \) is the time to reach error below 0.05 m. The total reward is:
$$
R_{s \to s’}^a = R_{\text{precision}} + R_{\text{stable}}
$$
This RL approach automates parameter tuning, adapting to different trajectories and disturbances for the quadrotor drone. The algorithm iterates through episodes, updating the Q-table until optimal parameters are found. Table 2 outlines the RL hyperparameters used in this work.
| Hyperparameter | Value |
|---|---|
| Learning rate (\( \alpha \)) | 0.1 |
| Discount factor (\( \gamma \)) | 0.9 |
| Exploration rate (\( \varepsilon \)) | 0.2 |
| Precision constants (\( \alpha, \beta \)) | 1.0, 0.5 |
| Stability constants (\( \varepsilon, \chi \)) | 0.8, 0.3 |
| Number of episodes | 100 |
| Actions per episode | 50 |
The integration of RL and ILC results in the RL-ILC method, which enhances the quadrotor drone’s trajectory tracking by dynamically optimizing learning gains. This synergy ensures rapid convergence and robustness, making the quadrotor drone capable of handling unknown environments effectively.
To validate the proposed RL-ILC method, I conduct simulation experiments in MATLAB, comparing it with standard ILC (without RL optimization), PID control, and sliding mode control (SMC) for a quadrotor drone. The scenario involves a landing trajectory with random noise disturbances. The desired trajectories for a 60-second flight are:
$$
\begin{aligned}
Z_d(t) &= -120 – \frac{1}{900} t^3 + \frac{1}{10} t^2, \quad t \in [0, 60] \\
Y_d(t) &= \frac{120}{\pi} – \frac{120}{\pi} \cos\left(\frac{\pi}{120} t\right), \quad t \in [0, 60] \\
X_d(t) &= 0.12 t^2 – \frac{1}{750} t^3, \quad t \in [0, 60]
\end{aligned}
$$
The quadrotor drone starts hovering at 120 m altitude. Simulation parameters include a time step of 0.01 s, Euler integration, and initial control inputs as in Table 3. Random noise with a bounded amplitude is added to simulate environmental disturbances.
| Variable | Initial Value |
|---|---|
| \( U_1 \) | 10 N |
| \( U_2, U_3, U_4 \) | 0 |
| \( \theta, \dot{\theta} \) | 0 rad, 0 rad/s |
| \( \gamma, \dot{\gamma} \) | 0 rad, 0 rad/s |
| \( \phi, \dot{\phi} \) | 0 rad, 0 rad/s |
The RL optimization starts with initial learning parameters \( (K_P, K_D) = (0.6, 0.9) \) and searches for optimal values. The Q-learning process yields a set of high-value parameters, as visualized in a scatter plot where each point represents a parameter pair and its associated reward. After training, the optimized parameters for the quadrotor drone are applied in ILC. The performance metrics include total absolute error over time and convergence speed.
Results show that RL-ILC significantly outperforms other methods. For the quadrotor drone, the total error after convergence is reduced to 0.2% of the initial error within 2-3 iterations, whereas standard ILC diverges due to poor parameter choices. PID control exhibits oscillations around the desired trajectory, with errors persisting throughout the flight. SMC converges but has longer settling times and slight chattering. The RL-ILC method maintains precise tracking even under noise, with minimal fluctuations. Table 4 summarizes the comparative analysis for the quadrotor drone trajectory tracking.
| Method | Total Error Reduction | Convergence Iterations | Noise Robustness | Settling Time |
|---|---|---|---|---|
| RL-ILC | 99.8% | 2-3 | High | Fast |
| Standard ILC | Diverges | N/A | Low | N/A |
| PID | ~40% | N/A (continuous) | Medium | Slow |
| SMC | ~90% | N/A (continuous) | High | Moderate |
Further analysis involves examining the time-domain responses. The RL-ILC-controlled quadrotor drone closely follows the desired path in all axes, with errors diminishing rapidly. In contrast, PID shows deviations, especially during trajectory changes, and SMC has overshoots. The robustness of RL-ILC is evident in its ability to reject random noise without significant trajectory波动, a critical advantage for real-world quadrotor drone applications. The integration of RL for parameter tuning ensures that the ILC adapts to the specific task, making the quadrotor drone more autonomous and reliable.
The success of RL-ILC hinges on its iterative learning mechanism, which leverages historical data to refine feedforward inputs. For the quadrotor drone, this means that repeated flights along similar paths, such as landing or inspection routes, can be optimized over time. The reinforcement learning component automates the tedious process of parameter tuning, which is often done manually or via heuristic methods. This not only improves accuracy but also reduces design time for quadrotor drone controllers. Moreover, the modular architecture allows easy integration into existing flight control systems, as it adds a feedforward layer without altering the core feedback loop. This is particularly beneficial for commercial quadrotor drones, where stability and safety are paramount.
In terms of computational load, the RL optimization is performed offline or during initial training phases, so it does not burden the real-time control of the quadrotor drone. The ILC updates are lightweight, involving simple matrix operations, making them suitable for onboard implementation. Thus, the proposed method balances performance and practicality for quadrotor drone operations. Future work could explore deep reinforcement learning for more complex parameter spaces or extend RL-ILC to multi-drone formations. Additionally, experimental validation with physical quadrotor drones would further confirm the method’s efficacy in dynamic environments.
In conclusion, this work presents a novel RL-ILC framework for enhancing trajectory tracking in quadrotor drones. By combining iterative learning control with reinforcement learning-based parameter optimization, the method achieves fast convergence, high precision, and robustness to disturbances. Simulations demonstrate its superiority over conventional approaches, paving the way for more intelligent and adaptive quadrotor drone systems. The key insights are that RL automates ILC tuning, overcoming a major practical hurdle, and that feedforward-feedback fusion significantly boosts performance. As quadrotor drones continue to evolve, such hybrid control strategies will be crucial for unlocking their full potential in diverse applications.