Object tracking has become a pivotal technology for unmanned aerial vehicles (UAVs or drones) due to their increasing applications in both military and civilian domains. The unique advantages of drones, such as their compact size, rapid response, and strong concealment, make them ideal platforms for surveillance, reconnaissance, and delivery. However, the very nature of drone operations—characterized by dynamic flight patterns, complex backgrounds like urban canyons or dense vegetation, and frequent target occurrences of occlusion, deformation, scale variation, and multi-view changes—poses significant challenges for traditional tracking algorithms. Therefore, the development of stable and efficient object tracking algorithms is of paramount importance for advancing drone capabilities.
In recent years, deep learning models have revolutionized computer vision, including the field of object tracking. Among these, the Transformer network, renowned for its powerful global feature modeling capacity, has been widely adopted. Trackers like OSTrack and SwinTrack leverage Transformer architectures to effectively fuse spatio-temporal features, capturing target representations across frames. Methods such as MixFormer employ a unified Vision Transformer backbone for joint feature extraction and fusion, enhancing performance and stability. Nonetheless, these Transformer-based trackers primarily rely on attention mechanisms for global feature interaction between the template and search region. They often lack dedicated modeling of local structures and fine-grained details, making them susceptible to performance degradation in complex drone scenarios involving severe occlusion, deformation, and scale changes. Furthermore, while online template update strategies in some trackers mitigate appearance variations, they often involve complex appearance score prediction modules to assess the suitability of the current frame for updating the template. This increases model parameters and computational cost, which is undesirable for real-time drone applications.
To address these shortcomings, recent works have explored frequency-domain features. For instance, integrating Gabor filter features with correlation filter models in the frequency domain can extract multi-scale target characteristics. Another approach combines frequency-domain filtering with CNN architectures to enhance the capture of edges and textures. However, these methods often rely on fixed filter banks or local convolutional operations, struggling to adapt to diverse scenarios or model global context effectively, thus leaving room for improvement in tracking accuracy for drone videos.
In summary, to tackle the challenges of occlusion, deformation, scale variation, and multi-view changes prevalent in UAV drone tracking, this paper proposes a novel tracking algorithm. Our core contributions are twofold. First, we design a hybrid feature extraction network. A distilled, lightweight Transformer network extracts robust global spatial features. These features are then fed into an Adaptive Frequency-Domain Perception Network (AFDPN) to supplement local detail features related to target texture and edges. Concurrently, we introduce a learnable image as an additional input to capture the correlation between the target template and the search region, which is used to update the initial template, thereby enhancing target representation and adaptability. Second, we propose a Multi-view Invariant Feature Learning Strategy based on Mutual Information Maximization. By maximizing the mutual information between the template and search image features, we design a novel loss function that guides the network to learn feature representations that are invariant to viewpoint changes, optimizing the tracking model. Finally, the target position is determined by a prediction head based on the feature response of the learnable image.
Proposed UAV Drone Tracking Algorithm
The proposed algorithm architecture is designed to handle the complexities of UAV-based tracking. The overall network structure comprises a Deep Network Module and an Adaptive Frequency-Domain Perception Network Module, as illustrated in Figure 1.
The Deep Network is based on a 12-layer Transformer encoder, which is made lightweight through dual-path distillation technology. The shallow distillation path uses the dense regression head output of the teacher model as a soft label, guiding the student model’s sparse prediction head via KL divergence to learn the localization distribution. The deep distillation employs a progressive pruning and feature fitting joint optimization strategy, gradually degenerating the weights of layers to be pruned while ensuring feature consistency, ultimately yielding a lightweight student model. At the input stage, alongside the template image $z$ and search image $x$, we introduce a learnable image $l$ initialized from the search image. This learnable image is initialized as four feature vectors. During feature extraction, the learnable image features progressively compress target information, providing an effective feature representation for subsequent regression and classification tasks.
The Adaptive Frequency-Domain Perception Network Module is constructed by stacking 6 layers, each consisting of a Frequency-Domain Feature Interaction layer, a Feed-Forward Network layer, and a Fully Connected layer. Its primary function is to extract and fuse fine-grained frequency-domain features, complementing the global features from the Transformer. Let $t_z$, $t_l$, and $t_x$ denote the feature sequences for the template image, learnable image, and search image output by the deep network, respectively, where $t_i \in \mathbb{R}^{N_i \times D}$, $i \in \{z, l, x\}$, $D=768$ is the hidden dimension, and $N_l=4$. $N_z$ and $N_x$ are calculated based on the input image size $(H_i, W_i)$ and patch resolution $P$:
$$ N_i = \frac{H_i \times W_i}{P^2}, \quad i \in \{z, x\} $$
First, all feature sequences are transformed into the frequency domain via a 2D Discrete Fourier Transform (2D FFT) along the spatial dimensions:
$$ \mathcal{T}_i = \mathcal{F}[t_i], \quad i \in \{z, l, x\} $$
where $\mathcal{F}[\cdot]$ denotes the 2D FFT, and $\mathcal{T}_i$ is a complex tensor representing the spectrum.
Subsequently, an adaptive learnable filter $\mathcal{H}(f_x, f_y; W, b)$ is introduced in the frequency domain. The filter is initialized using the template image spectrum:
$$ \mathcal{H}^{(0)}(f_x, f_y) = \sigma(W_0 \cdot \mathcal{T}_z(f_x, f_y) + b_0) $$
where $W_0$ and $b_0$ are initialization parameters, and $\sigma(\cdot)$ is the Sigmoid activation function. During training, the filter adaptively adjusts by combining the frequency-domain features of the template and learnable images through shared learnable parameters:
$$ \mathcal{H}(f_x, f_y) = \lambda \mathcal{H}^{(0)}(f_x, f_y) + (1-\lambda) \sigma\left(W \cdot [\mathcal{T}_z(f_x, f_y), \mathcal{T}_l(f_x, f_y)] + b\right) $$
Here, $[\cdot]$ denotes feature concatenation, and $W$, $b$, and $\lambda$ are learnable parameters. This mechanism allows the filter to dynamically learn cross-image frequency correlations, effectively fusing structural information. The learnable image provides auxiliary information related to the target, helping the network better understand appearance changes and focus on local detail regions.
The filter is then applied to optimize the frequency-domain features, enhancing target-relevant components (e.g., high-frequency details) while suppressing redundant background information:
$$ \mathcal{T}’_i(f_x, f_y) = \mathcal{H}(f_x, f_y) \odot \mathcal{T}_i(f_x, f_y), \quad i \in \{l, x\} $$
where $\odot$ denotes element-wise multiplication. An inverse transform is applied to obtain enhanced spatial-domain features:
$$ t’_x = \mathcal{F}^{-1}[\mathcal{T}’_x], \quad t’_l = \mathcal{F}^{-1}[\mathcal{T}’_l], \quad t’_z = \mathcal{F}^{-1}[\mathcal{T}’_z] $$
These enhanced features ($t’_x$, $t’_l$, $t’_z$) are further processed through feed-forward and fully connected layers for final fusion.
For training the feature extraction network, we use a Gaussian-weighted focal loss $L_{cls}$ for classification supervision. Let the true target center be $\hat{c} = (\hat{\phi}_c, \hat{\psi}_c)$ in the feature map. The corresponding Gaussian-weighted ground-truth map $\hat{C}$ is defined as:
$$ \hat{C}_{\phi\psi} = \exp\left(-\frac{(\phi – \hat{\phi}_c)^2 + (\psi – \hat{\psi}_c)^2}{2\sigma^2}\right) $$
where $\sigma$ is an adaptive standard deviation based on the target scale. The Gaussian-weighted focal loss is:
$$ L_{cls} = \sum_{\phi,\psi} \left[ \alpha (1 – \hat{C}_{\phi\psi})^\beta \mathbb{I}(\hat{C}_{\phi\psi}=1) \log(C_{\phi\psi}) + (1-\alpha) \hat{C}_{\phi\psi}^\beta \mathbb{I}(\hat{C}_{\phi\psi}<1) \log(1-C_{\phi\psi}) \right] $$
where $\mathbb{I}(\cdot)$ is the indicator function, and $\alpha=2$, $\beta=4$ are hyperparameters.
For bounding box regression, a combined loss of $L_1$ and Generalized IoU (GIoU) loss is used:
$$ L_{1} = |b – \hat{b}|, \quad L_{GIoU} = 1 – GIoU(b, \hat{b}) $$
where $b$ is the predicted box and $\hat{b}$ is the ground-truth box. The total training loss is:
$$ L_{total} = \lambda_{L1} L_{1} + \lambda_{GIoU} L_{GIoU} + \lambda_{cls} L_{cls} $$
with $\lambda_{L1}=1$, $\lambda_{GIoU}=5$, $\lambda_{cls}=2$.
Multi-view Invariant Feature Learning via Mutual Information Maximization
During drone flight, the camera viewpoint constantly changes. Severe changes in the drone’s own attitude and shooting angle can lead to significant target deformation, occlusion, and scale variation in the video, often causing tracking failure. While current state-of-the-art trackers employ template update strategies to handle appearance changes, these typically involve confidence estimation and update frequency control, introducing additional computational overhead and increasing model complexity, which is less suitable for lightweight applications on UAV drone platforms.
To address this, we propose a Multi-view Invariant Feature Learning Strategy based on Mutual Information Maximization. The core idea is to enhance the cross-view correlation of target features between the template and search images, thereby learning a feature representation that is invariant to viewpoint changes. This is achieved by modeling the statistical dependency between features under different views, extracting the essential characteristics independent of the observation perspective.
Let $x$ and $z$ represent the search and template images, respectively. Their mutual information $I(x, z)$ is defined as the Kullback-Leibler (KL) divergence between their joint distribution $p(x, z)$ and the product of their marginals $p(x)p(z)$:
$$ I(x, z) = D_{KL}(p(x, z) \| p(x)p(z)) = \mathbb{E}_{p(x,z)} \left[ \log \frac{p(x, z)}{p(x)p(z)} \right] $$
However, KL divergence is sensitive to non-overlapping support and can suffer from vanishing gradients. Therefore, we adopt the more stable MINE (Mutual Information Neural Estimation) estimator based on the Donsker-Varadhan representation, which uses a neural network $T_{\theta}$ parameterized by $\theta$:
$$ \hat{I}(x, z) = \sup_{\theta} \left( \mathbb{E}_{p(x,z)}[T_{\theta}(x, z)] – \log \mathbb{E}_{p(x)p(z)}[e^{T_{\theta}(x, z)}] \right) $$
In our context, as shown in Figure 3, let $t_z$ and $t_x$ be the feature sequences for the template and search image. Given the ground-truth target location, we extract a subset $t’_{x} \subset t_x$ from the search features corresponding strictly to the target region using bilinear interpolation. We then compute the mutual information loss $S_{vir}$ as:
$$ S_{vir} = – \hat{I}(t_z, t’_{x}) $$
By minimizing $S_{vir}$, we drive the network to maximize the mutual information between the template and the target region in the search image. This encourages the model to learn features that remain consistent across different views and appearance changes. Crucially, $S_{vir}$ is computed only during training and adds no computational cost during inference, maintaining model efficiency while effectively handling the challenges posed by multi-view variations in UAV drone tracking.
Simulation Experiments and Analysis
Experimental Setup
Experiments were conducted on a system with Ubuntu 20.04, Python 3.8, and PyTorch 2.0.0. Model training was performed using an NVIDIA RTX 4090 GPU, while inference speed was evaluated on an NVIDIA RTX 3080 Ti GPU. Training was conducted in two stages: 450 epochs for the backbone network and 50 epochs for the Adaptive Frequency-Domain Perception Network. Common data augmentation techniques like horizontal flipping and brightness adjustment were applied. The optimizer was AdamW with an initial learning rate of $1 \times 10^{-3}$ and weight decay of $1 \times 10^{-4}$. The template image size was set to $128 \times 128$ pixels, and the search region size was $288 \times 288$ pixels.
The training datasets included GOT-10K, COCO2017, and LaSOT. For evaluation, we used the UAV123 dataset (specifically designed for drone tracking with 123 sequences), OTB70, and the test set of LaSOT. The primary evaluation metrics were Precision (P, the percentage of frames where the center location error is less than 20 pixels), Success Rate (SR, the area under the curve of the overlap ratio between predicted and ground-truth bounding boxes), and the success rate at an overlap threshold of 0.5 ($SR_{0.5}$). The Average Overlap (AO) on GOT-10K and Frames Per Second (FPS) were also reported to measure accuracy and real-time performance, critical for UAV drone applications.
Ablation Study
To validate the effectiveness of the proposed components, ablation studies were conducted on the GOT-10K dataset, with results summarized in Table 1.
| Exp. No. | Standard Update | AFDPN | Multi-view Strategy | Learnable Image Update | Params (M) | AO (%) | SR0.5 (%) | FPS |
|---|---|---|---|---|---|---|---|---|
| 1 | √ | × | × | × | 20 | 71.7 | 80.2 | 165 |
| 2 | √ | √ | × | × | 26 | 73.3 | 83.4 | 152 |
| 3 | √ | √ | √ | × | 27 | 74.5 | 84.8 | 151 |
| 4 | × | √ | √ | √ | 24 | 74.6 | 84.2 | 156 |
Experiment 2, which introduces the Adaptive Frequency-Domain Perception Network (AFDPN), shows an improvement of 1.6% in AO and 3.2% in $SR_{0.5}$ over Experiment 1, demonstrating its effectiveness in extracting fine-grained details to handle occlusion and deformation, despite a minor FPS drop. Experiment 3, which adds the Multi-view Invariant Feature Learning Strategy, achieves further gains of 1.2% in AO and 1.4% in $SR_{0.5}$ with negligible parameter and speed overhead, confirming its value for learning viewpoint-robust features. Finally, Experiment 4 replaces the complex standard update module with the simpler learnable image update mechanism. It maintains comparable accuracy (74.6% AO) while reducing parameters to 24M and increasing FPS to 156, striking an optimal balance between performance and efficiency for real-time UAV drone tracking.
Comparative Experiments
We compared our algorithm against several state-of-the-art trackers on OTB70, LaSOT, and UAV123 datasets. The results are presented in Table 2.
| Tracker | OTB70 | LaSOT | UAV123 | FPS | |||
|---|---|---|---|---|---|---|---|
| Prec. | Succ. | Prec. | Succ. | Prec. | Succ. | ||
| Ours | 82.1 | 63.9 | 66.5 | 64.0 | 90.2 | 68.8 | 156 |
| F-Resnet | – | – | 62.0 | 59.9 | 77.2 | – | – |
| SpectralTracker | – | – | 69.1 | 65.7 | 78.2 | 67.4 | 52 |
| IRE-UFT | – | – | – | – | 69.1 | 65.3 | – |
| SMAT | 81.9 | 63.8 | 64.6 | 61.7 | 81.8 | 64.6 | 124 |
| DiMP | – | – | 56.7 | 56.9 | – | 65.4 | 77 |
| TCTrack | 81.2 | 62.2 | – | – | 80.8 | 60.6 | 139 |
| ABDNet | 76.8 | 59.6 | – | – | 79.3 | 60.7 | 130 |
| FEAT-L | – | – | 60.9 | 57.9 | 86.6 | 65.8 | 80 |
| HCAT | 60.6 | 58.3 | 60.5 | 59.0 | 63.6 | – | 195 |
| FDSST | 53.4 | 35.7 | – | – | 58.3 | 40.5 | – |
| AutoTrack | 71.6 | 47.8 | – | – | 68.9 | 47.2 | 58 |
Performance Analysis: On the OTB70 dataset, our algorithm achieves leading precision (82.1%) and success rate (63.9%). On the long-term LaSOT dataset, it obtains competitive results (66.5% precision, 64.0% success), second only to SpectralTracker which is significantly slower (52 FPS). Most importantly, on the UAV123 dataset, which simulates real drone challenges, our method achieves the highest precision (90.2%) and success rate (68.8%), demonstrating superior robustness to the specific difficulties encountered in UAV drone tracking.
Efficiency Analysis: Our tracker runs at 156 FPS, which, while slightly lower than HCAT (195 FPS), strikes an excellent balance by delivering significantly higher accuracy across all benchmarks. This makes it highly suitable for real-time applications on drone platforms where both accuracy and speed are crucial.
Qualitative results on challenging UAV123 sequences (person-21, car-6, wakeboard-5, uav-8) further demonstrate the robustness of our method. In scenarios with occlusion by buildings, severe scale variation, deformation, and low resolution, our tracker maintains accurate and stable bounding boxes, whereas comparators like SMAT and MixformerV2-S often suffer from drift or complete failure. This visually confirms the effectiveness of the adaptive frequency-domain features and the multi-view invariant learning strategy in handling the complex conditions typical of UAV drone operations.
Conclusion
This paper proposed a novel object tracking algorithm designed to address the key challenges in UAV drone applications: occlusion, deformation, scale variation, and multi-view changes. The main contributions are as follows:
1) A Hybrid Feature Extraction Network: We designed an Adaptive Frequency-Domain Perception Network (AFDPN) based on frequency-domain filtering theory and integrated it with a distilled Transformer-based deep network. The introduction of a learnable image at the input promotes deep feature fusion between the search region and the initial template. This architecture effectively extracts discriminative target features, including textures and edges, significantly enhancing the model’s capability to handle target occlusion, deformation, and scale variation common in drone footage.
2) A Multi-view Invariant Feature Learning Strategy: We proposed a novel strategy based on Mutual Information Maximization. By quantifying and maximizing the similarity between template and search image features, the network is guided to learn representations that are invariant to viewpoint changes. This strategy effectively mitigates tracking failures caused by the severe appearance variations induced by the drone’s dynamic movement and viewing angles, markedly improving tracking accuracy in complex environments.
Extensive experiments on benchmarks including UAV123, OTB70, and LaSOT demonstrate that our algorithm achieves state-of-the-art or highly competitive performance in terms of both precision and success rate, while maintaining a high inference speed suitable for real-time drone applications. The ablation studies confirm the individual and synergistic contributions of the proposed components. The proposed method provides a robust and efficient solution for vision-based object tracking on UAV drone platforms.