Unmanned Aerial Vehicles (UAVs), with their advantages of flexibility, intelligence, and low cost, are widely used in civilian and military fields. However, they also pose new threats to public safety, event security, and battlefield air defense. To address this challenge, anti-UAV detection has become a current research hotspot, primarily involving radar detection, radio monitoring, and electro-optical identification technologies. Among these, infrared imaging sensors in electro-optical equipment offer significant value in anti-UAV missions under complex environments due to their advantages of all-day, all-weather operation and long detection range.
Infrared target tracking is a key technology in infrared-based anti-UAV detection, aiming to continuously locate and estimate the size of a specified moving target in an infrared video, which falls under the category of single-object tracking research. For anti-UAV tasks in complex backgrounds, infrared target tracking algorithms must solve the following problems to achieve higher performance:
- Compared to visible light, infrared targets possess fewer discriminative features. Especially when the UAV is at a long distance, the target appears as a small, weak signature, making it highly susceptible to background clutter interference.
- In complex environments (e.g., trees, buildings, clouds), infrared targets are prone to occlusion and thermal crossover (TC), which can lead to tracking failure.
- Platform motion and the high-speed maneuvering of UAVs cause drastic changes in the target’s position within the video frame, potentially even moving it out of the field of view.
- Practical applications require algorithms capable of tracking targets in real-time.
Effectively addressing these challenges is critical for deploying reliable anti-UAV systems.
Currently, numerous high-performance tracking algorithms have been proposed. Based on the search area size within a video frame, these algorithms can be categorized into local trackers and global trackers. Local trackers crop a local image patch from the current frame as the search region based on the target’s location and size from the previous moment. This helps reduce computational load and minimize background interference. Recent algorithms like STARK, OSTrack, and SeqTrack leverage the powerful attention mechanism of Transformers to enhance discrimination of challenging targets, enabling successful tracking of small objects. However, local trackers depend on stable tracking; when the target disappears, they can easily select an incorrect local search region, leading to subsequent failure. While algorithms like LTMU and UTTracker add a global search module to re-capture reappearing targets, this often results in overly complex structures, severely reduced operational efficiency, and necessitates multi-stage training. In summary, local trackers only effectively address problem (1) and are not suitable for infrared UAV tracking in complex backgrounds.
Global tracking algorithms use the entire current video frame as the search area. Their advantage lies in autonomously performing a global search to re-capture disappeared targets, demonstrating stronger tracking robustness against problems (2) and (3), making them more widely applicable for infrared UAV tracking tasks. Existing global trackers are often based on object detection frameworks. For instance, GlobalTrack and SiamDT adopt a two-stage anchor-based framework like Faster R-CNN, while other works have used one-stage anchor-based frameworks like YOLO. These algorithms search for the target by fusing features from the initial target template with the global spatial features of the current frame, often neglecting the temporal features contained in the video sequence, which limits model performance to some extent. Furthermore, a larger search area introduces more background clutter or distractor interference. Especially in complex backgrounds, global trackers are susceptible to such interference, leading to tracking failure. Additionally, to accurately track infrared targets of different scales, global algorithms typically extract features from different depths—using high-resolution, low-level texture features for small targets and low-resolution, high-level semantic features for large targets. This “multi-level” approach, originally designed for detecting objects of various sizes in an image for detection tasks, introduces significant computational redundancy for the single, continuously size-varying target in tracking, reducing operational efficiency.
To address these issues with global trackers in complex background infrared UAV tracking, we propose a robust tracking algorithm. The main contributions are summarized in the following table:
| Component | Key Innovation | Addresses Challenge |
|---|---|---|
| Spatio-Temporal Global Tracker (STGT) | One-stage anchor-free framework with multi-level structure and ConvLSTM-based spatio-temporal fusion. | Target disappearance (occlusion/out-of-view), multi-scale targets, enhances feature discriminability. |
| Target Enhancement & Interference Suppression (TEIS) | Online memory of target/distractor features for adaptive weighting of score maps. | Background clutter and distractor interference. |
| Dynamic Hierarchical Acceleration (DHA) | Prunes redundant feature levels based on previous frame’s confidence. | Real-time performance requirement, computational efficiency. |
Proposed Methodology: The STGT++ Framework
Our proposed robust infrared UAV tracking algorithm, termed STGT++, is built upon a Spatio-temporal Global Tracker (STGT) core, augmented with a Target Enhancement and Interference Suppression (TEIS) module and a Dynamic Hierarchical Acceleration (DHA) method. The STGT is an end-to-end trainable deep learning model, while TEIS and DHA are applied during the inference/testing phase to enhance robustness and speed, respectively.
1. Spatio-Temporal Global Tracker (STGT) Model
The STGT model architecture comprises three main parts: spatial feature extraction, spatio-temporal feature fusion, and prediction heads. Compared to the baseline GlobalTrack, it offers key improvements: (1) Adopting a simpler, more efficient one-stage anchor-free structure, requiring only one feature fusion step; (2) Effectively integrating temporal features from historical frames with current spatial features via a memory network.
Spatial Feature Extraction: A Siamese network structure with a parameter-shared ResNet-50 backbone extracts features from the initial template frame (Frame 1) and the current search frame (Frame t). A shared Feature Pyramid Network (FPN) performs multi-level feature fusion and enhancement. The final output consists of 4 levels of spatial features for both template $z_i \in \mathbb{R}^{C \times H_i \times W_i}$ and search region $x^t_i \in \mathbb{R}^{C \times H_i \times W_i}$, where $i \in \{0,1,2,3\}$ denotes the level, $C$ is the channel dimension (256), and $H_i, W_i$ are the feature map dimensions. These levels are designed to handle targets of different size ranges: $[0,10)$, $[10,20)$, $[20,40)$, and $[40, +\infty)$ pixels, ensuring small targets leverage high-resolution textures and large targets leverage rich semantics.
Spatio-Temporal Feature Fusion (STFF) Module: This module fuses the spatial features of the current frame with spatio-temporal features from past frames. For each level $i$, the process involves:
- Template Feature Extraction: Using Region-of-Interest Align (RoIAlign), target region features $z^*_i \in \mathbb{R}^{C \times 7 \times 7}$ are cropped from the initial template features $z_i$ based on the first-frame bounding box annotation.
- Temporal Fusion via Memory: The current frame feature $x^t_i$, the template $z^*_i$, a memory state $c^{t-1}_i$, and a hidden state $h^{t-1}_i$ are fed into the STFF module. A Convolutional Long Short-Term Memory network (ConvLSTM) is employed to fuse historical temporal information stored in $c^{t-1}_i$ with the current spatial features $x^t_i$. The ConvLSTM equations govern the update:
$$
\begin{aligned}
(f^t_i, i^t_i, o^t_i, \tilde{c}^t_i) &= \mathcal{A}(\mathcal{S}(\mathcal{C}^{3\times3}_i([h^{t-1}_i, x^t_i])) \\
c^t_i &= f^t_i \odot c^{t-1}_i + i^t_i \odot \tilde{c}^t_i \\
h^t_i &= o^t_i \odot \tanh(c^t_i)
\end{aligned}
$$
where $[\cdot]$ denotes concatenation, $\mathcal{C}^{3\times3}_i$ is a $3\times3$ convolution layer, $\mathcal{S}$ splits the tensor, $\mathcal{A}$ represents activation functions, and $\odot$ is element-wise multiplication. The output $h^t_i$ is the temporally enhanced feature. - Spatial Fusion with Template: The template features $z^*_i$ are fused with $h^t_i$ to perform the target matching search in the current frame. This is achieved via a depth-wise correlation-like operation and a crucial concatenation step:
$$
\tilde{x}^t_i = \mathcal{C}^{1\times1}_i([\mathcal{C}^{7\times7}_i(z^*_i) \otimes h^t_i, h^t_i])
$$
where $\mathcal{C}^{7\times7}_i$ converts $z^*_i$ into a $C\times1\times1$ kernel for correlation $\otimes$, and $\mathcal{C}^{1\times1}_i$ adjusts channels. Concatenating $h^t_i$ provides additional contextual background information to the model, aiding robustness in complex anti-UAV scenes.
Prediction Heads (Output Module): Adopting and simplifying the FCOS (Fully Convolutional One-Stage) detection head for tracking. Two parallel branches—a classification branch and a regression branch—are attached to each fused feature $\tilde{x}^t_i$. Each branch consists of two “Conv-GroupNorm-ReLU” (CGR) blocks followed by a final prediction layer. The classification branch outputs a target score map $s^t_i \in \mathbb{R}^{1 \times H_i \times W_i}$, and the regression branch outputs a bounding box tensor $b^t_i \in \mathbb{R}^{4 \times H_i \times W_i}$ (representing distances to the top, bottom, left, and right boundaries). The total training loss $L_{total}$ is computed over $n$ consecutive frames in a sample and all 4 levels:
$$
L_{total} = \frac{1}{4n} \sum_{t=1}^{n} \sum_{i=0}^{3} \left( L_{cls}(s^t_i, s^{t*}_i) + L_{reg}(b^t_i, b^{t*}_i) \right)
$$
where $L_{cls}$ is Focal Loss, $L_{reg}$ is GIoU Loss, and $s^{t*}_i, b^{t*}_i$ are ground-truth values.
2. Target Enhancement and Interference Suppression (TEIS) Module
The global search region inevitably introduces background distractors. The TEIS module post-processes the raw score maps $s^t_i$ from STGT to suppress distractors and enhance the true target region. It operates by maintaining a dynamic memory of target and distractor features from the previous frame, $\tilde{x}^{t-1}_i = \{\tilde{x}^{t-1}_i(T), \tilde{x}^{t-1}_i(I)\}$, where $\tilde{x}^{t-1}_i(T)$ is the target feature and $\tilde{x}^{t-1}_i(I)$ contains $k-1$ distractor features.
Enhancement and Suppression:
- Similarity Computation: Compute cosine similarity maps between the stored features and every spatial location in the current frame’s feature $x^t_i$:
$$
\begin{aligned}
cs^t_i(T)(p,q) &= \frac{\tilde{x}^{t-1}_i(T) \cdot x^t_i(p,q)}{\|\tilde{x}^{t-1}_i(T)\| \|x^t_i(p,q)\|} \\
cs^t_i(I)(r,p,q) &= \frac{\tilde{x}^{t-1}_i(I)(r) \cdot x^t_i(p,q)}{\|\tilde{x}^{t-1}_i(I)(r)\| \|x^t_i(p,q)\|}
\end{aligned}
$$
where $(p,q)$ are spatial coordinates and $r$ is the distractor index. $cs^t_i(T)$ highlights regions similar to the target, while $cs^t_i(I)$ highlights regions similar to known distractors. - Weight Map Generation: A single distractor response map is obtained by taking the element-wise maximum across all $k-1$ distractor similarity maps. The final weight map $w^t_i$ is derived by subtracting this combined distractor response from the target similarity map and normalizing:
$$
\begin{aligned}
\tilde{w}^t_i(p,q) &= cs^t_i(T)(p,q) – \max_{r=1 \sim (k-1)} (cs^t_i(I)(r,p,q)) \\
w^t_i &= \frac{\tilde{w}^t_i – \min(\tilde{w}^t_i)}{\max(\tilde{w}^t_i) – \min(\tilde{w}^t_i)}
\end{aligned}
$$ - Score Map Refinement: The refined score map $\hat{s}^t_i$ is obtained by element-wise multiplication: $\hat{s}^t_i = s^t_i \odot w^t_i$. This suppresses areas with high distractor similarity while preserving or enhancing areas with high target similarity.
Feature Memory Update: After obtaining the refined scores $\hat{s}^t_i$, the top-$k$ candidate locations with the highest scores are identified. If the location with the global maximum score $\hat{s}^t_{max}$ (the predicted target) is within this list, it is removed. Otherwise, the last candidate in the list is removed. This yields $k-1$ distractor locations. The corresponding features are then extracted from $x^t_i$ at the target and distractor locations to form the updated memory $\tilde{x}^{t}_i$ for the next frame. An adaptive threshold $(0.5 \cdot \hat{s}^{1}_{max})$, based on the score from the first frame, determines if the target is present to control the update process, preventing erroneous updates when the target is lost.
3. Dynamic Hierarchical Acceleration (DHA) Method
While the multi-level structure is necessary for scale invariance, processing all four levels for a single target every frame is redundant. The DHA method dynamically selects which levels to compute in the current frame $t$ based on the tracking confidence and the active level from the previous frame $t-1$. The core logic is summarized in the table below:
| Previous Max Score $\hat{s}^{t-1}_{max}$ | Strategy | Expected Speed Gain |
|---|---|---|
| $[0, s_1)$ (Low Confidence) | Compute all 4 levels to prevent target loss. | Low |
| $[s_1, s_2]$ (Medium Confidence) | Compute the previous active level $i$ and its adjacent level(s) ($i-1$ and/or $i+1$) to handle scale change. | Medium |
| $(s_2, 1]$ (High Confidence) | Compute only the previous active level $i$, assuming stable scale. | High |
where $s_1=0.3$ and $s_2=0.7$ are thresholds. This method significantly reduces FLOPs (Floating Point Operations) and improves the overall frame rate, making real-time anti-UAV tracking feasible.
Experiments and Analysis
We evaluate our STGT++ algorithm on the challenging Anti-UAV datasets, using the 3rd Anti-UAV train set for training and the 1st, 2nd test-dev sets and 3rd validation set for testing. Performance is measured using standard One-Pass Evaluation (OPE) metrics: Precision (P) for location accuracy (threshold of 20 pixels) and Success (AUC, Area Under the Curve) for bounding box overlap accuracy.
Quantitative Comparison
We compare STGT++ against 14 state-of-the-art trackers, including 9 local trackers (ATOM, DiMP, PrDiMP, KYS, STARK, TOMP, OSTrack, SeqTrack, AQATrack) and 5 trackers with re-detection capability (DaSiamRPN, GlobalTrack, LTMU, SiamSTA, SiamDT). All were retrained on the same dataset for fairness. The key results are summarized below:
| Algorithm | 1st Anti-UAV | 2nd Anti-UAV | 3rd Anti-UAV | |||
|---|---|---|---|---|---|---|
| P (%) | AUC (%) | P (%) | AUC (%) | P (%) | AUC (%) | |
| GlobalTrack | 97.5 | 75.6 | 89.3 | 65.5 | 74.7 | 53.0 |
| SiamDT | 97.7 | 76.4 | 89.4 | 68.5 | — | — |
| OSTrack (Local) | 93.6 | 72.4 | 83.4 | 62.7 | 68.7 | 51.9 |
| STGT++ (Ours) | 98.0 | 76.5 | 92.4 | 69.4 | 78.7 | 56.5 |
Our STGT++ algorithm achieves the best performance across all datasets and metrics, demonstrating superior robustness in complex anti-UAV scenarios. It notably outperforms the baseline GlobalTrack, especially on the more challenging 2nd and 3rd datasets, with gains of up to 3.1% in Precision and 3.5% in AUC.
Ablation Study and Analysis
Ablation experiments on the 3rd Anti-UAV val set validate the contribution of each component:
| Model Configuration | P (%) | AUC (%) | Speed (fps) |
|---|---|---|---|
| STGT (w/ FF only, w/o LSTM) | 76.8 | 55.3 | 30.5 |
| + ConvLSTM Temporal Fusion | 77.4 | 55.8 | 20.8 |
| + TEIS Module | 78.8 | 56.6 | 18.7 |
| + DHA Method (STGT++) | 78.7 | 56.5 | 26.9 |
Key Findings:
- The one-stage anchor-free base (STGT w/ FF) already surpasses GlobalTrack in speed and matches performance, proving its efficiency.
- The ConvLSTM-based STFF module provides a consistent performance gain (e.g., +0.6% P), validating the benefit of spatio-temporal feature fusion for anti-UAV tracking.
- The TEIS module delivers the most significant performance boost (+1.4% P, +0.8% AUC), demonstrating its critical role in suppressing complex background interference.
- The DHA method recovers most of the speed loss from adding advanced modules, achieving a real-time frame rate of 26.9 fps with a negligible performance drop (0.1%), striking an optimal balance for practical deployment.
Further Analysis:
- Scale-wise Performance: STGT++ shows robust performance across all target sizes, with particular strength in tracking small targets ($<$20 pixels), where P improves from 62.4% (STGT) to 65.1% thanks to TEIS.
- Parameter Sensitivity: The number of CGR blocks in the head was optimized to 2 for this task. The distractor memory size $k$ in TEIS was set to 20 (1 target + 19 distractors) for optimal performance.
- Efficiency: Compared to other global trackers, STGT++ offers a superior speed-accuracy trade-off. It significantly outperforms complex models like LTMU (1.5 fps) and is faster than SiamDT (9.1 fps) while being more accurate.
Conclusion
In this work, we present a robust infrared target tracking algorithm, STGT++, designed to address the key challenges in complex background anti-UAV applications. The core innovation lies in a cohesive framework that integrates a one-stage global tracker with spatio-temporal feature learning, an online distractor suppression mechanism, and an adaptive inference accelerator. Our comprehensive evaluations on the challenging Anti-UAV benchmark demonstrate that the proposed algorithm sets a new state-of-the-art in tracking accuracy and success rate. Crucially, it achieves this while maintaining a real-time processing speed, making it a highly practical and effective solution for real-world infrared-based anti-UAV defense systems. Future work may explore more efficient backbone architectures and adaptive schemes for the TEIS and DHA parameters to further enhance generalization across diverse operational environments.