We present a multimodal detection framework that fuses infrared video and acoustic signals to improve small unmanned aerial vehicle (UAV) identification in complex environments. The core of our approach lies in a track-before-detect strategy for infrared data, combined with a convolutional neural network (CNN) for audio spectrogram analysis, followed by decision-level fusion. For infrared video, we design a dynamic saliency difference enhancement module that integrates gradient–grayscale features, motion cues, local entropy-guided window scaling, and temporal consistency verification to amplify weak target responses. A spatiotemporal trajectory encoding and association module is introduced to handle occlusion and trajectory breakages using an LSTM network and a dynamic fusion factor. For audio, we extract log-mel spectrograms and feed them to a CNN classifier. Experiments on a public multi-sensor drone dataset demonstrate that our method achieves a precision of 89.5%, recall of 85.7%, and average precision of 75.4%, outperforming several state-of-the-art single‑modality and fusion baselines. Both ablation and comparison studies validate the effectiveness of each proposed component in advancing drone technology.
1. Introduction
The proliferation of drone technology has brought unprecedented convenience to civilian and military applications, such as surveillance, agriculture, and logistics. However, the unauthorized or malicious use of small UAVs poses serious risks to critical infrastructure, airports, and public safety. Reliable detection of these small, agile targets is therefore an urgent requirement. Unlike larger aircraft, UAVs exhibit low radar cross section, high maneuverability, and can easily be confused with birds, making detection a challenging task in the domain of drone technology.
Visible-light cameras suffer from illumination variations and occlusion, while radar systems may be expensive or limited in low-altitude scenarios. Infrared (IR) sensors offer a passive, day‑and‑night solution, but IR images of small UAVs often suffer from low contrast and high background clutter. Acoustic sensing is another promising modality, as UAVs emit distinctive propeller noise; however, environmental background sounds can mask the signal. A natural solution is to fuse multiple modalities to compensate for individual weaknesses. In this work, we combine IR video and audio signals to achieve robust UAV detection. The main contributions of our work are:
- A dynamic saliency difference enhancement module (DSDEM) that adaptively fuses spatial, motion, and temporal cues to highlight small moving targets in infrared sequences.
- A spatiotemporal trajectory encoding and association module (STREAM) that uses an LSTM network to model short‑term motion patterns and computes a trajectory‑measurement matching score with a dynamic fusion factor to maintain track continuity under occlusion.
- A decision-level fusion framework that integrates the independent detection results from IR and audio modalities, leveraging the complementary strengths of each sensor.
- Comprehensive experiments on a multi‑sensor dataset, demonstrating superior performance over existing methods.
The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 details the proposed method including the IR detection pipeline, audio recognition, and fusion strategy. Section 4 presents experimental setups, ablation studies, and comparisons. Section 5 concludes the paper.
2. Related Work
Drone technology has inspired numerous detection methods based on different sensors. For infrared small‑target detection, approaches fall into single‑frame and multi‑frame categories. Single‑frame methods such as improved Faster R-CNN, RetinaNet, YOLOv5, and RT-DETR have been adapted for small objects, but they often struggle with low contrast and background clutter. Multi‑frame methods leverage temporal information; for example, spatial‑temporal networks like SSTNet and STDMANet use 3D convolutions or LSTM to capture motion features. However, these models often require heavy computation and may fail on extremely dim targets.
Acoustic‑based UAV detection typically extracts mel‑frequency cepstral coefficients (MFCC) or log‑mel spectrograms and feeds them into classifiers such as CNNs. While effective in quiet environments, performance degrades in noisy real‑world conditions. Multimodal fusion has recently gained attention. Svanström et al. proposed a dataset with visible, thermal, and acoustic sensors. Fusion at the feature or decision level can improve robustness, but existing fusion frameworks often treat modalities independently without adaptive weighting. Our work addresses these limitations by designing a track‑before‑detect IR pipeline and a complementary audio classifier, combined with a logical decision‑level fusion that is both simple and effective.
3. Proposed Method
3.1 Infrared Video Target Detection
The infrared detection pipeline follows a track‑before‑detect paradigm, which is inherently suitable for small moving targets. As depicted in Figure 2 (conceptual), the input infrared sequence undergoes preprocessing, dynamic saliency enhancement, candidate detection, feature extraction, and spatiotemporal trajectory association. We describe the key modules below.
3.1.1 Dynamic Saliency Difference Enhancement Module (DSDEM)
The DSDEM is designed to boost contrast between weak targets and complex backgrounds. It operates in three parallel pathways: gradient–grayscale fusion, multi‑scale local contrast, and motion feature extraction.
Gradient–Grayscale Fusion: For each input frame \(I_t\), we compute the gradient magnitude \(G(x,y) = \sqrt{G_x^2 + G_y^2}\) using Sobel operators. The fusion weight adapts to local texture: in high‑texture regions (e.g., tree edges), we assign higher weight to gradient; in smooth regions (e.g., sky), we rely more on raw intensity. The fused feature map is given by:
$$F_{\text{fusion}}(x,y) = \alpha(x,y) \cdot G(x,y) + (1-\alpha(x,y)) \cdot I_t(x,y)$$
where \(\alpha(x,y) = \sigma(\text{Var}_{\text{local}}(x,y))\) with \(\sigma\) being a sigmoid scaling function of local variance.
Multi‑Scale Local Contrast: We adopt a dual‑window strategy: an inner window (size \(w_i\)) covers the potential target, and an outer window (size \(w_o\)) covers the local background. The contrast measure at scale \(s\) is:
$$C_s(x,y) = \frac{\mu_{\text{inner}}(x,y) – \mu_{\text{outer}}(x,y)}{\sigma_{\text{outer}}(x,y) + \epsilon}$$
where \(\mu\) and \(\sigma\) denote mean and standard deviation, and \(\epsilon=10^{-6}\). To handle scale variation, we compute \(C_s\) for \(w_i \in \{3,5,7\}\) and \(w_o = 3w_i\). The resulting maps are up‑sampled via bilinear interpolation and fused by weighted averaging. Crucially, inspired by local entropy, we adaptively adjust \(w_o\): if the local entropy \(E_{\text{local}} = -\sum p_i \log_2 p_i\) exceeds a threshold \(\tau_{\text{entropy}}=1.8\), we shrink \(w_o\) from \(9\times9\) to \(5\times5\) to suppress complex clutter.
Motion Feature Extraction: To leverage temporal information, we compute the inter‑frame difference: \(D_t(x,y) = |I_t(x,y) – I_{t-1}(x,y)|\). For moving targets, \(D_t\) is consistently high. We combine the difference with the contrast map using adaptive weights learned from the scene dynamics. The final saliency map after temporal consistency check (averaging over three consecutive frames) is:
$$S_{\text{final}}(x,y) = \begin{cases} \frac{1}{3}\sum_{i=0}^{2} S_{t-i}(x,y), & \text{if } \frac{1}{3}\sum_{i=0}^{2} S_{t-i}(x,y) > T_{\text{motion}} \\ 0, & \text{otherwise} \end{cases}$$
where \(T_{\text{motion}} = \mu_S + 2\sigma_S\) with \(\mu_S,\sigma_S\) being the mean and standard deviation of the temporal average map. This mechanism filters out transient noise (e.g., birds, clouds) and retains only persistent moving targets.
3.1.2 Spatiotemporal Trajectory Encoding and Association Module (STREAM)
After obtaining candidate detections from the saliency map, we need to associate them across frames to form consistent tracks. STREAM consists of three parts: short‑term trajectory encoding, trajectory‑measurement matching, and dynamic fusion factor.
Short‑Term Trajectory Encoding: We store the last 5 positions of each tracked target as a sequence. This sequence is fed into a two‑layer LSTM (128 units each) to extract a trajectory feature vector \(h_t\). The LSTM is trained with Adam optimizer (learning rate 0.001, \(\beta_1=0.9,\beta_2=0.999\), dropout 0.3). The hidden state captures motion patterns such as velocity and acceleration, which is robust against temporary occlusions.
Trajectory‑Measurement Matching: Each new measurement \(z_j\) is projected into the same feature space via a small MLP: \(\phi(z_j) = \text{ReLU}(W_z z_j + b_z)\). The similarity between a trajectory and a measurement is computed by cosine similarity:
$$\text{TM}_{jt} = \frac{h_t \cdot \phi(z_j)}{\|h_t\| \|\phi(z_j)\|}$$
In addition, we compute appearance similarity using a CNN on the target patch (intensity distribution and shape). The final matching score combines motion and appearance cues.
Dynamic Fusion Factor: We define an association probability that balances spatial distance and trajectory similarity:
$$P(j,t) = \lambda \cdot \frac{\text{ED}(j,t)}{\text{MD}(j,t)} + (1-\lambda) \cdot \text{TM}_{jt}$$
where \(\text{ED}\) is Euclidean distance, \(\text{MD}\) is Mahalanobis distance, and \(\lambda\) is a dynamic weight. Initially set to 0.5, \(\lambda\) decreases by 0.1 (down to 0.3) when the target’s speed change rate exceeds a threshold (indicating high maneuver), thus increasing the importance of trajectory matching. This design helps maintain tracks during sharp turns and rapid accelerations, common in drone technology.
3.2 Audio Signal Recognition
Audio signals provide an independent source of evidence. We first apply a Hamming window and compute the short‑time Fourier transform (STFT) with 1440‑point window and 960‑point overlap, yielding a spectrogram. After applying 32‑band mel‑scale filters, we obtain a mel‑spectrogram. To reduce dimensionality while preserving discriminative features, we convert to log‑scale and resize to \(128 \times 128\) pixels. Then we perform Z‑score normalization using dataset‑wide mean and standard deviation.
The normalized log‑mel spectrogram is input to a CNN classifier with the following architecture:
- Input: 128×128×1
- Conv block 1: 32 filters, 3×3, ReLU, max‑pool 2×2
- Conv block 2: 64 filters, 3×3, ReLU, max‑pool 2×2
- Conv block 3: 128 filters, 3×3, ReLU, max‑pool 2×2
- Flatten, then fully connected (512 units, ReLU)
- Output layer: Softmax over classes (drone, helicopter, background)
3.3 Decision‑Level Fusion
For each frame (or audio segment), the IR classifier outputs a probability vector \(\mathbf{p}_{\text{IR}}\) over the three classes (drone, bird, helicopter) and the audio CNN outputs \(\mathbf{p}_{\text{audio}}\). The final decision is obtained by weighted averaging:
$$\mathbf{p}_{\text{fusion}} = \gamma \cdot \mathbf{p}_{\text{IR}} + (1-\gamma) \cdot \mathbf{p}_{\text{audio}}$$
where \(\gamma\) is tuned on the validation set (we found \(\gamma = 0.6\) yields best performance). The class with the highest probability is declared. This simple yet effective fusion leverages the complementary nature of the modalities: IR excels at capturing motion and shape, while audio is robust to visual occlusion.
4. Experiments
4.1 Dataset and Evaluation
We used the public multi‑sensor drone detection dataset introduced by Svanström et al. (2021). It contains 365 infrared video clips (320×256, 30 fps, 10s each) and 90 audio clips (44.1 kHz, 10s each). The dataset includes three drone types (Inspire2, Mavic3, Phantom4) and three background classes. We split the data into training (60%) and testing (40%) sets while preserving class balance. Preprocessing for IR includes histogram equalization and normalization; audio is resampled to 48 kHz and standardized.
Evaluation metrics: Precision, Recall, F1‑score, and Average Precision (AP). All experiments are repeated three times and we report mean ± std.
4.2 Ablation Study
We conducted ablation experiments to assess the contribution of each proposed module in the IR pipeline. The baseline (Method (1)) uses a conventional track‑before‑detect without DSDEM or STREAM. Method (2) adds DSDEM, Method (3) adds STREAM, and Method (4) includes both. Results are summarized in Table 1.
| Method | Precision (%) | Recall (%) | F1 | AP (%) |
|---|---|---|---|---|
| (1) Baseline | 75.2±0.13 | 68.1±0.15 | 0.71±0.0014 | 60.1±0.28 |
| (2) +DSDEM | 79.2±0.18 | 75.2±0.19 | 0.77±0.0016 | 62.4±0.26 |
| (3) +STREAM | 78.7±0.21 | 77.2±0.15 | 0.78±0.0018 | 63.1±0.29 |
| (4) +DSDEM+STREAM | 86.7±0.17 | 82.1±0.22 | 0.84±0.0019 | 70.5±0.34 |
From Table 1, each module brings clear improvements. DSDEM significantly increases recall (from 68.1% to 75.2%), indicating better target capture. STREAM improves both recall and F1, proving its role in maintaining track continuity. The full model (4) achieves the best results across all metrics, with precision 86.7%, recall 82.1%, F1 0.84, and AP 70.5%. The synergy between the two modules is evident: DSDEM provides higher‑quality detections, while STREAM robustly associates them over time.
4.3 Comparison with State‑of‑the‑Art
We compare our method with several representative object detectors: Faster R-CNN, RetinaNet, YOLOv5, CenterNet, and RT-DETR. We also test two variants of fusion: YOLOv5+feature fusion, YOLOv5+weighted fusion, and RT-DETR+weighted fusion. Table 2 reports the results.
| Method | Precision (%) | Recall (%) | F1 | AP (%) |
|---|---|---|---|---|
| Faster R-CNN | 73.2±0.21 | 60.3±0.17 | 0.66±0.0018 | 58.7±0.23 |
| RetinaNet | 70.1±0.15 | 62.4±0.26 | 0.66±0.0021 | 60.4±0.21 |
| YOLOv5 | 80.3±0.17 | 75.1±0.20 | 0.78±0.0018 | 67.2±0.20 |
| CenterNet | 56.4±0.18 | 70.2±0.13 | 0.63±0.0017 | 52.4±0.19 |
| RT-DETR | 82.4±0.16 | 78.7±0.19 | 0.81±0.0017 | 68.1±0.18 |
| Ours (IR only) | 86.7±0.17 | 82.1±0.22 | 0.84±0.0019 | 70.5±0.34 |
| YOLOv5+feature fusion | 82.4±0.23 | 76.8±0.21 | 0.80±0.0029 | 68.9±0.22 |
| YOLOv5+weighted fusion | 84.7±0.19 | 79.5±0.17 | 0.82±0.0023 | 70.3±0.21 |
| RT-DETR+weighted fusion | 86.1±0.18 | 82.4±0.15 | 0.84±0.0019 | 71.2±0.18 |
| Ours (Fusion) | 89.5±0.18 | 85.7±0.19 | 0.88±0.0018 | 75.4±0.23 |
Our IR‑only method already outperforms all single‑modality baselines. The fusion model achieves the highest scores: precision 89.5%, recall 85.7%, F1 0.88, and AP 75.4%. The improvement over the best single‑modal competitor (RT-DETR+weighted fusion) is about 3.4% in precision, 3.3% in recall, and 4.2% in AP. This demonstrates that our decision‑level fusion effectively exploits the complementarity of infrared and acoustic information, advancing the state of the art in drone technology.
5. Conclusion
We proposed a multimodal detection framework for small UAVs that integrates infrared video and audio signals. Our IR pipeline introduces a dynamic saliency difference enhancement module and a spatiotemporal trajectory encoding and association module, which together improve detection of low‑contrast moving targets and maintain track continuity. The audio path uses a CNN on log‑mel spectrograms, and decision‑level fusion combines both sources. Extensive experiments on a public dataset show that our method achieves superior precision (89.5%), recall (85.7%), and average precision (75.4%), outperforming existing approaches. The results confirm the effectiveness of our design in advancing drone technology for security and surveillance applications.