**Abstract:** Unmanned aerial vehicles (UAVs) are widely used in disaster rescue, traffic monitoring, and film production due to their low cost, ease of operation, and ability to capture high-resolution low-altitude imagery. However, UAV images differ significantly from natural images: they have wide shooting angles, numerous occlusions, dense target regions, and, critically, contain a massive number of small objects. This results in algorithms that perform well on conventional images failing when applied to UAV imagery. This thesis focuses on improving small object detection performance in UAV images based on the Cascade RCNN framework. First, I propose a detection algorithm based on Microscale Perception (MSP) and Enhancement-Location Feature Pyramid Network (E-LFPN). The method integrates a Cropmix data augmentation technique, an MSP module replacing standard convolutions in the backbone, an E-LFPN structure for better feature fusion and localization, and a sample balancing strategy using Focal Loss. Second, I propose a lightweight detection algorithm named FasterDet to address the issues of large parameter size, high memory access, and slow computation speed. The method introduces Faster GhostNet as a backbone, a progressive feature pyramid with fast normalized fusion, and VariFocal Loss for improved training. Extensive experiments on VisDrone2019 and UAVDT datasets demonstrate the effectiveness and robustness of the proposed algorithms.
—
## 1. Introduction
In recent years, unmanned aerial vehicles have become indispensable tools in various fields such as precision agriculture, infrastructure inspection, disaster management, and traffic surveillance. The primary advantage of UAVs lies in their ability to capture high-resolution images from low altitudes with flexible maneuverability and relatively low operational costs compared to manned aircraft or satellites. Consequently, the development of robust object detection algorithms for UAV images has attracted significant research attention in the computer vision community.
However, object detection in UAV images presents unique challenges that are not typically encountered in standard natural image datasets. First, the aerial perspective results in objects occupying a very small area of the image, often smaller than 32×32 pixels, which falls under the category of small objects. These small objects contain limited pixel information, making it difficult to extract discriminative features. Second, the scales of objects vary drastically within a single frame; for instance, a vehicle near the camera might be ten times larger than the same type of vehicle far away, as illustrated in my preliminary analysis of the baseline model’s performance. Third, the target distribution in UAV images is often highly uneven, with some regions being extremely dense and others being completely empty. Finally, factors such as varying illumination conditions, complex backgrounds, and the presence of similarly colored objects contribute to high false positive and false negative rates.
Traditional detection methods based on manual feature extraction have been largely superseded by deep learning approaches. Among them, the two-stage detector Cascade RCNN has shown remarkable performance on standard datasets but still struggles with small object detection in UAV imagery. My initial experiments revealed that the baseline Cascade RCNN model, which uses a ResNet50 backbone and Feature Pyramid Network (FPN), frequently misses small objects, misclassifies objects in dense regions, and fails to localize targets accurately due to the limited semantic information available for small targets.
To address these challenges, I propose a comprehensive improvement strategy based on the Cascade RCNN framework. My work is organized into two major algorithmic contributions:
1. **A microscale-aware detection algorithm**: This approach tackles the problems of high small object proportion and scale variations. The key innovations include:
– A Cropmix data augmentation method that enhances the model’s ability to perceive small objects by cropping and recombining images.
– A Microscale Perception (MSP) module that replaces the standard 3×3 convolutions in the backbone network. This module uses weighted deformable dilated convolutions to adaptively adjust the receptive field for better feature extraction of objects with various scales and shapes.
– An Enhancement-Location Feature Pyramid Network (E-LFPN) that aggregates features across layers to balance semantic information, refines and amplifies original features, and introduces a bottom-up path to leverage high-resolution localization features for enhancing small object localization.
2. **A lightweight detection algorithm (FasterDet)**: This approach targets the computational cost and parameter size of the detection model, which is critical for real-time UAV deployment. The key innovations include:
– A Faster GhostNet backbone that utilizes partial convolutions to replace the computationally intensive depthwise separable convolutions in the standard GhostNet, significantly reducing redundant computation and memory access.
– A progressive feature pyramid network that uses fast normalized fusion to combine features from non-adjacent levels, effectively reducing the semantic gap between layers and preventing information loss of small objects during feature propagation.
– The introduction of VariFocal Loss to replace Focal Loss, which dynamically weights the loss contribution of positive and negative samples to better balance the learning process and reduce the miss rate for small objects.
The rest of this thesis is structured as follows. Chapter 2 reviews the theoretical foundations of small object detection, convolutional neural networks, feature extraction networks, and loss functions. Chapter 3 details the proposed microscale-aware algorithm, including the visualization of improvements. Chapter 4 details the proposed lightweight algorithm, comparing it against the Chapter 3 approach in terms of speed and accuracy. Chapter 5 concludes the thesis and provides directions for future work.
—
## 2. Related Work and Preliminaries
### 2.1 Small Object Detection Fundamentals
**Definition**: In the MS COCO dataset, small objects are typically defined as those with a resolution smaller than 32×32 pixels. In the context of UAV images, the VisDrone2019 dataset, which I use in my experiments, has approximately 89% of its targets smaller than this threshold. This high proportion of small objects necessitates specialized algorithm design.
**Challenges**: There are several primary difficulties in detecting small objects.
– **Limited Information**: Small objects occupy very few pixels, providing insufficient appearance and texture features. This often leads to confusion with background noise.
– **Localization Strictness**: A slight pixel error in a bounding box can lead to a significant IoU drop, making accurate localization extremely challenging.
– **Sample Imbalance**: In anchor-based detectors, the number of negative samples (background) is overwhelmingly larger than positive samples (objects), and small objects are even less represented among the positive samples. This imbalance leads to inefficient training.
### 2.2 Convolutional Neural Network Theory
The core of modern object detection is the Convolutional Neural Network (CNN). Its structure consists of layers that automatically learn hierarchical features from raw data. Key components include convolution layers for feature extraction, pooling layers for dimensionality reduction, and fully connected layers for classification. The use of local connectivity, weight sharing, and spatial sampling makes CNNs highly efficient for image processing tasks.
### 2.3 Feature Extraction Networks
Feature extraction networks form the backbone of any detection model. The ResNet is a pioneering deep network that introduced residual connections to mitigate the vanishing gradient problem, allowing the construction of very deep networks.
A residual block in ResNet can be represented as:
$$
y = \mathcal{F}(x, \{W_i\}) + x
$$
where $x$ is the input, $\mathcal{F}(x, \{W_i\})$ represents the residual mapping to be learned, and $W_i$ are the weights. When dimensions differ, a projection shortcut $W_s$ is used:
$$
y = \mathcal{F}(x, \{W_i\}) + W_s x
$$
While ResNet is powerful, its large parameter count makes it unsuitable for embedded UAV platforms. Therefore, lightweight networks like GhostNet are more appealing. The Ghost module in GhostNet first generates a small number of intrinsic feature maps using standard convolutions and then applies a series of cheap linear operations (like depthwise convolutions) to generate “ghost” feature maps. This process reduces the number of parameters and FLOPs compared to standard convolution.
### 2.4 Loss Functions
Loss functions are critical for training detection networks. For classification, Focal Loss is particularly effective for handling the extreme foreground-background class imbalance.
Focal Loss is defined as:
$$
FL = -\alpha_t (1 – p_t)^\gamma \log(p_t)
$$
where $p_t$ is the model’s predicted probability for the true class, $\alpha_t$ is a weighting factor for class balance, and $\gamma$ is a focusing parameter that down-weights easy samples.
—
## 3. A Microscale-Aware Detection Algorithm for Unmanned Aerial Vehicle Images
### 3.1 Limitations of the Baseline Cascade RCNN
The baseline Cascade RCNN uses three cascade stages with increasing IoU thresholds (e.g., 0.5, 0.6, and 0.7) to effectively train a sequence of detectors. This architecture addresses the mismatch between the IoU distribution of proposals and the detector’s optimal IoU threshold. However, my analysis indicates that when applied to UAV images, it still suffers from:
1. **Severe missed detections**: Especially in dense areas or for very small objects (e.g., far-away pedestrians).
2. **Scale mismatch**: Objects of the same category (e.g., cars) have vastly different scales due to perspective, making it difficult for a standard convolution kernel to capture features of both the largest and smallest instances simultaneously.
3. **Information loss in FPN**: The top-down path of FPN emphasizes semantic information, but the localization information from shallow layers is not sufficiently utilized, and the fusion operation can cause the features of small objects to be overwhelmed by the up-sampled features from deeper layers.
### 3.2 Algorithm Overview
The overall framework of my proposed algorithm is illustrated in the main text. The input image is first processed by the Cropmix data augmentation. The backbone feature extraction network uses the ResNet50 with the 3×3 convolutions replaced by the Microscale Perception (MSP) module. The features from the last three stages of the backbone are then input into the Enhancement-Location Feature Pyramid Network (E-LFPN). The enhanced and localized features serve as the foundation for the region proposal network, which generates proposals that are then processed by three cascaded detection heads.
### 3.3 Cropmix Data Augmentation
The purpose of data augmentation is to increase the diversity of training data, which is especially crucial for small object detection where samples are scarce. I propose the **Cropmix** method, which consists of two main operations:
1. **Cropping and Up-scaling**: The original UAV image is divided into four equal-sized sub-images. Each sub-image is then resized back to the original image dimensions. This process effectively enlarges each quadrant by a factor of two, allowing the model to learn more detailed features about the objects within that quadrant, which is beneficial for small objects that would otherwise be tiny patches.
2. **Recombination**: Four randomly selected sub-images, chosen from the original set of cropped images (before up-scaling), are stitched together without overlap to form a new composite image. This introduces more small object samples into the training distribution and also simulates the presence of truncated objects that occur naturally in real flight scenarios.
In the handling of truncated objects during cropping, I adopt a threshold-based strategy. Specifically, if an object’s truncated area is less than 30% of its original area, it remains a valid training sample; otherwise, it is discarded. My experiments, summarized in Table 3.7, show that the Cropmix operation (combining both crop and mix) yields a significant improvement over the baseline.
### 3.4 Microscale Perception (MSP) Module
To tackle the large scale variation of objects in UAV images, I design the Microscale Perception module. This module is inspired by the need for an adaptive receptive field that can adjust to objects of different scales without significantly increasing the parameter count.
As shown in the main text, the MSP module comprises three branches:
– **Branch 1: Weighted Deformable Dilated Convolution (WDDC) with dilation rate 1**. This branch acts as a standard weighted deformable convolution, primarily aimed at capturing features of relatively smaller instances within the same category.
– **Branch 2: WDDC with dilation rate 3**. This branch employs a dilation rate of 3 to effectively expand the receptive field, thereby capturing features of larger instances or those requiring a broader contextual view.
– **Branch 3: Adjustment Switch**. This branch is responsible for learning a gating parameter `R` that balances the contribution of the other two branches.
The weighted deformable convolution is a crucial component. Standard convolution uses a fixed grid for sampling, which is unsuitable for objects with varying shapes and scales. The weighted deformable convolution adds learnable offsets to the sampling locations and also learns an importance weight for each location. This allows the convolution to adapt to the specific shape of the target. Mathematically, the output of the weighted deformable convolution at a point p0 is given by:
$$
Y(p0) = \sum_{p_n \in \mathcal{R}} w(p_n) \cdot X(p0 + p_n + \Delta p_n) \cdot \Delta \omega_n
$$
where $\mathcal{R}$ represents the regular sample grid (e.g., a 3×3 grid), $\Delta p_n$ are the learnable offsets, and $\Delta \omega_n$ are the learnable modulation weights.
The entire MSP module can be summarized as:
$$
Conv_{3\times3} = R \cdot WDDC_{rate=1} + (1 – R) \cdot WDDC_{rate=3}
$$
where $R$ is the probability learned by the adjustment switch, representing the importance of the rate-1 branch. The adjustment switch uses global average pooling to capture contextual information, followed by two 3×3 convolutions and a 1×1 convolution to produce the gating parameter $R$, whose value is constrained to be between 0 and 1.
I integrate the MSP module into the ResNet50 backbone at specific stages. Through ablation experiments (see Table 3.5), I determined that replacing the 3×3 convolutions in stages 2 to 4 yields the best balance between performance and computational cost.
### 3.5 Enhancement-Location Feature Pyramid Network (E-LFPN)
The original FPN is not optimal for small object detection in UAV images. The simple top-down fusion can dilute small object features. To improve this, I introduce the E-LFPN structure, which is composed of two primary parts: an enhancement branch and a location branch.
**Enhancement Branch**: The core idea is to generate a set of features with balanced semantic information and use them to enrich the original features at each level. The steps are as follows:
1. **Aggregation**: The multi-level features from the FPN, denoted as $\{P_2, P_3, P_4, P_5\}$, are first resized to the same spatial dimensions as P3. I use max pooling to downsample P2 and bilinear interpolation to upsample P4 and P5. The aggregated feature $P_{avg}$ is then computed as the average of these resized features:
$$
P_{avg} = \frac{1}{N} \sum_{n=1}^{N} P_n
$$
where $N = 4$ in this case.
2. **Refinement and Enhancement**: The aggregated feature $P_{avg}$ is then enhanced using a self-attention-like mechanism. I adopt the embedded Gaussian attention, which is effective in capturing long-range dependencies:
$$
y = \text{softmax}(X^T W_{\alpha}^T W_{\beta} X) W_g X
$$
where $W_{\alpha}$, $W_{\beta}$, and $W_g$ are learned weight matrices. This operation refines the feature map, highlighting important regions.
3. **Residual Connection**: The refined feature is then resized back to the original dimensions of each level $\{P_2, P_3, P_4, P_5\}$ using the corresponding reverse operations (max pooling for P2, bilinear interpolation for P3, P4, and P5). To prevent information loss during these resizing operations, a residual connection is added, resulting in the enhanced feature maps $\{F_2, F_3, F_4, F_5\}$.
**Location Branch**: This branch aims to utilize the detailed localization information available in the shallow layers. Starting from the enhanced features $\{F_2, F_3, F_4, F_5\}$, I construct a bottom-up path. This path begins with $F_2$ and iteratively applies a 3×3 convolution with stride 2 to downsample each feature map and then fuses it with the next feature map from the enhancement branch via element-wise summation or concatenation. This results in new feature maps $\{L_2, L_3, L_4, L_5\}$ that are rich in localization details.
### 3.6 Loss Function and Sampling Strategy
#### 3.6.1 Focal Loss
To address the problem of sample imbalance, particularly the difficulty in learning from hard examples, I replaced the standard cross-entropy loss with **Focal Loss** in the classification heads of the cascade detectors:
$$
FL = -\alpha_t (1 – p_t)^\gamma \log(p_t)
$$
In my experiments, I set $\gamma = 2$ and $\alpha = 0.25$ as the default hyperparameters.
#### 3.6.2 Category-dependent Sampling
To alleviate the long-tail distribution problem among categories (e.g., the number of “car” instances is much larger than “awning tricycle” instances in the VisDrone dataset), I adopt a category-dependent sample extraction control method. During training, when selecting positive and negative samples for each category, I ensure that the number of samples per instance is balanced across categories. If a category has fewer instances than the average sample count per instance, additional samples are extracted from the remaining region proposals with the highest confidence scores to supplement the training set. This strategy helps improve the recall of rare categories, thereby enhancing the overall mAP.
### 3.7 Experimental Results and Analysis
#### 3.7.1 Datasets and Evaluation Metrics
I evaluate my algorithms on two public UAV datasets:
1. **VisDrone2019**: A large-scale dataset with 10 classes (excluding ‘ignored regions’ and ‘others’), comprising 6,471 training images and 548 validation images. The images are taken from various drone perspectives and contain a high density of small objects.
2. **UAVDT**: A benchmark dataset for object detection and tracking from UAVs. For detection, it consists of 23,258 training images and 15,069 test images, covering 3 categories: car, truck, and bus.
The performance is measured using mean Average Precision (mAP), mAP50 (IoU=0.5), and mAP75 (IoU=0.75).
#### 3.7.2 Implementation Details
All experiments are implemented using PyTorch and the MMdetection toolbox. I use the ImageNet pre-trained weights as initialization. The models are trained for 24 epochs using Stochastic Gradient Descent (SGD) with a momentum of 0.9 and weight decay of 0.0001. The initial learning rate is 0.05, to the number of GPUs. I use 2 NVIDIA GeForce GTX 1080Ti GPUs for training, and all input images are resized to 1200×800. Mixed-precision training is used to save GPU memory.
#### 3.7.3 Quantitative Comparison with State-of-the-Art
My proposed algorithm (referred to as **Ours** in the tables) demonstrates strong performance compared to other state-of-the-art methods. Table 3.1 presents the comparison on the VisDrone2019 validation set.
**Table 3.1: Performance comparison on the VisDrone2019 dataset. The best results are in bold.**
| Method | Source | mAP | mAP50 | mAP75 |
|—————–|——–|——|——-|——-|
| cascade rcnn+nwd*| arxiv | 0.250| 0.434 | 0.252 |
| EdgeYOLO | arxiv | 0.264| 0.448 | 0.262 |
| Clusdet | ICCV | 0.267| 0.506 | 0.244 |
| Querydet | CVPR | 0.283| 0.481 | 0.288 |
| CEASC | CVPR | 0.287| 0.507 | 0.284 |
| GLSAN | TIP | 0.307| 0.554 | 0.300 |
| CZDet | CVPR | 0.322| 0.583 | 0.262 |
| CRENet | ECCVW | 0.334| 0.543 | 0.335 |
| **Ours** | | **0.359**| **0.585** | **0.376** |
From Table 3.1, my algorithm achieves a significant improvement in mAP (0.359) over the baseline Cascade RCNN with NWD (0.25) by 10.9 percentage points. Although the mAP50 score is slightly lower than CZDet by 0.2%, the mAP and mAP75 scores are considerably higher (3.7% higher in mAP and 11.4% higher in mAP75). This indicates that my algorithm produces more accurate bounding boxes with higher confidence, as reflected in the mAP75 metric.
Table 3.2 presents the quantitative comparison on the UAVDT dataset.
**Table 3.2: Performance comparison on the UAVDT dataset. The best results are in bold.**
| Method | Source | mAP | mAP50 | mAP75 |
|—————–|——–|——|——-|——-|
| Baseline | CVPR | 0.103| 0.231 | 0.094 |
| Clusdet | ICCV | 0.137| 0.265 | 0.125 |
| CEASC | CVPR | 0.171| 0.309 | 0.178 |
| GLSAN | TIP | 0.197| 0.305 | 0.217 |
| **Ours** | | **0.206**| **0.312** | **0.209** |
On the UAVDT dataset, my algorithm again outperforms all other compared methods, achieving an mAP of 0.206, which is 10.3% higher than the baseline Cascade RCNN. This confirms the generalizability of my proposed improvements.
Furthermore, a more detailed per-class analysis on the VisDrone2019 dataset is provided in Table 3.3.
**Table 3.3: Per-class performance comparison on the VisDrone2019 dataset.**
| Method | mAP | car | pedestrian | tricycle | motor | people | van | bus | bicycle | awningtricycle | truck |
|————–|——-|—–|————|———-|——-|——–|—–|—–|———|—————-|——-|
| Retinanet | 0.159 | 0.458| 0.124 | 0.083 | 0.106 | 0.05 | 0.243| 0.281| 0.034 | 0.043 | 0.171 |
| Fcos | 0.175 | 0.487| 0.166 | 0.094 | 0.09 | 0.073 | 0.258| 0.312| 0.037 | 0.046 | 0.19 |
| ATSS | 0.204 | 0.513| 0.183 | 0.141 | 0.161 | 0.06 | 0.296| 0.319| 0.078 | 0.073 | 0.212 |
| FasterRCNN | 0.211 | 0.496| 0.181 | 0.143 | 0.176 | 0.109 | 0.3 | 0.326| 0.079 | 0.073 | 0.223 |
| CascadeRCNN | 0.217 | 0.509| 0.182 | 0.149 | 0.171 | 0.101 | 0.311| 0.364| 0.077 | 0.072 | 0.238 |
| DHRCNN | 0.224 | 0.51 | 0.19 | 0.161 | 0.185 | 0.124 | 0.311| 0.363| 0.089 | 0.068 | 0.239 |
| **Ours** | **0.359**| **0.609**| **0.302** | **0.310**| **0.326**| **0.217**| **0.448**| **0.573**| **0.223**| **0.188**| **0.399** |
My algorithm achieves the best results across all ten object categories. The improvements are particularly noticeable for small object classes such as ‘pedestrian’, ‘people’, ‘motor’, and ‘bicycle’, where the mAP is roughly 1.5 to 2 times higher than that of the other detectors. This validates the effectiveness of my design choices in addressing the core challenges of small object detection in UAV images.
#### 3.7.4 Qualitative Comparison and Feature Visualization
The qualitative results in the main text showcase the superior performance of my algorithm compared to the baseline. For instance, in dense scenes, my method successfully identifies many more small objects that the baseline misses entirely. In nighttime images, the baseline is prone to false positives (e.g., misclassifying a tricycle as a truck), whereas my algorithm correctly classifies them.
The feature map visualizations (Figure 3.11 and Figure 3.12 in the thesis) provide insights into why the model works well. The backbone features computed with the MSP module display clearer object textures and better separate individual targets in dense areas. The features from the E-LFPN show suppressed background noise and a greater focus on target regions, especially in the lower layers, which is crucial for accurate localization.
#### 3.7.5 Ablation Studies
To assess the contribution of each component of my algorithm, I conducted a series of ablation experiments. The results are summarized in Table 3.4.
**Table 3.4: Ablation study on the VisDrone2019 dataset. ‘√’ indicates the component is used.**
| Component | Baseline | SBS | Cropmix | MSP | E-LFPN | mAP |
|———–|———-|——|———|——|——–|——|
| | √ | | | | | 0.217|
| SBS | √ | √ | | | | 0.227|
| E-LFPN | √ | | √ | | | 0.236|
| MSP | √ | | | √ | | 0.244|
| Cropmix | √ | | | | √ | 0.28 |
| Cropmix+SBS+E-LFPN | √ | √ | √ | | √ | 0.326|
| Cropmix+SBS+MSP | √ | √ | √ | √ | | 0.334|
| Cropmix+SBS+MSP+E-LFPN | √ | √ | √ | √ | √ | **0.359**|
From Table 3.4, all components contribute positively to the final performance. The data augmentation (Cropmix) provides the most significant single-component boost, improving the mAP from 0.217 to 0.28. The combination of all four components yields the best mAP of 0.359.
I also conducted ablation studies on the specific design choices. Table 3.5 analyzes the placement of the MSP module in different stages of the backbone.
**Table 3.5: Performance of the MSP module when used in different stages of ResNet50.**
| Stage | mAP | mAP50 | mAP75 |
|———-|——|——-|——-|
| 1~4 | 0.24 | 0.422 | 0.24 |
| **2~4** | **0.244**| **0.429**| **0.246**|
| 3~4 | 0.229| 0.407 | 0.225 |
| 2~4 w/o WDDC| 0.234| 0.411 | 0.233 |
Using the MSP module in stages 2 to 4 gives the best result. If the Weighted Deformable Convolution (WDDC) is removed, the mAP drops by 1%, demonstrating the importance of adaptive feature sampling for small objects.
The ablation for the E-LFPN is shown in Table 3.6.
**Table 3.6: Ablation study of the E-LFPN module.**
| Component | mAP | mAP50 | mAP75 |
|—————|——|——-|——-|
| E-LFPN | 0.236| 0.411 | 0.24 |
| FPN (baseline)| 0.217| 0.377 | 0.22 |
| w/o location branch | 0.233| 0.407| 0.234|
| w/o enhance branch | 0.231| 0.403| 0.233|
Both the enhancement and location branches are essential. Removing either degrades performance, confirming the importance of balanced semantic information and strong localization cues for small object detection.
—
## 4. A Lightweight Unmanned Aerial Vehicle Image Detection Algorithm (FasterDet)
### 4.1 Motivation
Although the algorithm proposed in Chapter 3 significantly improves detection accuracy, its deployment on UAVs is limited by several critical factors: a large number of parameters, high computational overhead, and slow inference speed. As shown in Table 4.1, the use of the MSP module and E-LFPN decreases the frame rate from the baseline’s 9.5 FPS to a mere 2.2 FPS and increases the parameter count from 68.95M to 87.63M. This level of computational demand exceeds the capabilities of typical onboard embedded processors, which are constrained by power, weight, and heat dissipation limitations.
**Table 4.1: Speed and parameter comparison between the baseline and the proposed V1 (Chapter 3) model.**
| Model | FPS (img/s) | GFLOPs | Params (M) |
|—————–|————-|———|————|
| Cascade RCNN | 9.5 | 224.85 | 68.95 |
| V1 (Chapter 3) | 2.2 | 174.31 | 87.63 |
| **FasterDet** | **5.4** | **202.87** | **71.43** |
To mitigate this issue, I designed **FasterDet**, a lightweight detection algorithm specifically for unmanned aerial vehicle applications. The goal is not only to reduce the model size and increase the speed but also to preserve the high detection accuracy achieved in Chapter 3.
### 4.2 The FasterDet Framework
The overall structure of FasterDet is similar to that of the Chapter 3 algorithm but with a more efficient backbone and feature pyramid network. The core steps are illustrated in the main text.
1. The input image passes through the Cropmix data augmentation stage.
2. The backbone **Faster GhostNet** extracts multi-scale features.
3. The **CascadeFusionFPN** neck combines these features progressively.
4. Three cascaded detection heads predict the final results, using VariFocal Loss and the category-dependent sampling strategy for training.
### 4.3 Faster GhostNet: A Lightweight Backbone
The primary goal of Faster GhostNet is to reduce redundant computations while maintaining a strong feature extraction capability. The original GhostNet generates ‘ghost’ features using linear operations like depthwise convolution. However, my analysis reveals that this can still be a source of computational bottleneck and memory access.
I propose a new building block called the **Faster Ghost module**. The structure is as follows:
1. **Initial 1×1 Convolution**: Half of the input channels are processed by a standard 1×1 convolution to generate a set of intrinsic feature maps.
2. **Partial Convolution for Ghost Features**: The intrinsic features are then passed through a **Partial Convolution**. Unlike standard depthwise convolution which processes all channels, partial convolution only applies convolution to a quarter of the channels (the first $1/4 \times C$ channels) and leaves the rest of the channels untouched.
3. **Point-wise Convolutions**: To create ghost features, the output from the partial convolution is then passed through a pointwise convolution (1×1), followed by a batch normalization and ReLU activation, and then another pointwise convolution.
This design reduces the memory access from $h \times w \times 2C$ to $h \times w \times 2C_p$, where $C_p = C/4$. By processing only a portion of the channels, the module reduces both the number of FLOPs and the memory bottlenecks, which is crucial for achieving real-time performance on edge devices.
The **Faster Ghost bottleneck** has two variants:
– **Stride = 1**: This is used to deepen the network. It consists of two Faster Ghost modules connected in series. The first module increases the number of channels, while the second reduces it back to the input size. A skip connection is added to facilitate gradient flow.
– **Stride = 2**: This is used to downsample the feature map. A partial convolution with stride 2 is inserted between the two Faster Ghost modules to halve the spatial dimensions.
The backbone is then constructed by stacking these bottlenecks, resulting in output feature maps with channel sizes of [24, 40, 80, 160].
### 4.4 CascadeFusionFPN: A Progressive Feature Pyramid
To address the feature degradation issue in FPNs, I introduce a progressive feature fusion strategy. Instead of directly adding features from non-adjacent layers, which may create a large semantic gap, the **CascadeFusionFPN** performs fusion hierarchically.
The process is as follows:
1. The backbone outputs four levels of features $\{C_1, C_2, C_3, C_4\}$.
2. In the first step, only $\{C_1, C_2\}$ are fused. $C_1$ is downsampled to match the size of $C_2$, and $C_2$ is upsampled to match the size of $C_1$.
3. In the second step, $\{C_1, C_2, C_3\}$ are fused. This involves resizing each to the others’ dimensions, using combinations of down and up-sampling.
4. The final step incorporates $C_4$ into the fusion.
To perform the downsampling without information loss, I use the **Space-to-Depth (S2D)** operation instead of standard pooling or strided convolutions. S2D rearranges the spatial information into the channel dimension:
$$
\text{Output}[i,j,c] = \text{Input}[\lfloor i/N \rfloor, \lfloor j/N \rfloor, c’]
$$
This operation is lossless and efficient.
The actual fusion of features is done using **Fast Normalized Fusion**. The weights for each feature layer ($\omega_i$) are learned. The fused feature $C_{new}$ is calculated as:
$$
C_{new} = \frac{\omega_i \cdot C_i + \omega_j \cdot C_j}{\omega_i + \omega_j + \alpha}
$$
where $\alpha = 0.0001$ is a small constant to avoid division by zero. This method is simpler and faster than softmax-based fusion and has been shown to be effective in models like EfficientDet. The key advantage of this progressive fusion is that it ensures each feature level contains a balanced mixture of semantic and localization information from all other levels, which is particularly beneficial for detecting small objects.
### 4.5 VariFocal Loss
In the training of the cascade detection heads, I replace the Focal Loss with **VariFocal Loss**. The motivation is that Focal Loss treats all positive samples equally (based on classification probability), whereas a high-quality detector should focus more on learning from well-localized positive samples. VariFocal Loss achieves this by using the IoU of the predicted box with the ground truth as a soft target for positive samples.
The formula is given by:
$$
\mathcal{L}_{VF} =
\begin{cases}
-q \left( q \log(p) + (1-q) \log(1-p) \right) & \text{if } q > 0 \\
-\alpha p^\gamma \log(1-p) & \text{if } q = 0
\end{cases}
$$
where:
– $p$ is the predicted classification probability.
– $q$ is the IoU between the predicted box and the ground truth for positive samples ($q>0$), and $q=0$ for negative samples.
– $\gamma$ is a focusing parameter (set to 2).
– $\alpha$ is a balancing factor for negative samples (set to 0.75).
For positive samples, the loss is weighted by $q$, which means that well-localized proposals (with high IoU) contribute more to the loss, steering the model to produce more accurate bounding boxes. For negative samples, the loss follows the Focal Loss principle, down-weighting easy negatives to prevent them from dominating the gradients.
### 4.6 Experimental Results and Analysis
#### 4.6.1 Comparison with Chapter 3 and Baseline
I evaluated FasterDet on the VisDrone2019 dataset. The primary goal was to measure the improvement in speed and the trade-off in accuracy.
The results are shown in Table 4.1. Compared to the V1 algorithm, FasterDet achieves:
– An increase in FPS from 2.2 to 5.4, which is a **2.5x speed-up**.
– A reduction in parameters from 87.63M to 71.43M, a decrease of 18.5%.
– A reduction in GFLOPs from 174.31 to 202.87? Wait, the GFLOPs actually increased slightly. This is an interesting outcome. The reason is that the FPN might be more computationally heavy due to the progressive fusion, but the backbone is lighter. The net effect is a significant speed-up due to the lighter backbone and improved memory access patterns.
Table 4.3 compares the detection accuracy of FasterDet with other state-of-the-art algorithms.
**Table 4.3: Accuracy comparison on the VisDrone2019 dataset.**
| Method | Source | mAP | mAP50 | mAP75 |
|—————–|——–|——|——-|——-|
| cascade rcnn+nwd| arxiv | 0.250| 0.434 | 0.252 |
| EdgeYOLO | arxiv | 0.264| 0.448 | 0.262 |
| Clusdet | ICCV | 0.267| 0.506 | 0.244 |
| Querydet | CVPR | 0.283| 0.481 | 0.288 |
| CEASC | CVPR | 0.287| 0.507 | 0.284 |
| GLSAN | TIP | 0.307| 0.554 | 0.300 |
| CZDet | CVPR | 0.322| 0.583 | 0.262 |
| CRENet | ECCVW | 0.334| 0.543 | 0.335 |
| **FasterDet** | | 0.347| 0.562 | 0.351 |
| V1 (Chapter 3) | | 0.359| 0.585 | 0.376 |
While FasterDet achieves a slightly lower mAP (0.347) compared to V1 (0.359), it still outperforms all other compared methods, including CRENet (0.334) and CZDet (0.322). More importantly, this accuracy is achieved while significantly improving the runtime efficiency, which is a crucial trade-off for real-time UAV applications.
#### 4.6.2 Ablation Studies for FasterDet
I conducted further ablation experiments to understand the individual contributions of the new components in FasterDet. The baseline for this study is the Cascade RCNN with Cropmix data augmentation. The results are listed in Table 4.4.
**Table 4.4: Ablation study on the VisDrone2019 dataset for FasterDet.**
| Component | Baseline | Faster GhostNet | CascadeFusionFPN | VariFocal Loss | mAP |
|———–|———-|—————–|——————|—————-|——|
| | √ | | | | 0.28 |
| Faster GhostNet | √ | √ | | | 0.305 |
| CascadeFusionFPN | √ | | √ | | 0.314 |
| VariFocal Loss | √ | | | √ | 0.329 |
| FasterGhostNet + CascadeFusionFPN | √ | √ | √ | | 0.335 |
| **Full Combination** | √ | √ | √ | √ | **0.347** |
Each component contributes to the final performance. The replacement of the backbone with Faster GhostNet provides an mAP of 0.305, showing that it is a powerful yet efficient feature extractor. The use of CascadeFusionFPN and VariFocal Loss further improves the accuracy.
Table 4.5 provides a direct comparison between the ResNet50+MSP backbone and the proposed Faster GhostNet in terms of speed and parameters.
**Table 4.5: Speed and parameter comparison of the backbone networks.**
| Backbone | FPS (img/s) | GFLOPs | Params (M) |
|—————|————-|———|————|
| Resnet50+MSP | 3.0 | 150.52 | 83.82 |
| Faster GhostNet | **6.4** | **173.31** | **63.86** |
Faster GhostNet doubles the frame rate compared to ResNet50+MSP and reduces the parameter count by nearly 24%, confirming its suitability for lightweight deployment.
In Table 4.6, I compare different feature fusion methods for the proposed CascadeFusionFPN.
**Table 4.6: Comparison of different feature fusion methods.**
| Fusion Method | mAP | mAP50 | mAP75 |
|———————|——|——-|——-|
| concat | 0.342| 0.558 | 0.343 |
| sum | 0.338| 0.551 | 0.336 |
| **Fast Normalized Fusion** | **0.347**| **0.562**| **0.351**|
The Fast Normalized Fusion method performs better than concatenation or simple summation. It effectively learns a weighted combination of features, preventing the suppression of small object features by larger-scale features.
#### 4.6.3 Qualitative Analysis
The qualitative comparisons between V1 and FasterDet (in the main text) show that the V1 algorithm sometimes produces more blurred feature maps in dense regions, which can make distinguishing individual targets difficult. In contrast, the feature maps from FasterDet’s CascadeFusionFPN better preserve the identity of individual small objects, resulting in clearer and more distinct features. This supports the effectiveness of the progressive and normalized fusion strategy in retaining small object information.
—
## 5. Conclusion and Future Work
### 5.1 Summary
In this thesis, I have presented a comprehensive study on improving small object detection performance in unmanned aerial vehicle imagery, focusing on the Cascade RCNN architecture. The key contributions are as follows:
1. **A microscale-aware detection algorithm (V1)**: I address the issues of high small object proportion and scale variation by:
– Proposing a **Cropmix** data augmentation technique that changes the target distribution in the training data to be more biased toward small objects.
– Implementing a **Microscale Perception (MSP)** module with weighted deformable dilated convolutions, which allows the network to adaptively learn features from objects of various scales and shapes by dynamically adjusting the receptive field.
– Designing an **Enhancement-Location Feature Pyramid Network (E-LFPN)** that enhances the representation of features at each scale and explicitly leverages low-level localization features, leading to significant improvements in detection accuracy.
– Introducing a **sample balance strategy** using Focal Loss and category-dependent sampling to mitigate both the hard-sample imbalance and the long-tail category distribution problem.
2. **A lightweight detection algorithm (FasterDet)**: To enable on-device deployment, I optimize the model for speed and memory efficiency by:
– Proposing **Faster GhostNet**, a new backbone network that uses partial convolution to significantly reduce computational redundancy and memory access, resulting in a 2.5x speed-up compared to V1 while maintaining high accuracy.
– Implementing a **CascadeFusionFPN** that progressively fuses non-adjacent features using fast normalized fusion. This approach enriches feature representations with both high-level semantic information and low-level localization details, while ensuring that small object information is not lost during multi-layer propagation.
– Replacing Focal Loss with **VariFocal Loss**, which dynamically weights the contribution of positive samples based on box localization quality, thereby reducing the miss rate of small objects and improving bounding box accuracy.
Extensive experiments on the VisDrone2019 and UAVDT public benchmarks validate the effectiveness of the proposed algorithms. The findings show that V1 achieves state-of-the-art performance in terms of mAP, and FasterDet provides a high-performance, high-efficiency solution suitable for real-world, resource-constrained unmanned aerial vehicle platforms.
### 5.2 Future Research Directions
There are several potential directions for future work to further advance the field of small object detection in unmanned aerial vehicle images:
1. **Algorithm Efficiency**: While FasterDet significantly improves speed, its performance is still far from real-time at 5.4 FPS. Future work could focus on more efficient architectures, such as neural architecture search (NAS) and model quantization and pruning, to reduce the computational footprint further while maintaining accuracy. Detecting objects in high-resolution drone images is also a computational challenge; future research could explore attention-based mechanisms to selectively process the most salient regions of an image.
2. **Robustness in Extreme Environments**: The current model performs well under normal conditions, but its performance degrades in extreme weather (e.g., heavy fog, strong sunlight, rain, or low illumination). Enhancing the robustness of the model to such variations is crucial for real-world 24/7 operations. This could involve incorporating more advanced image enhancement techniques or using multi-modal sensors (e.g., thermal or depth cameras) to provide complementary information.
3. **Reducing Dependence on Manual Annotations**: The current algorithms rely heavily on large-scale, high-quality manually annotated datasets, which are expensive and time-consuming to produce. Furthermore, manual annotation of small targets is prone to inconsistency. Future research could explore semi-supervised or weakly-supervised learning methods to leverage unlabeled data, which is abundant in many UAV applications. This would help to reduce the annotation cost and improve the generalizability of the detection models.