We present a comprehensive first-person review on the recent advances of large multimodal models (LMMs) for unmanned aerial vehicle (UAV) visual understanding, with a special focus on China UAV drone research. Over the past decade, China UAV drone technology has evolved from remote‑controlled platforms to autonomous aerial agents capable of perception, reasoning, and decision‑making. Visual understanding is the backbone of this evolution, enabling tasks such as object detection, scene parsing, navigation, and mission planning. The emergence of visual foundation models (VFMs), vision‑language models (VLMs), and multimodal large language models (MLLMs) has fundamentally transformed how China UAV drone systems interpret and interact with complex, open environments. In this work, we adopt a first‑person perspective to systematically review the paradigm shift from task‑specific deep learning models to general‑purpose large models, covering three core capability layers: basic perception enhancement, semantic reasoning, and decision planning. We also summarize key datasets, evaluation benchmarks, and discuss future challenges including real‑time deployment, safety, and embodied intelligence. By highlighting contributions from China UAV drone research groups, we aim to provide a roadmap for the next generation of intelligent aerial systems.
Let us begin by outlining the unique challenges of UAV visual understanding. Unlike ground‑level vision, aerial images suffer from severe scale variations, long‑range observations, small‑object dominance, complex backgrounds, and dynamic environmental changes. Early methods relied on hand‑crafted features and supervised deep learning with fixed category sets, which failed in open‑vocabulary or cross‑domain scenarios. The integration of large models has brought three key breakthroughs: (1) open‑vocabulary perception through vision‑language alignment (e.g., CLIP, SAM), (2) commonsense reasoning via LLMs (e.g., GPT‑4, Qwen), and (3) direct perception‑to‑action mapping via vision‑language‑action (VLA) models (e.g., RT‑2, OpenVLA). China UAV drone research has actively contributed to these advances by developing specialized datasets, benchmarks, and lightweight deployment techniques.
Task Taxonomy and Challenges
Table 1 summarizes the typical UAV visual understanding tasks along with their primary challenges and application domains. The tasks are categorized into four levels: basic perception, event semantics, spatial understanding, and flight decision planning. Each level imposes unique requirements on model capability.
| Level | Task | Main Challenges | Applications |
|---|---|---|---|
| Basic Perception | Object Detection | Small objects, scale variation, occlusion, low resolution | Traffic monitoring, industrial inspection |
| Segmentation & Scene Parsing | Complex background, high annotation cost, cross‑domain gap | Land cover mapping, road extraction | |
| Target Tracking | Scale change, similar distractors, motion blur | Surveillance, traffic flow analysis | |
| Event Semantics | Action Recognition | Long temporal dependency, fine‑grained differences | Air‑ground interaction, security patrol |
| Visual Question Answering | Cross‑modal alignment, complex reasoning | Intelligent inspection query | |
| Accident Detection | Rare events, real‑time requirement | Emergency response, traffic monitoring | |
| Spatial Understanding | 3D Spatial Reasoning | Metric depth ambiguity, structural complexity | 3D mapping, environment perception |
| Geo‑localization | GPS denial, visual‑language semantic gap | Search and rescue, military recon | |
| Flight Decision Planning | Visual Obstacle Avoidance | Dynamic obstacles, low‑latency requirement | Autonomous flight, cruise |
| Vision‑Language Navigation | Instruction ambiguity, sim‑to‑real gap | Package delivery, exploration | |
| Task Planning | Complex mission decomposition, resource scheduling | Inspection, disaster search | |
| Swarm Coordination | Communication asymmetry, path optimization | Drone shows, cooperative reconnaissance |
From this taxonomy, we note that traditional deep learning models tackle each task independently with dedicated architectures and closed‑set label spaces. In contrast, large multimodal models aim to unify these tasks under a common semantic interface. For example, a single MLLM can perform VQA, captioning, and grounding without task‑specific heads, dramatically simplifying the system architecture. China UAV drone researchers have been at the forefront of leveraging such models to overcome the limitations of prior approaches.
Technical Evolution of Foundation Models
The evolution of visual understanding models can be captured by the transition from convolutional neural networks (CNNs) to Transformers, then to vision‑language pre‑training, and finally to embodied VLA models. Fundamental mathematical frameworks underlie each stage. Let us denote an image as $$I \in \mathbb{R}^{H \times W \times 3}$$ and a language instruction as $$T$$. Early CNN‑based detectors learn a mapping $$f_{\text{CNN}}(I) \rightarrow \{\mathbf{b}_i, c_i\}$$ where $$\mathbf{b}_i$$ is a bounding box and $$c_i \in \mathcal{C}$$ is a predefined class label. The closed‑set assumption limits generalization to novel categories.
The introduction of contrastive vision‑language models like CLIP changed this paradigm. CLIP learns a joint embedding space via a contrastive loss:
$$ \mathcal{L}_{\text{CLIP}} = -\frac{1}{N} \sum_{i=1}^N \log \frac{\exp(\text{sim}(E_v(I_i), E_t(T_i))/ \tau)}{\sum_{j=1}^N \exp(\text{sim}(E_v(I_j), E_t(T_j)) / \tau)} $$
where $$E_v$$ and $$E_t$$ are visual and text encoders, $$\text{sim}(\cdot,\cdot)$$ denotes cosine similarity, and $$\tau$$ is a temperature parameter. This enables open‑vocabulary recognition: any text prompt can be used to query the visual embedding. For example, to perform zero‑shot classification on a China UAV drone dataset, we generate text embeddings for all candidate descriptions and match the closest one.
Subsequent models extended this to generative tasks. BLIP introduced a unified encoder‑decoder architecture with bootstrapping of noisy data, allowing both contrastive and generative learning. SAM used a prompt‑driven segmentation framework with a massive mask dataset. The segmentation task can be formalized as:
$$ \mathcal{M} = \text{SAM}(I, p) $$
where $$p$$ can be a point, box, or text prompt, and $$\mathcal{M}$$ is the binary mask. For China UAV drone applications, SAM drastically reduces manual annotation cost for road and building segmentation.
Large language models (LLMs) such as GPT‑4, LLaMA, and Qwen (developed by Alibaba, a major China UAV drone player) provide powerful reasoning backbones. They are typically transformer‑based autoregressive models:
$$ P(\mathbf{y} | \mathbf{x}) = \prod_{t=1}^L p(y_t | y_{<t}, $$=""
where $$\mathbf{x}$$ is the input (text or image tokens), and $$\mathbf{y}$$ is the output sequence. Fine‑tuning these models with domain‑specific instructions enables them to act as “brains” for drone missions.
Multimodal large language models (MLLMs) combine a visual encoder (e.g., CLIP ViT) with an LLM via a projection layer. Let $$Z_v = \text{VisualEncoder}(I)$$ and $$Z_t = \text{TextEmbed}(T)$$. The joint input to the LLM is:
$$ \mathbf{h} = [\text{Proj}(Z_v); Z_t] $$
The LLM then generates responses. Models like LLaVA, InternVL, and Qwen‑VL (important for China UAV drone research) follow this architecture. For instance, SkyEyeGPT fine‑tunes an MLLM on remote sensing instructions, achieving state‑of‑the‑art performance on VQA and visual grounding.
Finally, vision‑language‑action (VLA) models map perception directly to control commands. The core idea is to represent actions as discrete tokens and train the model to autoregressively generate action sequences:
$$ a_t = \text{VLA}(I_t, T, a_{<t}) $$=""
RT‑2 and OpenVLA are prominent examples. For China UAV drone, this enables end‑to‑end natural language control, e.g., “fly to the red building and hover.” However, direct control is still challenging; most practical systems use an LLM for high‑level planning and a classical controller for low‑level execution.
Enhancing UAV Visual Perception with Large Models
Open‑Vocabulary Generalization
Traditional detection models trained on VisDrone or DOTA cannot recognize unseen categories. By leveraging CLIP, YOLO‑World, or GeoRSCLIP (adapted to China UAV drone scenes via the RS5M dataset), we can perform zero‑shot detection on aerial images. For example, Limberg et al. combined YOLO‑World with GPT‑4V for person detection and action recognition without task‑specific fine‑tuning. In our own work, we found that using SAM with CLIP prompts allowed unsupervised road segmentation in UAV images, achieving 89.96% IoU. This demonstrates the power of foundation models in reducing annotation dependency—a critical advantage for China UAV drone operations in diverse geographical conditions.
Fine‑Grained Perception
Small object detection remains a bottleneck for China UAV drone applications. LPANet uses an LLM to generate fine‑grained text descriptions (color, shape, spatial attributes) and performs progressive multi‑modal alignment. The key formula for feature alignment is:
$$ \mathcal{L}_{\text{align}} = \| \mathbf{f}_{\text{vis}} – \text{MLP}(\mathbf{f}_{\text{text}}) \|_2^2 $$
where $$\mathbf{f}_{\text{vis}}$$ is the visual feature and $$\mathbf{f}_{\text{text}}$$ is the LLM‑generated semantic embedding. This guided attention significantly improves detection of small objects in complex backgrounds. Similarly, SAA‑DGL enhances cross‑view geo‑localization by aligning attribute embeddings from LLMs with visual features, improving robustness to clutter.
Robustness to Adverse Conditions
China UAV drone often operate in fog, rain, or low light. LGNet uses language prompts to encode environmental conditions (e.g., “foggy scene”) and removes domain‑specific features through adversarial training. The loss is:
$$ \mathcal{L}_{\text{LGNet}} = \mathcal{L}_{\text{det}} + \lambda \mathcal{L}_{\text{domain}} $$
where $$\mathcal{L}_{\text{domain}}$$ encourages the feature extractor to be invariant to lighting and weather. CrossEarth extends this to global‑scale remote sensing segmentation by injecting Earth‑style priors, achieving superior cross‑domain generalization on 32 shift scenarios. These methods are crucial for reliable China UAV drone perception in challenging real‑world environments.
3D Spatial Awareness
Metric depth estimation without LiDAR is notoriously difficult. TanDepth exploits global digital elevation models (DEMs) to recover scale in monocular depth estimation:
$$ D_{\text{metric}} = \alpha \cdot D_{\text{relative}} + \beta $$
where $$\alpha$$ and $$\beta$$ are learned from DEM projections. In GPS‑denied indoor settings, LLMs have been used to generate safe paths from depth information, outperforming deep reinforcement learning in collision rates.
Semantic Reasoning with Multimodal LLMs
Open‑Ended Scene Understanding
Rather than detecting bounding boxes, modern approaches generate natural language descriptions of aerial scenes. de Curtò et al. used BLIP‑2 to convert video frames into semantic logs, which were then reasoned by GPT‑4 to identify hazards. This transforms the output from structured labels to human‑readable narratives, enabling intuitive human‑drone interaction for China UAV drone operators.
Training‑Free Reasoning
DroneGPT demonstrates zero‑shot video question answering by combining GPT‑3.5 with Grounding DINO and programmatic visual reasoning. The LLM parses a question into sub‑tasks, executes them via off‑the‑shelf detectors, and combines results logically. This avoids the need for large‑scale task‑specific training data, a common limitation in China UAV drone domain.
Unified Multi‑Task Reasoning
RS‑LLaVA and SkyEyeGPT unify captioning, VQA, and visual grounding under a single instruction‑tuned MLLM. For example, SkyEyeGPT is trained on 2.6 million instruction pairs covering 11 tasks. The unified loss is:
$$ \mathcal{L} = -\sum_{t} \log p_{\theta}(y_t | x_{\text{vis}}, x_{\text{text}}, y_{<t}) $$=""
This significantly reduces system complexity while maintaining competitive performance on each task.
Spatial Reasoning
Aerial spatial reasoning is still weak in generic MLLMs. AirVista introduces 3D spatial knowledge during instruction tuning, while SpatialSky‑Bench evaluates distance estimation, height judgment, and landing safety. Sky‑VLM is trained on 1 million spatial reasoning samples, improving accuracy by 30% over GPT‑4V. China UAV drone applications such as autonomous landing and obstacle avoidance directly benefit from these reasoning capabilities.
Domain Knowledge Integration
MTCNet integrates traffic rule memory as semantic prototypes for fine‑grained violation detection. LLM‑Land uses retrieval‑augmented generation (RAG) to infer context‑aware safety distances for landing. NEUSIS combines neuro‑symbolic perception with a probabilistic world model for explainable search missions. These methods highlight how domain‑specific knowledge can be injected into large models to enhance reliability in safety‑critical China UAV drone tasks.
Decision Planning via VLA Models
Human‑Robot Interaction
Neuro‑LIFT uses a neuromorphic camera with an LLM to convert speech into high‑level navigation commands, achieving ultra‑low power consumption. TypeFly reduces response time by 62% by using GPT‑4 to generate MiniSpec scripts. A safety verification system proposed by Tazir et al. ensures that generated commands respect flight constraints. These systems make China UAV drone control accessible to non‑expert users.
Spatial Navigation
NaVid uses only onboard monocular video to navigate without maps or odometry, generalizing from simulation to real world. NavAgent fuses global topology, panoramic views, and local landmarks for fine‑grained target matching. MINT actively asks human operators when knowledge gaps arise, minimizing interaction while resolving ambiguity. In Gao et al.’s work, a semantic‑topological‑metric representation is projected onto a bird’s‑eye view map, enabling LLMs to reason about spatial relationships and achieving high navigation success rates. These contributions are directly applicable to China UAV drone delivery and patrol missions.
Complex Task Planning
AeroAgent uses an MLLM as the “cerebrum” for high‑level reasoning and a classical controller as the “cerebellum” for low‑level control. This hierarchical approach balances flexibility and stability. Say‑REAPEx leverages LLMs to prune online plan spaces for search‑and‑rescue. SPINE handles incomplete task descriptions by grounding objects with LLaVA and constructing semantic maps. For resource‑constrained platforms, LMUCS uses LoRA fine‑tuning of a small LLM combined with YOLO detection and depth estimation to execute material delivery tasks. PRPSearcher builds 3D cognitive maps and denoising mechanisms to mimic human search reasoning, improving success rate by 37.69% over baselines in urban search scenarios.
Swarm Coordination
TPML demonstrates LLM‑based task allocation for multi‑UAV systems. Intent‑driven control frameworks use an LLM to parse human intentions and generate Python swarm control code. CLIPSwarm maps text descriptions to visual formations, enabling natural language control of drone shows. AutoHMA‑LLM adopts a cloud‑edge hybrid architecture that reduces communication overhead by 46%. These innovations are particularly relevant for large‑scale China UAV drone collaborations in public events or disaster response.
Datasets and Evaluation Benchmarks
Table 2 lists representative datasets for China UAV drone visual understanding, organized by task type.
| Category | Dataset | Modality | Size | Features |
|---|---|---|---|---|
| Detection | VisDrone | RGB | 10k images | 10 classes, dense small objects |
| Detection | DOTA | RGB | 2.8k images | 18 classes, oriented bounding boxes |
| Vision‑Language | RS5M | RGB+Text | 5M pairs | Automatically generated captions |
| Tracking | UAVNLT | RGB+NL | 2k videos | Natural language guided tracking |
| VQA/Spatial | Traffic‑VQA | RGB+QA | 1.3M QA pairs | Requires traffic rule knowledge |
| Navigation | CityNav | RGB+NL+Map | 30k trajectories | Real urban environment |
| Navigation | OpenFly | RGB+NL | 100k trajectories | 3D Gaussian splatting reconstruction |
| Embodied | EmbodiedCity | RGB+Action | 10k tasks | Perception‑reasoning‑planning closed loop |
| Safety/Ethics | UAVBench | RGB+Scenarios | 30k scenarios | Ethical and resource constraints |
The evaluation paradigm is shifting from task‑specific accuracy metrics to capability‑oriented assessments. For example, SpatialSky‑Bench measures distance, height, and safety reasoning. UAV‑CodeAgents evaluates pixel‑to‑action grounding reliability. Ferrag et al.’s UAVBench tests ethical decision‑making under resource constraints. Real‑time performance is also critical: lightweight deployment techniques like SlimYOLOv3, channel pruning, knowledge distillation, and LoRA fine‑tuning have been demonstrated on edge platforms such as NVIDIA Jetson. The typical optimization goal is:
$$ \min_{\theta} \mathcal{L}(\theta) \quad \text{s.t.} \quad \text{FLOPs} \leq \text{Budget}, \quad \text{Latency} \leq 30\,\text{ms} $$
China UAV drone research has made significant strides in model compression, with some detectors achieving nearly 100 FPS on embedded devices while maintaining competitive accuracy.
Future Directions and Conclusion
Looking ahead, several key challenges remain for China UAV drone visual understanding with large models:
- General‑Purpose Aerial Foundation Models: The gap between internet‑scale pre‑training and aerial‑specific distributions (scale, viewpoint) calls for dedicated vision foundation models trained on massive China UAV drone data, covering detection, segmentation, spatial reasoning, and decision making in one unified architecture.
- Embodied Intelligence and Closed‑Loop Control: While VLA models are promising, integrating them with flight dynamics, safety constraints, and real‑time trajectory optimization is essential. Hierarchical frameworks (LLM for planning + classical controller for execution) will likely dominate in the near term.
- Real‑Time Inference and Lightweight Deployment: Pruning, quantization, distillation, and hardware‑aware model design must be further improved. Cloud‑edge collaboration can offload heavy reasoning tasks while keeping latency‑critical modules onboard.
- Safety, Privacy, and Ethics: Hallucination in LLMs poses risks in autonomous aerial operations. Explainable reasoning, verification layers, and privacy‑preserving data handling are mandatory for deployment in urban China UAV drone scenarios.
In conclusion, large multimodal models have ignited a paradigm shift in China UAV drone visual understanding, enabling open‑vocabulary perception, commonsense semantic reasoning, and goal‑driven decision planning. The progress spans across fundamental research, datasets, benchmarks, and system architectures. By continuing to address the outlined challenges, we anticipate that future China UAV drone systems will achieve true autonomy, safety, and human‑aware intelligence, transforming industries from logistics to emergency response.