As the low-altitude economy expands and intelligent unmanned systems mature, China UAV drones have evolved from simple remotely piloted platforms into sophisticated aerial agents capable of environmental perception, semantic reasoning, and autonomous decision-making. The core of this transformation lies in visual understanding, which serves as the primary sensory modality for acquiring information and enabling situational awareness. In recent years, the emergence of visual foundation models (VFMs), vision-language models (VLMs), and multimodal large language models (MLLMs) has fundamentally reshaped the paradigm for visual understanding in China UAV drones, providing unprecedented opportunities for operation in complex, open environments.
To systematically analyze this evolving landscape, we propose a capability-oriented analytical framework that organizes visual understanding into three hierarchical levels: basic perception, semantic reasoning, and decision planning. This framework allows us to examine recent research from a unified, capability-evolution perspective. From the task perspective, we construct a comprehensive taxonomy of UAV visual understanding tasks, including basic object perception, event semantic analysis, spatial environment understanding, and flight decision-making. From the technical perspective, we review the methodological evolution from conventional deep learning and vision foundation models to large language models (LLMs), multimodal large language models, and embodied vision-language-action (VLA) models. From the capability perspective, we analyze how large models enhance UAV visual perception, semantic reasoning, and decision planning.
The unique characteristics of drone-captured imagery present significant challenges. High-altitude perspectives lead to extreme scale variations, long-range observations result in small object representations, and complex backgrounds combined with dynamic environmental changes require robust temporal reasoning. Early approaches relied on fully supervised deep learning, but these task-specific models were limited by their dependence on large-scale annotated data and their inability to generalize to unseen categories or cross-domain scenarios. The introduction of large models has provided a breakthrough by enabling open-vocabulary recognition, complex semantic reasoning, and direct mapping from high-level instructions to low-level flight control.
In this paper, we refer to China UAV drones as the primary application platform, as this region has witnessed significant advancements in both drone technology and the integration of large models for visual understanding. Our survey aims to provide a systematic overview of the research progress, key datasets, evaluation benchmarks, and future directions for this rapidly evolving field.
To begin, we categorize the typical visual understanding tasks in China UAV drones across four dimensions: basic perception, event semantics, spatial understanding, and decision planning. The following table summarizes the representative tasks, their primary challenges, and application scenarios.
| Understanding Level | Task Type | Primary Challenges | Application Scenarios |
|---|---|---|---|
| Basic Object Perception | Object Detection, Segmentation, Tracking | Small objects, dense distribution, severe occlusion, low resolution, background clutter | Traffic monitoring, urban management, industrial inspection, disaster rescue |
| Event Semantic Analysis | Action Recognition, Visual Question Answering, Intention Recognition, Accident Detection | Long temporal dependencies, fine-grained action differences, complex reasoning, lack of domain knowledge | Ground-air interaction, security patrol, low-altitude safety, emergency response |
| Spatial Environment Understanding | 3D Spatial Reasoning, Geo-localization | Monocular depth uncertainty, complex spatial structures, GPS-denied environments | 3D mapping, environmental sensing, wild search and rescue, military reconnaissance |
| Flight Decision Planning | Visual Obstacle Avoidance, Autonomous Landing, Vision-Language Navigation, Target Search, Task Planning, Collaborative Planning, Formation Control | Dynamic obstacle prediction, high real-time response, multi-agent coordination, path optimization | Autonomous flight, complex environment cruise, package delivery, collaborative inspection |
The technical evolution that underpins these capabilities began with general vision foundation models. For instance, CLIP aligns image and text through contrastive learning, enabling open-vocabulary recognition by transforming classification into an image-text retrieval problem. BLIP extends this with a unified encoder-decoder architecture for generative tasks like image captioning and visual question answering. SAM achieves prompt-driven segmentation, showing strong generalization in pixel-level spatial modeling. These models provide the robust visual representations necessary for downstream UAV tasks.
Large language models like GPT, LLaMA, Qwen, and DeepSeek have introduced powerful reasoning capabilities. Multimodal large language models (MLLMs) integrate visual encoders with LLMs to enable complex visual question answering and scene understanding. Notable examples include GPT-4V, Gemini, LLaVA, Intern-VL, and Qwen-VL. These models allow China UAV drones to not only detect objects but also interpret events, understand spatial relationships, and interact with humans through natural language. The final stage of this technical evolution is embodied VLA models, such as RT-2 and OpenVLA, which connect perception to action by mapping visual observations and language instructions directly to control policies. This enables drones to perform complex task planning and interactive decision-making in real-world environments.
The following table provides a comparative analysis of these technical paradigms as applied to China UAV drones.
| Technical Paradigm | Core Capability | Suitable UAV Tasks | Advantages | Main Limitations |
|---|---|---|---|---|
| Vision-Language Alignment Models (CLIP, DINO) | Image-text alignment, open-vocabulary recognition | Basic perception: scene classification, open-vocabulary object recognition, retrieval | Strong open-category transferability, reduces annotation dependency | Cannot perform precise detection alone; insensitive to small objects and fine-grained differences |
| Image-Text Generative Models (BLIP, BLIP-2) | Image captioning, VQA, cross-modal generation | Basic perception & semantic analysis: aerial image captioning, scene QA, event summarization | Converts visual data to natural language for human-drone interaction | Limited understanding of complex spatial relations, abnormal events, and domain-specific knowledge |
| General Segmentation Models (SAM) | Prompt-driven segmentation, region mask generation | Basic perception: object segmentation, land-cover extraction, semi-automatic annotation | Significantly reduces annotation costs for UAV data | Lacks category semantics; unstable for small, low-contrast, or complex textured objects |
| Multimodal Large Language Models (GPT-4V, LLaVA, Intern-VL) | Image/video understanding, VQA, complex semantic reasoning | Semantic analysis & planning: aerial VQA, anomaly explanation, task instruction understanding, scene reasoning, preliminary decision support | Unified generative interface for diverse UAV tasks; extends from perception to planning | |
| Vision-Language-Action Models (RT-2, OpenVLA) | Vision-language-action mapping, task planning, embodied decision-making | Flight decision planning: visual navigation, target search, autonomous landing, obstacle avoidance, multi-agent collaboration | Bridges visual understanding with flight control; enables closed-loop perception-decision-control | Primarily focused on robotic manipulation; adapting to UAVs requires solving real-time and safety constraints |
We now detail how these large models enhance three core capability dimensions of China UAV drones: visual perception, semantic reasoning, and decision planning.
Visual Perception Enhancement
Large models significantly improve the perceptual capabilities of China UAV drones in four key areas.
Open-Vocabulary Generalization: Traditional models are constrained by fixed class sets. By leveraging the zero-shot capabilities of models like CLIP and YOLO-World, researchers have enabled drones to detect and segment objects described only by natural language, without task-specific training. For example, combining YOLO-World with GPT-4V enables zero-shot person detection and action recognition in drone imagery. Similarly, fine-tuning a VLM on large-scale remote sensing image-text pairs, like in the GeoRSCLIP model, significantly boosts zero-shot classification performance. SAM, when combined with language prompts, allows for unsupervised road scene parsing from high-resolution UAV imagery, achieving high accuracy without manual pixel-level annotations. These approaches significantly enhance the flexibility of China UAV drones in dealing with novel targets.
Fine-Grained Perception: Aerial images often contain small or similarly-appearing targets. To address this, researchers have used LLMs to generate fine-grained text descriptions (e.g., shape, color) that serve as semantic anchors to guide multi-modal feature alignment, improving small object detection. For geo-localization, LLM-based attribute alignment helps encode color, structure, and orientation details into visual features, enhancing robustness in cluttered scenes. In multi-object tracking, using natural language descriptions instead of just bounding boxes helps resolve identity switches among visually similar targets, with CLIP’s multi-modal alignment effectively reducing tracking ambiguity.
Cross-Domain Robustness: Drones often operate under adverse conditions like fog or low-light, which cause domain shifts. Language-guided mechanisms can reduce dependence on specific visual domains, enabling detectors to remain stable under changes in scale and viewpoint. MLLMs can adaptively extract environmental context information to aid detection in extreme weather, while large vision foundation models trained with global-scale data and style injection can effectively mitigate domain shifts caused by geographical differences, demonstrating superior generalization in cross-domain segmentation tasks.
3D Spatial Awareness: Lacking LiDAR, many China UAV drones rely on monocular depth estimation, which suffers from scale uncertainty. Recent work leverages LLMs to integrate geographical priors, such as global digital elevation models, to recover metric depth. Furthermore, LLMs have been shown to effectively analyze depth information for path planning in GPS-denied environments, achieving lower collision rates than traditional deep reinforcement learning models.
Vision-Language Semantic Reasoning
MLLMs endow China UAV drones with the ability to reason about visual scenes, moving beyond simple detection to interpreting events and making logical inferences.
Open Semantic Understanding: Drones can now transform raw video streams into human-readable semantic logs, describing objects, potential hazards, and scene contexts. This paradigm shift represents a move from black-box detection results to interpretable, near-human semantic explanations, providing a unified semantic interface for subsequent decision-making.
Training-Free Inference: LLMs demonstrate impressive zero-shot reasoning capabilities. By combining LLMs with existing visual tools (like Grounding DINO), a framework like DroneGPT can parse natural language instructions into modular programs for visual reasoning on drone video, without any task-specific fine-tuning. This shows that complex compositional tasks can be addressed without a dedicated training data loop.
Unified Multi-Task Reasoning: To handle diverse tasks like captioning, QA, and localization, researchers are adopting unified visual-language modeling. Models like RS-LLaVA and SkyEyeGPT are fine-tuned to perform multiple remote sensing tasks with a single architecture, significantly improving system efficiency and cross-task knowledge sharing for China UAV drones.
Complex Spatial Relation Reasoning: Drone navigation inherently requires 3D spatial reasoning. Benchmarks like SpatialSky-Bench, which include distance estimation, altitude assessment, and landing safety analysis, have revealed that even the most advanced MLLMs struggle with fine-grained spatial perceptions from aerial views. Models fine-tuned with 3D spatial knowledge, like Sky-VLM, show that multi-granularity spatial reasoning is critical for effective drone scene analysis.
Domain-Knowledge Reasoning: In specific applications like traffic violation detection, models need to integrate domain-specific rules. By incorporating external knowledge bases (e.g., traffic rules), models can perform logical reasoning to parse complex violations. In safety-critical tasks like autonomous landing, MLLMs can infer context-aware safety margins for obstacle avoidance, dynamically adapting thresholds for different target types, improving flight safety over traditional controllers.
Visual Decision Planning
The integration of VLA models is transforming China UAV drones from passive observers into active agents capable of autonomous decision-making and task execution.
Human-Drone Interactive Decision-Making: LLMs significantly lower the barrier for drone control by allowing users to issue commands in natural language. Systems can map human speech to high-level flight plans, enabling intuitive interaction. To address the latency issue inherent in LLM token generation, lightweight task-planning languages have been developed to minimize redundant generation, reducing response times by over 60%. Verification systems have also been proposed to ensure that natural language commands are legally translatable into flight actions, enhancing safety in human-drone collaboration.
Complex Spatial Navigation: Visual-language navigation (VLN) for drones requires understanding ambiguous instructions and complex spatial relations. To combat ambiguity, multi-modal prompts combining images with text have been introduced. Active querying mechanisms allow the agent to ask clarifying questions when instructions are vague. Improving spatial reasoning involves fusing global topological maps, panoramic views, and local landmarks, or projecting semantic masks onto a top-down map to enhance the LLM’s spatial reasoning for action prediction. Some models have shown that navigation is possible using only a monocular camera, without maps or odometry, simulating human-like navigation logic and avoiding cumulative sensor noise.
Complex Task Planning: For multi-stage tasks like search and rescue, a “brain-cerebellum” architecture is often adopted. An MLLM acts as the “brain”, handling high-level task decomposition and reasoning, while a traditional controller acts as the “cerebellum,” executing precise flight commands. LLMs can also be used to filter online planning spaces, increasing efficiency, or to infer safety margins for replanning. This demonstrates that larger models are best suited for task understanding and strategy generation, while classic control theory and real-time optimization remain essential for safety and stability. A collaborative architecture where the VLM handles semantic planning and a smaller module handles safety constraints is a promising approach.
Multi-Agent Collaboration: LLMs are also being applied to multi-UAV systems for swarm control, collaborative search, and formation flight. Large models can act as a central planner to parse high-level human intentions and automatically generate swarm control code, or to interpret leader drone images for formation planning in real-time. Vision-language models can also be used to map text descriptions to visual formations for drone shows. For heterogeneous swarms with resource constraints, a hybrid architecture with a cloud-based central planner and device-specific miniature models significantly reduces communication overhead while maintaining task accuracy.
To support this research, a wide range of datasets and benchmarks have been developed. These are transitioning from single-modality, closed-set annotated data to multi-modal, open-vocabulary, task-oriented resources. Key datasets for task-specific evaluation include VisDrone and UAVDT for detection and tracking, UAV-Human for behavior analysis, RS5M for large-scale image-text alignment, and benchmarks like SpatialSky-Bench for spatial reasoning. For application-specific needs, datasets like FloodNet and TrafficNight cover disaster response and night-time traffic, respectively. Simulation platforms like EmbodiedCity provide high-fidelity digital twins for safe and repeatable evaluation of embodied agents.
The evaluation metrics are also evolving. Traditional metrics focused on task accuracy (e.g., mAP for detection). However, the community is now emphasizing a capability-oriented evaluation framework that assesses open-world generalization, long-horizon spatial reasoning, instruction following, and real-time inference performance. Evaluating the “perception-reasoning-action” closed loop is becoming crucial, as is assessing the consistency of a model’s reasoning with physical reality. The benchmark, UAVBench, even incorporates ethical safety and resource-constrained decision-making, testing models for hallucinations when faced with ambiguous instructions. This shift reflects the broader move from evaluating isolated perception performance to evaluating integrated visual intelligence in China UAV drones.
In conclusion, large multimodal models are profoundly reshaping the landscape of visual understanding for China UAV drones. They have enabled a leap from closed-set recognition to open-vocabulary perception, from simple detection to complex semantic reasoning, and from passive observation to autonomous, embodied decision-making. Despite this tremendous progress, significant challenges remain. The development of general-purpose vision foundation models specifically tailored for aerial scenarios is a crucial next step. Realizing embodied drone intelligence requires bridging the semantic-to-action gap for stable, closed-loop control in dynamic environments. Real-time inference and lightweight deployment on resource-constrained edge platforms remain a critical bottleneck, requiring advances in model compression, hardware-software co-design, and cloud-edge collaboration. Finally, ensuring safety, reliability, and privacy is paramount, especially given the hallucination risks of large models in safety-critical aviation tasks. Addressing these challenges will pave the way for the next generation of highly autonomous, trustworthy, and fully intelligent aerial systems, further solidifying the role of China UAV drones in shaping the future of the low-altitude economy and beyond.