With the widespread application of UAV drones in logistics, inspection, emergency rescue, and other fields, autonomous obstacle avoidance capability in complex dynamic environments has become a key technological bottleneck limiting their development. Traditional single-modal perception schemes exhibit significant limitations under conditions such as sudden illumination changes and unstructured obstacles. Although multi-sensor fusion technology has partially addressed environmental representation issues in recent years, it still faces severe challenges in real-time performance and reliability. Existing research primarily focuses on optimizing obstacle avoidance algorithms for static environments, lacking systematic design for motion prediction and collision avoidance strategies regarding dynamic obstacles. Simultaneously, sensor fault tolerance mechanisms and energy efficiency balance issues have not received sufficient attention.
In this study, we propose a collaborative sensing and decision-making architecture based on multi-sensor fusion to address the autonomous obstacle avoidance needs of UAV drones in complex environments. Through heterogeneous sensor spatio-temporal calibration and feature-level fusion methods, a LiDAR-vision-IMU joint perception model is constructed, and a hybrid path planning strategy integrating dynamic threat spheres and rolling time domain optimization is designed. This approach aims to enhance the safety and adaptability of UAV drones operating in GNSS-denied environments.
Demand Analysis
Environmental Perception Requirements
The autonomous obstacle avoidance capability of UAV drones in complex environments first relies on an accurate environmental perception system. Visual sensors provide rich texture information due to their high resolution, but they are prone to misdetection under environmental influences such as lighting changes and haze. LiDAR acquires three-dimensional point cloud data with centimeter-level accuracy by actively emitting laser pulses, yet it suffers from scattering interference in rain or snow. Millimeter-wave radar operates in all-weather conditions with a detection range exceeding 200 meters, but its spatial resolution is insufficient for identifying small obstacles. The rapid identification of dynamic targets requires sensors with millisecond-level response speeds, while static map construction demands centimeter-level spatial accuracy. A single sensor cannot meet these multi-dimensional perception requirements.
To quantify these requirements, we analyze the performance characteristics of various sensors commonly used in UAV drones. The following table summarizes key parameters:
| Sensor Type | Strengths | Limitations | Typical Response Time | Spatial Accuracy |
|---|---|---|---|---|
| Visual Camera | High resolution, rich texture | Sensitive to lighting, weather | 30-100 ms | Pixel-level |
| LiDAR | Accurate 3D point clouds | Affected by precipitation, cost | 10-50 ms | Centimeter-level |
| Millimeter-wave Radar | All-weather, long range | Low spatial resolution | 5-20 ms | Meter-level |
| IMU | High-frequency motion data | Drift over time | 1-10 ms | N/A |
The fusion of these sensors is crucial for UAV drones to achieve robust perception. The environmental perception system must handle dynamic obstacles with speeds up to 20 m/s, requiring update rates faster than 50 Hz. Additionally, the system should maintain accuracy in diverse conditions, such as low-light environments or cluttered scenes.
Real-Time Decision Requirements
The real-time decision-making ability of UAV drones directly determines the safety and adaptability of the obstacle avoidance system. As flight speed increases from 5 m/s to 20 m/s, the tolerance for computational delay in obstacle avoidance algorithms decreases exponentially. When speed exceeds 15 m/s, the entire process from environmental perception to control command must be completed within 50 ms. Traditional threshold-triggered mechanisms ensure nanosecond-level response speeds through preset safe distances but struggle with dynamic obstacle trajectory prediction and path optimization in complex terrains.
In contrast, deep learning-based predictive decision-making models can predict obstacle movement trends 300 ms in advance through spatio-temporal sequence modeling. However, the computational complexity of convolutional neural networks introduces an additional delay of 10–30 ms. Under this contradiction, the system needs to establish a dynamic balance mechanism between response time and decision accuracy, requiring obstacle avoidance response times not exceeding 80 ms and path re-planning efficiency reaching over 5 times per second.
We formulate the real-time decision problem as an optimization challenge. Let the UAV drone state at time $t$ be $s(t)$, and the obstacle state be $o(t)$. The decision function $D(s(t), o(t))$ must generate a control action $a(t)$ within a time constraint $\Delta t$. The objective is to minimize the collision probability while maximizing path efficiency:
$$ \min_{a(t)} \mathbb{P}_{\text{collision}}(s(t), o(t), a(t)) \quad \text{subject to} \quad \text{computation time} \leq \Delta t $$
For UAV drones operating at high speeds, $\Delta t$ is typically less than 100 ms. This necessitates efficient algorithms that can process sensor data and compute actions rapidly.
System Reliability Requirements
The reliability requirements of UAV drone systems must balance hardware fault tolerance and energy efficiency constraints. Under strong electromagnetic interference or extreme temperature and humidity environments, the failure probability of a single sensor increases significantly. A cross-validation mechanism among multi-source heterogeneous sensors is required, automatically switching to backup data channels when a certain sensor’s data is abnormal. For conditions where communication links are interrupted or delayed by over 500 ms, the system should possess local decision-making capabilities to maintain basic obstacle avoidance functions, while monitoring communication status in real-time through heartbeat detection mechanisms.
Fault-tolerant control algorithms must achieve fault mode identification and isolation under limited computational resources to avoid cascading failures due to a single module fault. In terms of energy efficiency, sensor sampling frequencies should be dynamically adjusted based on task priority. For example, in hover state, the LiDAR scanning frequency can be reduced from 20 Hz to 5 Hz to save power, while during flight, computational resources are adaptively allocated according to obstacle density.
We define a reliability metric $R_{\text{sys}}$ for the UAV drone system as a function of sensor redundancy, fault detection latency, and energy consumption:
$$ R_{\text{sys}} = \alpha \cdot R_{\text{sensor}} + \beta \cdot R_{\text{fault}} – \gamma \cdot E_{\text{consumption}} $$
where $R_{\text{sensor}}$ represents sensor redundancy, $R_{\text{fault}}$ is fault tolerance, $E_{\text{consumption}}$ is energy consumption, and $\alpha, \beta, \gamma$ are weighting coefficients. The system aims to maximize $R_{\text{sys}}$ under operational constraints.
System Architecture and Module Design
Multi-Sensor Fusion Framework Design
The multi-sensor fusion framework employs a hierarchical progressive architecture to achieve efficient integration of heterogeneous data. In time synchronization, a unified time baseline is constructed by combining hardware trigger signals and software compensation. An improved linear interpolation method compensates for differences in sensor sampling frequencies, ensuring time alignment errors of less than 2 ms among LiDAR 10 Hz point cloud data, camera 30 Hz image frames, and IMU 100 Hz inertial data.
For spatial calibration, a camera-LiDAR joint calibration model is established, deriving the rigid body transformation matrix:
$$ T = \begin{bmatrix} R & t \\ 0 & 1 \end{bmatrix} $$
where $R \in \mathbb{R}^{3 \times 3}$ denotes the rotation matrix and $t \in \mathbb{R}^{3 \times 1}$ is the translation vector. Calibration parameters are obtained using a checkerboard calibration board, and nonlinear optimization algorithms minimize reprojection errors.
A dual-channel fusion mechanism is designed for multi-modal data characteristics: Kalman filtering handles motion state estimation of sequential data such as IMU gyroscopes and accelerometers, achieving real-time estimation of UAV drone pose through state prediction and measurement update stages. A deep neural network constructs a vision-point cloud feature encoder, using 3D convolutional layers to extract point cloud voxel features and connecting them with edge features extracted by a 2D convolutional neural network from images. These are fused via an attention mechanism to form a joint feature vector.
In the feature fusion layer, a cross-modal attention module is introduced:
$$ \alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{N} \exp(e_{ik})} $$
This module automatically computes the association weight $\alpha_{ij}$ between LiDAR point cloud feature point $i$ and image pixel $j$, achieving semantic-level fusion of geometric information and texture features. During data preprocessing, outlier rejection rules are set. For instance, when IMU angular velocity exceeds 200°/s or LiDAR point cloud density falls below 50 points/m², data quality detection is triggered, activating backup sensor data compensation mechanisms.
To enhance real-time performance, a feature extraction pipeline is deployed on an FPGA chip, hardware-accelerating computation-intensive operations such as point cloud downsampling and image pyramid construction, ensuring single-frame data processing delay is controlled within 15 ms. This is critical for UAV drones that require rapid responses.
Environmental Perception Module Design
The environmental perception module adopts a three-level progressive processing architecture to achieve dynamic obstacle identification. In the obstacle detection phase, LiDAR point clouds are downsampled via voxel grids and quickly segmented using an improved Euclidean clustering algorithm:
$$ d(p_i, p_j) = \sqrt{(x_i – x_j)^2 + (y_i – y_j)^2 + (z_i – z_j)^2} < \theta_{\text{cluster}} $$
where $p_i, p_j$ are adjacent point cloud coordinates, and $\theta_{\text{cluster}}$ is set as a clustering threshold of 0.3 m, combined with a region-growing method to eliminate ground point cloud interference. The visual detection channel employs a lightweight YOLOv7-tiny model; when deployed on a Jetson AGX Orin platform, the parameter count is compressed to 40% of the original model through layer pruning, achieving real-time detection speed of 30 fps.
The classification submodule constructs a multi-modal feature fusion network, concatenating bounding box size features of point cloud clusters with HOG features from image ROI regions, and inputs them into a support vector machine for obstacle type discrimination. For motion tracking, a cross-sensor data association mechanism is designed, projecting the 3D center coordinates of obstacles detected by LiDAR onto the image plane and matching them with pixel displacement vectors computed by optical flow methods.
A state prediction model based on Kalman filtering is established:
$$ x_k = F x_{k-1} + w_k $$
where $x_k$ represents the obstacle state vector at time $k$, containing 6-dimensional parameters such as position and velocity, $F$ is the state transition matrix, and $w_k$ is process noise. Dual matching of HSV color histogram features and point cloud shape features addresses ID switching issues in multi-target tracking.
Sensor anomaly monitoring logic sets a triple verification mechanism: when the number of point cloud clusters drops suddenly by 50% for five consecutive frames, a LiDAR fault alarm is triggered; when image detection confidence remains below 0.3, camera lens contamination diagnosis is activated; if the deviation between IMU angular velocity and visual optical flow estimates exceeds 15°/s, sensor misalignment is determined. In fault states, the system automatically switches to backup perception modes, such as generating pseudo point cloud data using binocular stereo matching when LiDAR fails. To reduce computational load, a dynamic resource scheduling strategy is designed; when fewer than three obstacles are detected, point cloud normal vector computation is disabled, concentrating GPU computing power on target classification tasks. This optimization is essential for UAV drones operating with limited onboard resources.
Path Planning Module Design
The path planning module uses a hierarchical architecture to coordinate global optimality and local obstacle avoidance. For 3D space characteristics, A* algorithm and RRT algorithm applicability are compared: A* algorithm quickly generates minimum-cost paths in structured environments through a heuristic function $h(n)$, but its computational complexity reaches $O(n^2)$ when grid map resolution is set to 0.5 m, making it difficult to handle dynamic obstacles; RRT algorithm builds a search tree through random sampling, showing better expansibility in complex unstructured spaces, but requires over 3,000 iterations to converge to an optimal path.
Therefore, a hybrid planning strategy is designed: during initialization, the A* algorithm generates coarse global paths; when dynamic obstacles are detected, it switches to RRT for local re-planning. A dynamic threat sphere model constructs a safety protection zone based on obstacle motion prediction:
$$ B(t) = \left\{ p \in \mathbb{R}^3 \mid \| p – (o_k + v_k t) \| \leq r_{\text{safe}} + v_{\text{max}} \Delta t \right\} $$
where $o_k$ is the obstacle’s current position, $v_k$ is its estimated velocity, $r_{\text{safe}}$ is a safety radius of 1.5 m, $v_{\text{max}}$ is the UAV drone’s maximum speed of 15 m/s, and $\Delta t$ is a prediction horizon of 3 s. This model transforms the future motion envelope of dynamic obstacles into spatio-temporal constraints, updating trajectory segments every 0.5 s via a rolling time domain optimization strategy.
The optimization objective function is:
$$ \min \int \left( \| q(t) – q_{\text{ref}} \|^2 + \lambda \| u(t) \|^2 \right) dt $$
where $q(t)$ is the UAV drone pose, $q_{\text{ref}}$ is the reference path point, $u(t)$ is control input, and $\lambda$ is a smoothing weight coefficient of 0.3. Trajectory optimization introduces flight attitude constraints, limiting pitch angle $\theta$ and roll angle $\phi$ to no more than 30°, and heading angle change rate constraint as:
$$ | \dot{\psi} | \leq \frac{v}{R_{\text{min}}} $$
where $v$ is current speed and $R_{\text{min}} = 2$ m is the minimum turning radius. B-spline curves parameterize the trajectory, ensuring curvature continuity through convex optimization, with node vector intervals set to 0.2 s, guaranteeing generated trajectories satisfy dynamic feasibility. To improve real-time performance, collision detection is parallelized on GPU, using octree spatial indexing to accelerate threat sphere queries, compressing a single planning cycle to within 120 ms. This efficiency is vital for UAV drones navigating dense environments.
Decision Control Module Design
The decision control module adopts a hierarchical hybrid control architecture to coordinate multi-modal commands. Based on a finite state machine, six core obstacle avoidance modes are designed: normal cruise, static obstacle circumvention, dynamic obstacle evasion, emergency hover, low-battery return, and fault safety mode. State transition conditions are defined by:
$$ T(s_i \to s_j) = \begin{cases} 1 & \text{if } d_{\text{obs}} < 1.2 \text{ m} \wedge E_{\text{bat}} > 20\% \\ 0 & \text{otherwise} \end{cases} $$
where $d_{\text{obs}}$ is the nearest obstacle distance and $E_{\text{bat}}$ is remaining battery level. The underlying controller uses an improved PID structure for attitude tracking:
$$ u(t) = K_p e(t) + K_i \int_0^t e(\tau) d\tau + K_d \frac{de(t)}{dt} $$
where $e(t)$ is attitude angle error; $K_p, K_i, K_d$ parameters are adjusted online via a deep deterministic policy gradient algorithm, with the Actor network outputting parameter increments $\Delta K \in [-0.2, 0.2]$. The reinforcement learning policy module constructs state space $S = \{ v, \theta, \phi, d_{\text{obs}} \}$, action space $A = \{ \Delta K_p, \Delta K_i, \Delta K_d \}$, and reward function designed as:
$$ r = – \left( \alpha \| e \|^2 + \beta \| \dot{u} \|^2 + \gamma \mathbb{I}_{\text{collision}} \right) $$
where $\alpha = 0.6$, $\beta = 0.3$, $\gamma = 10$ are weight coefficients for tracking error, control rate change, and collision indicator, respectively. Control command priority is set to four levels: emergency obstacle avoidance command (L1) > fault handling command (L2) > trajectory tracking command (L3) > energy optimization command (L4). When multiple commands conflict, a weighted voting mechanism is used, with L1 commands having veto power.
For actuator saturation issues, a dynamic limiting algorithm is designed:
$$ u_{\text{clipped}} = \begin{cases} u_{\text{max}} & \text{if } u > u_{\text{max}} \\ u_{\text{min}} & \text{if } u < u_{\text{min}} \\ u & \text{otherwise} \end{cases} $$
$u_{\text{max}}$ and $u_{\text{min}}$ are dynamically calculated based on current motor speed; when the speed difference among four rotors exceeds 200 rpm, torque balance compensation is triggered. Mode switching processes include a 50 ms state retention period to avoid high-frequency oscillations, and historical state caching enables abnormal rollback functionality. These controls ensure stable operation of UAV drones under varying conditions.
Simulation and Practical Testing
Simulation Environment Setup and Testing
The simulation environment is built on ROS Melodic and Gazebo 11 platforms, constructing a digital twin system containing mixed indoor-outdoor scenes. The UAV drone model integrates digital twins of RealSense D435i camera, Velodyne VLP-16 LiDAR, and BMI088 inertial measurement unit. Sensor noise model parameters are configured as shown in the following table:
| Sensor Type | Noise Model | Mean | Variance |
|---|---|---|---|
| LiDAR | Gaussian Noise | 0.02 m | 0.005 m² |
| IMU | Random Walk | 0.05° | 0.1° |
| Visual Sensor | Salt-and-Pepper Noise | N/A | 5% pixel ratio |
Static obstacle maze scenes set three typical structures: continuous Z-shaped channels (minimum width 1.8 m), cross intersections (channel width 2.4 m), and spiral ascending corridors (slope angle 15°), with obstacle surface reflectivity gradually changing from 0.1 (light-absorbing material) to 0.9 (specular material) to test sensor adaptability. Dynamic obstacle raid scenes deploy four threat models: horizontal moving obstacles (speed 3 m/s), vertical diving obstacles (acceleration 2 m/s²), random bouncing spheres (restitution coefficient 0.7), and group flying objects (quantity 10, spacing 1.5 m).
The testing metric quantification system includes spatio-temporal dual dimensions. Obstacle avoidance success rate is calculated as:
$$ P_{\text{success}} = \frac{N_{\text{collision-free}}}{N_{\text{total}}} \times 100\% $$
Trajectory optimization is evaluated via path length ratio $\eta_l = L_{\text{actual}} / L_{\text{ideal}}$ and smoothness $\eta_s = \sum | \kappa_i | \Delta s_i$, where $\kappa_i$ is curvature and $\Delta s_i$ is path segment arc length. Typical test scenario parameters are summarized below:
| Scenario Type | Obstacle Density (per m³) | Dynamic Obstacle Speed (m/s) | Lighting Conditions (lux) |
|---|---|---|---|
| Simple Maze | 3 | 0 | Constant 200 |
| Complex Dynamic Scene | 5 | 2–5 | Gradual 50–500 |
In extreme condition tests, when sensor noise variance increases to 150% of calibrated values, the system still maintains basic obstacle avoidance functions, but trajectory smoothness indicator $\eta_s$ increases by approximately 30%. In dynamic obstacle raid scenarios, emergency steering success rate is negatively correlated with obstacle speed, dropping to 78% at a 5 m/s speed threshold. These results highlight the challenges for UAV drones in high-speed dynamic environments.
Actual Flight Scene Verification
Actual flight verification selects temperate broadleaf forest and urban CBD areas as typical test environments. Terrain characteristics are compared as follows:
| Scene Type | Average Obstacle Height (m) | Obstacle Density (per 100 m³) | Dynamic Interference Sources | Light Variation Range (lux) |
|---|---|---|---|---|
| Forest Inspection | 8.2 | 4.3 | Birds (2–3 per minute) | 800–1,200 |
| Urban Building Complex | 54 | 17 | Pedestrians/Vehicles (20 per minute) | 200–1,500 |
In forest scenes, non-rigid obstacles formed by tree branches cause LiDAR point cloud density fluctuations up to ±15%, and visual sensors experience a 23% detection frame rate drop under dappled light effects through canopy gaps. Through multi-modal fusion, the system combines LiDAR ranging accuracy (±0.1 m) with visual semantic information, successfully identifying dead branch obstacles with diameters above 0.3 m. In building complex tests, glass curtain walls reflect 85% of LiDAR signals, causing 5% false alarm targets; after cross-sensor verification, the misjudgment rate reduces to 1.2%. These practical tests demonstrate the robustness of our approach for UAV drones in real-world settings.
Conclusion
The autonomous obstacle avoidance system for UAV drones established in this study significantly improves obstacle avoidance reliability in complex environments through multi-sensor collaborative perception and intelligent decision-making mechanisms. Actual tests show that the system outperforms traditional schemes in dynamic obstacle identification response speed and trajectory smoothness. However, perception performance under extreme weather conditions still requires improvement. Future research will focus on algorithm lightweight design and fusion with novel millimeter-wave radars to further expand application scenarios. The continuous advancement of multi-sensor fusion technologies will empower UAV drones to operate safely and efficiently in increasingly challenging environments, paving the way for broader adoption in critical fields such as autonomous delivery, infrastructure inspection, and disaster response.