In modern infrastructure inspection, multi-rotor drones have become indispensable tools due to their agility and efficiency. However, ensuring safe navigation in complex environments, such as around power lines and towers, remains a critical challenge. As a researcher focused on autonomous systems, I have developed a real-time obstacle avoidance technology for multi-rotor drone inspection systems. This approach leverages binocular vision and feature-based algorithms to detect and avoid obstacles dynamically, enhancing the safety and reliability of drone operations. The core of this technology involves extracting foreground targets from stereo images, detecting key features using scale-invariant methods, and calculating minimal distance points for evasion. Throughout this article, I will elaborate on the system design, algorithmic details, experimental validation, and practical implications, incorporating mathematical formulations and data summaries to provide a comprehensive understanding.
The automated inspection system for multi-rotor drones consists of several integrated modules that work in harmony to control flight, process data, and execute obstacle avoidance. The primary components include the intelligent terminal, flight module, and inspection operation module. The intelligent terminal serves as the control center, managing drone flight through data and image channels. It enables real-time monitoring of the multi-rotor drone’s status, task assignment, and safety checks. The flight module, often equipped with Real-Time Kinematic (RTK) positioning, achieves centimeter-level accuracy in locating the multi-rotor drone. It utilizes sensors like LiDAR to capture point cloud data of structures such as transmission towers, along with visible light and infrared cameras for image acquisition. The inspection operation module handles data processing, storage, and supports automated flight with real-time obstacle detection. This modular architecture ensures that the multi-rotor drone can perform inspections efficiently while adapting to dynamic environments. To illustrate the system’s workflow, consider the following table summarizing the key modules and their functions:
| Module | Primary Function | Key Components |
|---|---|---|
| Intelligent Terminal | Controls drone flight and visualizes results | Data channels, image processing units |
| Flight Module | Acquires inspection data and precise positioning | RTK GPS, LiDAR, cameras |
| Inspection Operation Module | Processes data and enables automatic obstacle avoidance | Data storage, algorithm processors |
The real-time obstacle avoidance technology relies heavily on binocular vision to perceive the environment. By processing stereo images, the system can reconstruct three-dimensional information and identify potential obstacles. The first step involves determining foreground targets in the inspection video using ORB-SLAM techniques. This simultaneous localization and mapping method builds a 3D feature point cloud model of the background and identifies foreground elements by analyzing pixel distributions. For a global inspection environment video, the background 3D feature point cloud is represented as $P_G = \{p_i^G\}$ for $i = 1, \dots, N$, where $N$ is the number of background feature points, and each point $p_i^G$ has coordinates $(X_i^G, Y_i^G, Z_i^G)$. Similarly, for a selected segment of the inspection video, the point cloud is $P_A = \{p_i^A\}$ for $i = 1, \dots, M$ with $M < N$. The transformation between these point clouds is given by the homogeneous transformation matrix:
$$ \begin{bmatrix} X_G \\ Y_G \\ Z_G \\ 1 \end{bmatrix} = \begin{bmatrix} a_{11} & a_{12} & a_{13} & t_1 \\ a_{21} & a_{22} & a_{23} & t_2 \\ a_{31} & a_{32} & a_{33} & t_3 \\ 0 & 0 & 0 & 1 \end{bmatrix} \cdot \begin{bmatrix} X_A \\ Y_A \\ Z_A \\ 1 \end{bmatrix} $$
where $a_{ij}$ are rotation matrix elements and $t_k$ are translation components. To extract foreground features, a 3D mean shift algorithm clusters background points based on their spatial distribution. For each point $p_i^A$ in the partial video, its neighborhood $N_{p_i^A} = \{p_k : d < T_r\}$ is defined, where $d$ is the Euclidean distance between points, calculated as:
$$ d = \sqrt{(X_i^G – X_k^G)^2 + (Y_i^G – Y_k^G)^2 + (Z_i^G – Z_k^G)^2} $$
and $T_r$ is a distance threshold. The mean of the neighborhood points is computed as $\overline{N_{p_i^A}} = K^{-1} \sum_{p_k \in N_{p_i^A}} p_k$, where $K$ is the number of points in the neighborhood. This process iterates until point coordinates converge below a threshold $\Delta T$. A point is classified as foreground if the number of global background points in its neighborhood is less than $K/3$, effectively isolating obstacles for the multi-rotor drone to avoid.
Once foreground targets are identified, the Scale-Invariant Feature Transform (SIFT) algorithm is employed for robust feature detection and matching. This step is crucial for the multi-rotor drone to recognize obstacles under varying conditions, such as changes in scale or orientation. The SIFT process begins by generating a scale space representation of the foreground image. The scale space $L(x, y, \sigma)$ is constructed by convolving the image $I(x, y)$ with a Gaussian kernel $G(x, y, \sigma)$:
$$ L(x, y, \sigma) = G(x, y, \sigma) * I(x, y) $$
where the Gaussian function is defined as:
$$ G(x, y, \sigma) = \frac{1}{2\pi\sigma^2} e^{-\frac{x^2 + y^2}{2\sigma^2}} $$
Here, $\sigma$ represents the scale parameter, controlling the level of smoothing. Larger $\sigma$ values produce coarser scales, capturing general features, while smaller $\sigma$ values preserve fine details. To detect stable features, a Difference of Gaussian (DoG) scale space is built by subtracting adjacent scale images:
$$ D(x, y, \sigma) = L(x, y, k\sigma) – L(x, y, \sigma) $$
where $k$ is a constant multiplier. This DoG space highlights potential keypoints that are invariant to scale changes. A Gaussian pyramid is constructed by repeatedly down-sampling the image, with each octave containing multiple scales. Preliminary keypoint detection involves comparing each point in the DoG space to its neighbors across scales and locations. To refine keypoint localization, a Taylor expansion of the DoG function is used:
$$ D(\mathbf{x}) = D + \frac{\partial D^T}{\partial \mathbf{x}} \mathbf{x} + \frac{1}{2} \mathbf{x}^T \frac{\partial^2 D}{\partial \mathbf{x}^2} \mathbf{x} $$
where $\mathbf{x} = (x, y, \sigma)^T$. Setting the derivative to zero yields the offset $\hat{\mathbf{x}} = – \frac{\partial^2 D^{-1}}{\partial \mathbf{x}^2} \frac{\partial D}{\partial \mathbf{x}}$, and substituting back gives the response value $D(\hat{\mathbf{x}}) = D + \frac{1}{2} \frac{\partial D^T}{\partial \mathbf{x}} \hat{\mathbf{x}}$. Keypoints with low response magnitudes are discarded. Additionally, edge responses are eliminated using the Hessian matrix $H$:
$$ H = \begin{bmatrix} D_{xx} & D_{xy} \\ D_{xy} & D_{yy} \end{bmatrix} $$
Let $\alpha$ and $\beta$ be the larger and smaller eigenvalues, respectively. The trace and determinant of $H$ are:
$$ \text{Tr}(H) = D_{xx} + D_{yy} = \alpha + \beta $$
$$ \text{Det}(H) = D_{xx} D_{yy} – D_{xy}^2 = \alpha \beta $$
Keypoints are retained if $\frac{\text{Tr}(H)^2}{\text{Det}(H)} < \frac{(\gamma + 1)^2}{\gamma}$, where $\gamma$ is a ratio threshold. For orientation assignment, the gradient magnitude $m(x, y)$ and direction $\theta(x, y)$ are computed as:
$$ m(x, y) = \sqrt{(L(x+1, y) – L(x-1, y))^2 + (L(x, y+1) – L(x, y-1))^2} $$
$$ \theta(x, y) = \arctan\left( \frac{L(x, y+1) – L(x, y-1)}{L(x+1, y) – L(x-1, y)} \right) $$
A histogram of gradient orientations in the keypoint’s neighborhood is built, and the peak direction is assigned as the keypoint’s orientation. Finally, a 128-dimensional descriptor vector is generated by dividing the region around the keypoint into sub-blocks and computing orientation histograms. This descriptor enables robust matching between stereo images. For feature matching, the Euclidean distance between descriptors $E_i$ and $S_i$ from left and right images is calculated:
$$ d(E_i, S_i) = \sum_{j=1}^{128} (e_{i,j} – s_{i,j})^2 $$
The nearest neighbor distance ratio test is applied, where a match is accepted if the ratio of the smallest to second-smallest distance is below a threshold. This process allows the multi-rotor drone to accurately identify obstacle points in real-time.
To achieve real-time obstacle avoidance, the system calculates the spatial coordinates of the point with the minimum distance to the multi-rotor drone. Suppose the closest obstacle target is $K(x, y, z)$, with corresponding points $K_{\text{left}}$ and $K_{\text{right}}$ in the left and right images. The 3D coordinates can be derived using stereo vision geometry:
$$ x = \frac{N_1 X_{\text{left}}}{D}, \quad y = \frac{N_1 Y}{D}, \quad z = \frac{N_1 f}{D} $$
where $N_1$ is the baseline distance between the cameras, $f$ is the focal length, $X_{\text{left}}$ is the horizontal distance from $K_{\text{left}}$ to the left image’s optical center, and $D = X_{\text{left}} – X_{\text{right}}$ is the disparity. The obstacle point with the peak disparity value is selected as the avoidance target. The multi-rotor drone then adjusts its flight path to maintain a safe distance, ensuring continuous inspection without collisions. The following table outlines the key parameters involved in this distance calculation:
| Parameter | Description | Typical Value |
|---|---|---|
| $N_1$ | Baseline distance between cameras | 0.1 m |
| $f$ | Focal length | 500 pixels |
| $D$ | Disparity | Variable (pixels) |
| $T_r$ | Distance threshold for neighborhood | 0.5 m |
In experimental validation, the proposed obstacle avoidance technology was implemented on a multi-rotor drone inspection system tasked with surveying power transmission lines. The multi-rotor drone was equipped with binocular cameras and RTK positioning, flying at a speed of 1 m/s. The actual distance to an obstacle was set at 200 m, and the system’s performance was evaluated based on its ability to detect and avoid the obstacle accurately. The foreground extraction process successfully isolated key targets, such as power towers, from the stereo images. Even under rotations and scaling of the inspection images, the SIFT-based feature detection maintained high matching accuracy, demonstrating the robustness of the approach for multi-rotor drone operations. The distance calculation results showed minimal error compared to ground truth data, with deviations typically below 5%. This precision is critical for ensuring the multi-rotor drone maintains safe clearance from obstacles during autonomous flights. The table below summarizes the experimental results for distance estimation under different conditions:
| Test Scenario | Actual Distance (m) | Calculated Distance (m) | Error (%) |
|---|---|---|---|
| Static obstacle | 200.0 | 198.5 | 0.75 |
| Rotated image (30°) | 200.0 | 199.2 | 0.40 |
| Scaled image (0.8x) | 200.0 | 201.1 | 0.55 |
| Dynamic flight | 200.0 | 197.8 | 1.10 |
The effectiveness of the obstacle avoidance technology can be further analyzed through mathematical models of drone kinematics and sensor fusion. For instance, the motion of the multi-rotor drone can be described using state equations that incorporate position, velocity, and acceleration. Let $\mathbf{p} = [x, y, z]^T$ represent the drone’s position, and $\mathbf{v} = [v_x, v_y, v_z]^T$ its velocity. The discrete-time state update can be expressed as:
$$ \mathbf{p}_{k+1} = \mathbf{p}_k + \mathbf{v}_k \Delta t + \frac{1}{2} \mathbf{a}_k (\Delta t)^2 $$
$$ \mathbf{v}_{k+1} = \mathbf{v}_k + \mathbf{a}_k \Delta t $$
where $\mathbf{a}_k$ is the acceleration vector at time step $k$, and $\Delta t$ is the sampling interval. The obstacle avoidance algorithm influences $\mathbf{a}_k$ by generating repulsive forces based on the detected minimal distance points. If the obstacle position is $\mathbf{p}_{\text{obs}}$, the repulsive acceleration can be modeled as:
$$ \mathbf{a}_{\text{rep}} = -k_{\text{rep}} \frac{\mathbf{p} – \mathbf{p}_{\text{obs}}}{\|\mathbf{p} – \mathbf{p}_{\text{obs}}\|^3} $$
for a gain constant $k_{\text{rep}}$. This ensures that the multi-rotor drone smoothly deviates from obstacles while maintaining stable flight. Additionally, sensor noise and uncertainties are addressed through Kalman filtering, which fuses data from RTK, LiDAR, and visual sensors. The state estimation equation is:
$$ \hat{\mathbf{x}}_{k|k} = \hat{\mathbf{x}}_{k|k-1} + K_k (\mathbf{z}_k – H \hat{\mathbf{x}}_{k|k-1}) $$
where $\hat{\mathbf{x}}$ is the estimated state, $K_k$ is the Kalman gain, $\mathbf{z}_k$ is the measurement vector, and $H$ is the observation matrix. This integration enhances the reliability of distance calculations for the multi-rotor drone, even in noisy environments.
In conclusion, the real-time obstacle avoidance technology presented here significantly advances the capabilities of multi-rotor drone inspection systems. By combining ORB-SLAM for foreground extraction and SIFT for feature detection, the method achieves high accuracy in identifying and evading obstacles. The modular system design ensures seamless interaction between components, enabling autonomous flight and data processing. Experimental results confirm that the approach effectively handles image transformations and calculates distances with minimal error, underscoring its practicality for real-world applications. Future work will focus on optimizing computational efficiency for faster response times and integrating deep learning techniques to enhance feature recognition. As multi-rotor drones continue to evolve, such technologies will play a pivotal role in expanding their use in critical infrastructure monitoring and beyond.