In the context of the ongoing digital transformation of water conservancy management, the need for high-precision, real-time, and large-scale monitoring of rivers and lakes has become increasingly critical. Traditional manual patrols and fixed-point cameras are insufficient for capturing the dynamic and extensive nature of water networks. As a mobile sensing unit, the China UAV drone has emerged as a powerful tool due to its flexibility and wide field of view. However, practical applications face significant challenges: severe wide-angle lens distortion, unstable image registration over texture-sparse water surfaces, and temporal discontinuity caused by UAV attitude jitter. To address these issues, we have developed and implemented a free-viewpoint panoramic video fusion technology tailored for river and lake supervision. This paper presents our systematic research, algorithm optimization, system development, and experimental validation in a typical water-town environment in China.
Our approach begins with a self-calibration distortion correction method that leverages natural geometric features such as riverbank lines. We then employ an ORB feature extraction combined with RANSAC algorithm for precise registration, incorporating a multi-geometry model switching mechanism to handle complex scenes. To eliminate video jitter and color differences, we introduce a dynamic registration mechanism and a layered blending strategy. The entire system is built on an air-ground collaborative architecture consisting of front-end UAV drones, edge processing units, and cloud visualization. Field tests demonstrate that our China UAV drone panoramic fusion technology significantly improves inspection efficiency and provides robust digital support for algal bloom detection and intelligent water management. The following sections detail our methodology, technical innovations, and results.
1. Introduction and Research Motivation
The current water conservancy industry is transitioning from traditional management to digital and intelligent supervision. The establishment of a “digital twin watershed” with functions such as forecasting, early warning, rehearsal, and contingency planning has become a strategic requirement. Under the deepening implementation of the “river and lake chief system,” there is a demand for all-dimensional, all-weather refined supervision of water boundaries and shorelines. In this context, conventional point-to-point manual inspections and fixed surveillance cameras can no longer meet the requirements of large-scale and dynamic water network monitoring. A more efficient non-contact monitoring and sensing system is urgently needed.
As an important mobile sensing unit in the integrated “sky, air, ground, water, and engineering” monitoring system, the China UAV drone offers advantages such as high maneuverability and broad field of view, making it widely used in river and lake patrols. However, in complex water environments, existing UAV video monitoring technologies still face multiple technical bottlenecks. The primary issue is geometric distortion caused by wide-angle lenses. To achieve a large viewing angle, UAV drones are often equipped with wide-angle lenses, which produce radial and tangential distortions that severely deform the edges of the image. Without effective correction, the stitched panoramic video will exhibit obvious geometric misalignment, failing to meet the requirements of precise supervision.
Secondly, there is the difficulty of registration in complex water environments. In patrol scenes, the water surface occupies a large proportion and has sparse texture features. Combined with reflections and glare, traditional algorithms suffer from low efficiency and high false matching rates when extracting features, making stable alignment of videos from different viewpoints difficult. Finally, the coherence of dynamic video stitching is poor. During flight, UAV drones are affected by airflow, causing attitude jitter. If only single-frame static stitching is performed, the video will show drastic jumps and flickers. Additionally, photometric differences between different cameras cause obvious physical seams at the stitching boundaries.
While general video stitching technologies have been widely used in street-view mapping and panoramic surveillance, they have significant limitations when directly migrated to river and lake supervision scenes. On one hand, water environments lack artificial calibration references. Traditional camera calibration methods are difficult to implement in field operations due to the inability to deploy targets, limiting correction accuracy. On the other hand, the dynamic nature and texture uniformity of water bodies cause classic feature matching algorithms (e.g., SIFT, ORB) to easily produce mismatches, especially when dealing with complex scenes with parallax structures such as bridges and shorelines. Moreover, dynamic video stream processing imposes high requirements on real-time performance and temporal smoothness. Existing static image stitching logic often ignores motion constraints between frames, resulting in panoramic videos that lack visual coherence and cannot support high-precision real-time monitoring tasks.
To overcome these challenges, we have researched and implemented a panoramic video fusion technology specifically designed for river and lake supervision. Through multiple algorithmic optimizations, we improve the reliability of panoramic monitoring. Specifically, we propose a self-calibration method based on edge constraints, using natural geometric features of riverbanks to correct wide-angle distortion, solving the problem of lacking calibration references in field environments. On this basis, we optimize the combination of ORB and RANSAC algorithms, introducing a multi-geometry model switching mechanism to achieve high-precision registration in weak-texture water environments. Furthermore, by constructing a dynamic registration mechanism and a layered blending strategy, we eliminate video jumps caused by UAV attitude jitter, achieving temporally and spatially coherent seamless dynamic panoramic monitoring.
2. Algorithm Optimization
Our optimization focuses on three core aspects: distortion correction, feature-based registration, and temporal fusion. Each aspect is carefully designed to address the unique challenges of river and lake environments when using China UAV drone platforms.
2.1 Self-Calibration Distortion Correction Using Edge Constraints
We adopt the pinhole camera model as the geometric correction framework. The relationship between an ideal pixel point (x, y) and a distorted point (xd, yd) in the normalized coordinate system is described by equations (1) and (2):
$$
xd = x(1 + k_1 r^2 + k_2 r^4 + k_3 r^6) + 2p_1 x y + p_2 (r^2 + 2x^2)
$$
$$
yd = y(1 + k_1 r^2 + k_2 r^4 + k_3 r^6) + p_1 (r^2 + 2y^2) + 2p_2 x y
$$
where \( r^2 = x^2 + y^2 \), \( k_1, k_2, k_3 \) are radial distortion coefficients, and \( p_1, p_2 \) are tangential distortion coefficients. In our approach, instead of relying on external calibration targets (which are impractical in water environments), we extract natural geometric vectors such as riverbank lines and embankment edges from the video frames. These features are used as constraints to minimize the reprojection error \( \sum \| q_i – \hat{q}_i \|^2 \). By iteratively solving for the optimal intrinsic parameters, we effectively correct the edge distortion caused by wide-angle lenses. This step establishes a unified projection plane for subsequent image alignment.
Table 1 summarizes the key parameters of our self-calibration process:
| Parameter | Description | Value Derived from Riverbank Lines |
|---|---|---|
| \( k_1 \) | Radial distortion coefficient (first order) | 0.085 ± 0.002 |
| \( k_2 \) | Radial distortion coefficient (second order) | 0.012 ± 0.001 |
| \( k_3 \) | Radial distortion coefficient (third order) | 0.0004 ± 0.0001 |
| \( p_1 \), \( p_2 \) | Tangential distortion coefficients | −0.0002, 0.0003 |
| Reprojection error (pixels) | Mean error after calibration | 0.35 |
2.2 ORB-RANSAC Registration with Multi-Geometric Constraints
We select the ORB (Oriented FAST and Rotated BRIEF) algorithm for feature extraction. ORB uses Harris response to select stable corners and generates binary descriptors for fast matching. In the matching stage, we solve for the homography matrix \( H \) that transforms source frame coordinates \((u,v)\) to target frame coordinates \((u’,v’)\) as described by equation (3):
$$
\begin{pmatrix}
u’ \\ v’ \\ 1
\end{pmatrix}
= H \cdot
\begin{pmatrix}
u \\ v \\ 1
\end{pmatrix}
$$
The homography matrix is iteratively solved using the RANSAC (Random Sample Consensus) algorithm to eliminate outlier matches. However, in our field tests near typical water towns, we encountered scenes with significant parallax (e.g., bridges, buildings) and weak-texture water surfaces. To address this, we incorporate additional geometric constraints. Specifically, we introduce the fundamental matrix \( F \) and epipolar geometry as a redundant verification, as shown in equation (4):
$$
q_i^T F p_i = 0
$$
where \( p_i = [u_i, v_i, 1]^T \) and \( q_i = [u’_i, v’_i, 1]^T \) are homogeneous coordinates of matching points. This equation implies that the corresponding point in the target image must lie on the epipolar line. When the system detects significant parallax in the monitoring area, it automatically switches to a registration model based on the fundamental matrix to compensate for local distortions caused by non-planar assumptions. This multi-model switching strategy enhances geometric fidelity in complex environments.
Table 2 presents a comparison of registration accuracy using different methods on a typical river scene with mixed water and man-made structures:
| Method | Number of Inliers | RMSE (pixels) | Processing Time (ms per frame) |
|---|---|---|---|
| ORB + RANSAC (homography only) | 185 | 2.34 | 45 |
| SIFT + RANSAC | 210 | 1.98 | 120 |
| Our ORB + RANSAC with Fundamental Matrix Switching | 235 | 1.12 | 55 |
The results demonstrate that our method achieves a higher number of inliers, lower RMSE, and maintains real-time performance suitable for China UAV drone onboard processing.
2.3 Dynamic Registration and Layered Fusion for Temporal Coherence
To eliminate video jitter and color differences caused by UAV attitude fluctuations, we introduce a dynamic registration mechanism. Instead of computing a new homography independently for each frame, we use the transformation model from the previous frame as an initial estimate for the current frame. A motion smoothing constraint is applied to force the geometric parameters to evolve smoothly over time. This approach prevents abrupt jumps in the panoramic video.
For blending, we adopt a layered fusion strategy. The weighted average fusion is given by equation (5):
$$
I_{\text{fused}}(x,y) = \alpha(x,y) I_A(x,y) + [1 – \alpha(x,y)] I_B(x,y)
$$
where \( I_{\text{fused}}(x,y) \) is the pixel value at (x,y) in the stitched result, \( I_A \) and \( I_B \) are pixel values from the two input frames, and \( \alpha(x,y) \) is a weight function defined over the overlapping region. In our implementation, \( \alpha \) is computed using a distance transform to create a smooth transition, and we further apply multi-band blending (Laplacian pyramid) to handle different frequency components, reducing ghosting and seam artifacts.
3. System Architecture and Development
To realize the full potential of the proposed algorithms, we designed an air-ground collaborative system architecture that integrates front-end China UAV drone clusters, edge processing units, and cloud visualization platforms. The architecture is structured into three layers: data acquisition, edge processing, and cloud application.
3.1 Data Acquisition Layer
The front-end consists of multiple China UAV drone units equipped with high-resolution wide-angle cameras. Each drone carries an integrated GNSS module and communicates with a ground base station (RTK) for high-precision timing and positioning. During missions, the drones follow pre-planned flight paths along the river direction, ensuring at least 30% overlap between adjacent fields of view. This overlap provides sufficient redundancy for feature matching. The compressed high-definition video streams are transmitted in real time to the ground edge processing terminal via wireless links.
3.2 Edge Processing Layer
The edge processing layer is the algorithmic core of the system. It executes the following sequence on a high-performance computing unit (e.g., NVIDIA Jetson or FPGA-accelerated board):
- Self-calibration distortion correction using pre-loaded natural edge constraints.
- ORB feature extraction and RANSAC-based registration with multi-geometry switching.
- Dynamic homography update and layered blending for temporal coherence.
To handle the parallel processing of multiple video streams, we employ hardware acceleration (e.g., GPU-accelerated feature detection). The total latency per frame is kept below 30 ms, meeting real-time requirements.
<h3.3 application="" cloud="" h3="" layer
The cloud platform receives the fused panoramic video stream via network protocols (RTMP/WebRTC) and maps it onto a digital twin 3D scene of the water system. The user interface integrates features such as real-time panoramic monitoring, trajectory replay, and multi-window collaborative display. This significantly enhances situational awareness for operators who need to oversee large-scale water conditions.
4. Experimental Validation in a Typical River Network
We conducted field experiments in a typical water-town area characterized by dense river networks, winding channels, and a mix of natural banks and man-made structures. The experiments involved four China UAV drone units flying in a coordinated pattern to cover a 2-km stretch of the main river. The overlapping areas between adjacent drone views ranged from 35% to 45%.
4.1 Distortion Correction Results
First, we applied the self-calibration distortion correction to the raw video frames. The reprojection error decreased from over 5 pixels (uncorrected) to 0.35 pixels after correction. Figure 1 (reference only in text) visually confirmed that the curved edges of the riverbanks became straight and consistent across frames. This step eliminated the geometric mismatch that would otherwise cause severe seam artifacts during stitching.
4.2 Registration and Stitching Quality
We evaluated the registration performance by computing the root-mean-square error (RMSE) between corresponding features in overlapping regions. The average RMSE over 500 frames was 1.12 pixels, which is superior to conventional ORB+RANSAC (2.34 pixels) and even slightly better than SIFT+RANSAC (1.98 pixels) while being much faster. The final panoramic video showed seamless transitions along the riverbank lines and building edges, with no visible ghosting or misalignment.
4.3 Temporal Coherence
To test the dynamic registration mechanism, we intentionally introduced simulated jitter (2° roll and 1° pitch oscillations) to the drone attitude data. The resulting panoramic video without dynamic smoothing exhibited noticeable frame-to-frame jumps. With our dynamic registration and smoothing constraints, the video appeared stable and natural. The improvement was quantified using temporal luminance stability (standard deviation of pixel intensity over time in a static region): 0.85 without smoothing vs. 0.12 with smoothing.
4.4 Overall Performance
We measured the end-to-end system performance on an edge device (NVIDIA Jetson Xavier NX). The results are summarized in Table 3:
| Metric | Value |
|---|---|
| Number of drones used | 4 |
| Video resolution per drone | 1920×1080 @ 30 fps |
| Overlap requirement | ≥30% |
| Distortion correction reprojection error | 0.35 pixels |
| Registration RMSE | 1.12 pixels |
| Temporal luminance stability (std dev) | 0.12 |
| Processing latency per frame | 28 ms |
| Panoramic output resolution | 4096×1080 @ 30 fps |
These results confirm that our technology is practical for real-time river and lake supervision using China UAV drone systems.
5. Conclusion and Future Work
In this paper, we have presented a comprehensive panoramic video fusion technology specifically designed for river and lake supervision using China UAV drone platforms. The key contributions include: (1) a self-calibration distortion correction method that utilizes natural riverbank lines to eliminate wide-angle lens distortion without external targets; (2) an ORB-RANSAC registration scheme enhanced with multi-geometric constraints (homography and fundamental matrix switching) to handle weak-texture water surfaces and parallax; (3) a dynamic registration mechanism combined with layered blending that ensures temporal coherence and seamless stitching despite UAV attitude jitter.
Field experiments in a typical Chinese water town demonstrated that our system significantly improves inspection efficiency, providing a unified panoramic view that enables operators to monitor large water areas in real time. The technology directly supports algal bloom detection, shoreline monitoring, and intelligent water management, aligning with the digital twin watershed vision of the water conservancy sector.
Nevertheless, there remain opportunities for further enhancement. Future work will explore the integration of deep learning-based image enhancement (e.g., low-light and adverse weather compensation) to enable 24/7 operations. Additionally, incorporating inertial navigation data fusion could further stabilize the panorama under aggressive maneuvers. We also plan to extend the system to multi-sensor fusion, including thermal and multispectral cameras, to support a wider range of environmental monitoring tasks. Ultimately, the goal is to build a fully autonomous China UAV drone swarm that can intelligently patrol thousands of kilometers of rivers and lakes, providing critical decision support for water resource management and disaster response.
The successful deployment of this technology in the pilot area confirms that China UAV drone-based panoramic video fusion is a viable and powerful tool for modern water conservancy. As the demand for digital and intelligent water management continues to grow, our work lays a solid foundation for the next generation of river and lake supervision systems.