We present a novel method for online depth estimation and mesh reconstruction from UAV drone imagery by integrating a Truncated Signed Distance Function (TSDF) with global planar priors. Our approach addresses the challenges of large disparity variations, low‑texture regions, and the need for real‑time performance in drone‑based 3D reconstruction. The key idea is to leverage the TSDF field built from historical frames to cast prior depth and normal maps, which then guide the computation of multi‑view matching costs via planar homography transformations. We further incorporate a cost‑ratio strategy to mitigate occlusions and fuse global planar information to refine depth in challenging areas. Experimental results on four typical UAV drone datasets demonstrate that our method generates accurate and complete depth maps and surface meshes at a rate better than one second per frame, outperforming state‑of‑the‑art online reconstruction methods.
1. Introduction
Unmanned Aerial Vehicles (UAV drones) have become essential platforms for acquiring 3D spatial information in fields such as high‑precision mapping, smart city planning, and disaster response. Traditional offline 3D reconstruction pipelines, while achieving centimeter‑level accuracy, are time‑consuming and cannot meet the online requirements of emergency scenarios. Recent advances in multi‑view depth estimation and incremental mesh reconstruction have paved the way for online 3D modelling, yet many existing methods suffer from degraded accuracy in regions with large disparity changes, low texture, or repeated patterns.
To overcome these limitations, we propose a method that fuses a TSDF field with global planar surfaces. Instead of relying solely on photometric consistency, we use ray‑casting from a historical TSDF fusion to obtain a down‑sampled prior depth map and normal map for the current frame. These priors are used to compute plane‑induced homography transformations for multi‑view matching, significantly improving the cost accuracy in areas of large disparity. A cost‑ratio based occlusion handling step further eliminates false matches. For high‑resolution depth, we incrementally fuse global planar priors extracted from the down‑sampled depth map, optimizing the matching cost within a constrained depth range derived from the initial depth. This not only reduces computational time but also enhances depth quality in low‑texture and repetitive‑texture regions. Finally, the optimized depth map is fused into the TSDF volume, from which a surface mesh is extracted using Marching Cubes. The entire pipeline achieves online performance (better than 1 second per frame) on a desktop platform with GPU acceleration.
2. Method
Our method consists of three main modules: down‑sampled depth map computation, global‑plane‑guided depth optimization, and incremental mesh reconstruction. The overall framework is illustrated by the following conceptual diagram (note: the actual figure is represented by the link below).
2.1 Down‑sampled Depth Map Computation
For each incoming UAV drone image, we first reduce its resolution (e.g., to 800×600) to enable efficient processing. The depth space is discretized into L=64 non‑uniform samples according to the formula
$$
z_l = \frac{z_{\min} z_{\max}}{z_{\min} + l \left( z_{\max} – z_{\min} \right) / (L-1)},
$$
where \(l = 0,1,\dots,L-1\) and \([z_{\min},z_{\max}]\) is the search range. For a pixel \(p\) at depth \(z_l\) in the current frame \(f_i\), its projected pixel coordinate in a reference frame \(f_j\) is obtained via the homography based on the camera poses estimated by an ORB‑SLAM3 and RTK loosely‑coupled system:
$$
\begin{aligned}
p’_{f_j} &= H_{f_j}^{f_i}(p, p’_{f_j}, z_l)\, p, \\
H_{f_j}^{f_i}(p, p’_{f_j}, z_l) &= K_{f_j} \left( R_{f_i \to f_j} – \frac{t_{f_i \to f_j} n_p^T}{z_l} \right) K_{f_i}^{-1}.
\end{aligned}
$$
Here \(K\) is the intrinsic matrix, \(R\) and \(t\) denote rotation and translation between frames, and \(n_p\) is the prior normal vector from ray‑casting. The matching cost is computed using a modified Census transform within a local window:
$$
c_{f_j}^{f_i}(u,v,z_l) = \text{Hamming}\left( \otimes s(I_{f_i}(u\pm i, v\pm j)),\; \otimes s(I_{f_j}[H_{f_j}^{f_i}(u’\pm i, v’\pm j,1)]) \right).
$$
To handle occlusions, we introduce a cost‑ratio strategy. For two reference frames (typically the previous and next frames in the sequence), we compute two costs \(c_{f_i-1}^{f_i}\) and \(c_{f_i+1}^{f_i}\). If one cost is below a threshold \(\hat{c}=5\) and the relative difference between them exceeds 0.5, we adaptively reduce the weight of the larger cost:
$$
C(p,z_l) = w_{f_i-1}^{f_i} c_{f_i-1}^{f_i} + w_{f_i+1}^{f_i} c_{f_i+1}^{f_i},
$$
where
$$
w_{f_i-1}^{f_i} = 0.5 – 0.5 \times \frac{|c_{f_i-1}^{f_i} – c_{f_i+1}^{f_i}|}{c_{f_i-1}^{f_i} + c_{f_i+1}^{f_i}}.
$$
After building the 3D cost volume \(C(p,z_l)\), we apply semi‑global matching (SGM) with eight‑path aggregation. The aggregated cost is minimized via winner‑take‑all (WTA) plus parabolic interpolation to obtain the sub‑pixel depth. This produces the initial down‑sampled depth map. Table 1 summarises the edge error statistics at different stages of this module, demonstrating the effectiveness of occlusion handling and prior guidance.
| Stage | Completeness | Edge MAE (m) |
|---|---|---|
| No occlusion, no prior | 0.8238 | 0.1730 |
| With occlusion handling | 0.8892 | 0.1560 |
| Final down‑sampled depth (with TSDF prior) | 0.9439 | 0.1380 |
2.2 Global Plane Guided Depth Optimization
To obtain a high‑resolution depth map from the original UAV drone image, we refine the down‑sampled depth by incorporating global planar priors. First, we extract planar surfaces from the down‑sampled depth map, obtaining for each plane its equation \((n_\pi, p_\pi)\) and a set of contour points \(\beta\) via the Proena method and OpenCV’s contour detection. Then, for a new frame, we fuse its planes into a global planar model: if the distance between plane centers and plane normals is below a threshold, we average the normals and update the convex hull of the contour points.
During cost optimization, we compute the original‑resolution matching cost within a constrained depth range around the initial depth \(l_p\) with a small search interval \(\delta l = 8\):
$$
\hat{l}_p^{\max} = l_p + \delta l,\quad \hat{l}_p^{\min} = l_p – \delta l.
$$
The optimized cost is defined as
$$
\hat{C}(p, z_l) = C(p, z_l) \times \left(1 – e^{-(z_l – \tilde{z}_l)^2 / \lambda_z} \cdot e^{-\|n_\pi^T \tilde{n}_p\| / \lambda_n} \right),
$$
where \(\tilde{z}_l\) and \(\tilde{n}_p\) are the initial depth and normal from the down‑sampled map, \(n_\pi\) is the global plane normal, and \(\lambda_z = \lambda_n = 0.5\). This formulation encourages the depth to stay close to the initial estimate and to align with global planar surfaces. For aggregation, we incorporate an adaptive penalty: the discrete jump step between two pixels in a neighbourhood is set to \(\Delta L = \text{round}(l_p – l_q)\), and the smoothing term becomes
$$
\psi(l_p, l_q) =
\begin{cases}
0, & l_p + \Delta L = l_q, \\
P_1, & |l_p + \Delta L – l_q| = 1, \\
P_2, & |l_p + \Delta L – l_q| > 1.
\end{cases}
$$
The aggregated cost is then minimized similarly to the down‑sampled stage, yielding the final high‑resolution depth map.
2.3 Incremental Mesh Reconstruction
We adopt an incremental TSDF fusion approach. Each pixel with its final depth \(\bar{z}_l\) is back‑projected into 3D space using
$$
P_V = \bar{z}_l K^{-1} R_{f_i} p + T_{f_i}.
$$
The signed distance between the point and the voxel center along the ray is truncated to \([-1,1]\) if it lies within a threshold \(\tau\):
$$
tt(V) = \text{clamp}\left(\frac{\bar{z}_l – z_v}{\tau}, -1, 1\right).
$$
The global TSDF value and weight are updated incrementally:
$$
T_t(V) = \frac{T_{t-1}(V) \cdot W_{t-1}(V) + tt(V) \cdot W_t(V)}{W_{t-1}(V) + W_t(V)},\quad W_t(V) = W_{t-1}(V) + 1.
$$
After all frames are processed, Marching Cubes extracts the zero‑crossing surface to produce the final mesh.
3. Experiments and Analysis
3.1 Datasets
We evaluate our method on four UAV drone datasets covering typical scenarios: urban buildings, mountainous terrain, residential areas, and rural villages. Table 2 provides the specifications of each dataset.
| Dataset | UAV drone model | GSD (cm) | Image resolution | Number of images |
|---|---|---|---|---|
| Urban | DJI Phantom 4 | 1.67 | 4000×3000 | 85 |
| Mountain | eBee Classic | 5.40 | 5472×3648 | 347 |
| Residential | eBee X | 2.39 | 3648×5472 | 101 |
| Rural | DJI Phantom 4 | 1.88 | 2736×1824 | 482 |
We use MetaShape to generate ground‑truth meshes and rendered depth maps. For the Mountain and Rural datasets, ground‑control points (9 and 13 respectively) are available for absolute accuracy assessment. For comparison, we apply PatchmatchNet and Mobile3DRecon, both followed by the same TSDF‑based mesh reconstruction as our method for fair comparison.
3.2 Depth Estimation Evaluation
We evaluate the depth maps quantitatively using three metrics: absolute relative error (Abs Rel), mean absolute error (MAE), and root‑mean‑square error (RMSE). Outliers beyond 3σ are removed. Table 3 shows the results.
| Dataset | Method | Abs Rel (%) | MAE (m) | RMSE (m) |
|---|---|---|---|---|
| Urban | PatchmatchNet | 0.5571 | 0.1679 | 0.2426 |
| Mobile3DRecon | 0.5223 | 0.1583 | 0.2271 | |
| Ours | 0.4027 | 0.1221 | 0.1695 | |
| Mountain | PatchmatchNet | 0.1722 | 0.3516 | 0.4054 |
| Mobile3DRecon | 0.1398 | 0.2874 | 0.3074 | |
| Ours | 0.1125 | 0.2323 | 0.2656 | |
| Residential | PatchmatchNet | 0.3098 | 0.1957 | 0.2375 |
| Mobile3DRecon | 0.3418 | 0.2251 | 0.2658 | |
| Ours | 0.2491 | 0.1398 | 0.1624 | |
| Rural | PatchmatchNet | 0.1975 | 0.1357 | 0.1723 |
| Mobile3DRecon | 0.1527 | 0.1102 | 0.1548 | |
| Ours | 0.1231 | 0.1039 | 0.1357 |
Our method consistently achieves the lowest errors across all metrics. Compared to Mobile3DRecon (the best competitor), the average reduction in MAE is about 19.2% (e.g., from 0.1583 m to 0.1221 m in Urban). The improvements are especially pronounced in regions with large disparity (Mountain) and low‑texture surfaces (Residential).
3.3 Mesh Reconstruction Evaluation
Table 4 reports the mesh reconstruction quality metrics: completeness (Comp), F1‑score, MAE, standard deviation (STD), and check‑point MAE for the two datasets with ground‑control points.
| Dataset | Method | Comp | F1‑Score | MAE (m) | STD (m) | Check‑point MAE (m) |
|---|---|---|---|---|---|---|
| Urban | PatchmatchNet | 0.8970 | 0.8576 | 0.1611 | 0.1624 | — |
| Mobile3DRecon | 0.8587 | 0.8973 | 0.1380 | 0.1396 | — | |
| Ours | 0.8694 | 0.9285 | 0.1100 | 0.1192 | — | |
| Mountain | PatchmatchNet | 0.9549 | 0.8306 | 0.3853 | 0.2977 | 0.2455 |
| Mobile3DRecon | 0.9190 | 0.8763 | 0.2430 | 0.2455 | 0.2397 | |
| Ours | 0.9345 | 0.9075 | 0.1961 | 0.2102 | 0.1980 | |
| Residential | PatchmatchNet | 0.9565 | 0.8714 | 0.2963 | 0.2665 | — |
| Mobile3DRecon | 0.7742 | 0.8267 | 0.3143 | 0.3070 | — | |
| Ours | 0.9286 | 0.9182 | 0.1578 | 0.1545 | — | |
| Rural | PatchmatchNet | 0.9692 | 0.9229 | 0.1160 | 0.1116 | 0.0920 |
| Mobile3DRecon | 0.9517 | 0.9383 | 0.0813 | 0.0851 | 0.0785 | |
| Ours | 0.9529 | 0.9442 | 0.0782 | 0.0808 | 0.0694 |
Our method achieves the best F1‑score and lowest MAE on all four datasets. On average, the mesh MAE is reduced by 19.3% compared to Mobile3DRecon. The absolute check‑point errors also confirm the superior geometric accuracy of our meshes, especially in the Mountain and Rural datasets where ground‑truth control points are available.
3.4 Performance Evaluation
Table 5 lists the per‑frame runtime of each module in our method at 800×600 resolution, and Table 6 compares the total average time per frame with other methods.
| Module | Time (ms) |
|---|---|
| Down‑sampled depth | 2.96 |
| Global plane optimization | 5.39 |
| TSDF fusion | 4.13 |
| Mesh extraction (Marching Cubes) | 546.2 |
Note that depth estimation and TSDF fusion run in parallel on GPU, so the wall‑clock time per frame is dominated by the depth estimation part (<10 ms) plus mesh extraction (~546 ms), yielding < 1 second per frame. Table 6 shows the average total time for different datasets and resolutions.
| Dataset | Image resolution | PatchmatchNet (ms) | Mobile3DRecon (ms) | Ours (ms) |
|---|---|---|---|---|
| Urban | 800×600 | 857.2 | 819.3 | 546.0 |
| Mountain | 800×600 | 768.5 | 715.2 | 491.6 |
| Residential | 532×800 | 757.6 | 712.8 | 489.6 |
| Rural | 1280×852 | 985.6 | 957.8 | 654.9 |
Our method is consistently faster, with an average reduction of 39.9% compared to PatchmatchNet and 32.1% compared to Mobile3DRecon. The speed gain stems from the constrained depth search and the efficient plane‑guided cost volume.
4. Conclusion
We have presented an online depth estimation and mesh reconstruction method tailored for UAV drone imagery. By combining a TSDF field with global planar surfaces, our approach effectively handles large disparity variations, low‑texture areas, and occlusions. The key contributions include: (1) using ray‑casted prior depth and normal maps from historical TSDF fusion to compute plane‑homography guided multi‑view matching costs; (2) a cost‑ratio occlusion handling strategy that improves cost fusion accuracy; (3) global plane information incrementally fused to optimize the cost volume in a narrowed depth range, boosting both quality and speed; and (4) an online pipeline that processes each frame in under one second. Experimental evaluations on four diverse UAV drone datasets demonstrate that our method consistently outperforms PatchmatchNet and Mobile3DRecon in both depth and mesh accuracy, while requiring less computation time. The proposed method is promising for applications such as emergency response, where rapid and reliable 3D reconstruction from UAV drone imagery is critical.