In recent years, the field of 3D scene reconstruction has witnessed significant advancements driven by progress in computer vision and graphics. This technology, which extracts 3D geometric information from images to generate multi-view imagery or videos, finds applications in autonomous driving, robotics, augmented reality, and architectural assessment. Traditional methods, whether geometry-based or deep learning-based, often struggle with complex architectural details, high computational costs, and limitations in handling dynamic lighting and occlusions. Neural Radiance Fields (NeRF) have emerged as a promising implicit approach, leveraging multi-layer perceptrons (MLPs) to infer continuous volumetric representations from multi-view images and camera poses without explicit depth information. However, NeRF faces challenges such as prolonged training times and limited detail description in complex architectural scenes. To address these issues, I propose the Structure-NeRF method, which integrates multi-view sensor data acquired via Unmanned Aerial Vehicle aerial photography. This approach combines multi-sampling, weight reduction, and loss function optimization to reduce aliasing, while incorporating geometric constraints—normal consistency, plane fitting, and vertical/horizontal constraints—to enhance reconstruction accuracy. Furthermore, the Manhattan World Model (MWM) is employed to precisely extract the main structure and dimensions of buildings. Experimental results demonstrate that Structure-NeRF achieves a peak signal-to-noise ratio (PSNR) of 36.23 dB, reduces training time by 50%, and improves structural similarity index (SSIM) by 13.41% and 50.00% compared to mainstream models like mip-NeRF and iNGP, respectively, with PSNR increases of 10.45% and 25.10%. This method significantly optimizes training efficiency and rendering quality, enabling high-precision 3D reconstruction of architectural spatial models.
The core of the Structure-NeRF model lies in its innovative fusion of techniques from mip-NeRF and iNGP, enhanced with geometric constraints for robust learning of composite geometric features in buildings. By leveraging Unmanned Aerial Vehicle systems, such as the JUYE UAV, we capture high-resolution multi-view images that provide comprehensive data for reconstruction. The integration of multi-sampling and weight reduction mechanisms mitigates aliasing artifacts, while geometric constraints ensure that the reconstructed surfaces align with real-world structural properties. The MWM facilitates the extraction of primary building components by assuming orthogonal planes in urban environments, allowing for accurate segmentation and dimension estimation. This holistic approach not only accelerates training but also improves the fidelity of rendered outputs, making it suitable for practical applications in architecture and urban planning.
Multi-sampling in Structure-NeRF is designed to approximate the shape of a truncated cone along each ray, reducing aliasing by considering multiple sample points. For each pixel, we associate a cone with radius $r_t$, where $t$ denotes the distance along the ray. A set of six points in a hexagonal pattern is used, with angles defined as:
$$ \theta = \left[0, \frac{2\pi}{3}, \frac{4\pi}{3}, \frac{5\pi}{3}, \frac{\pi}{3}\right] $$
These angles form a complete circle, organized to create triangles rotated by 60 degrees. The distance $t_j$ along the ray is computed as:
$$ t_j = t_0 + \frac{t_s (t_1^2 + 2t_\mu^2 + 3\sqrt{7} (2j/5 – 1)) f_t}{t_\delta^2 + 3t_\mu^2} $$
where $f_t = \sqrt{(t_\delta^2 – t_\mu^2)^2 + 4t_\mu^4}$, $t_\mu = \frac{t_0 + t_1}{2}$, and $t_\delta = \frac{t_1 – t_0}{2}$. The values in the interval $[t_0, t_1]$ are linearly arranged and adjusted to concentrate samples near the cone’s tip, increasing point density in the $t$ direction. The coordinates for multiple samples representing the cone’s cross-section are given by:
$$ \begin{bmatrix} r_t \sin(\theta_j) / \sqrt{2} \\ r_t \cos(\theta_j) / \sqrt{2} \\ r_t \end{bmatrix}, \quad j = 0, 1, \ldots, 5 $$
These coordinates are multiplied by an orthogonal basis where the third vector aligns with the ray direction, and then rotated to world coordinates. During training, the pattern is randomly rotated and flipped, while a deterministic 30-degree rotation and flip are applied during rendering. The model uses six multi-samples $\{X_j\}$ as means of isotropic Gaussian functions with standard deviation $\sigma_j$, related to $r_t / \sqrt{2}$ via a hyperparameter set to 0.5 in all experiments.
Weight reduction is another pre-filtering technique applied to grid-feature interpolations for each multi-sample point, reducing interpolation artifacts. For each point $X_j$ associated with a Gaussian distribution of mean and standard deviation $\sigma_j$, the similarity to the grid at layer $l$ with size $n_l$ is calculated using the error function:
$$ \omega_{j,l} = \text{Erf}\left(\frac{1}{\sqrt{8}} \cdot \sigma_j n_l\right) $$
As $\sigma_j \ll n_l$, $\omega_{j,l}$ approaches 0, and as $\sigma_j \gg n_l$, it approaches 1. The interpolated feature for each $X_j$ is obtained via trilinear interpolation:
$$ f_{j,l} = \text{Trilerp}(n_l \cdot X_j) $$
The weighted feature is then computed as:
$$ f_{j,l_\omega} = \omega_{j,l} f_{j,l} $$
The fused feature for grid layer $l$ is the mean of all weighted features:
$$ f_l = \text{mean}(f_{j,l_\omega}) $$
This process is iterated for each grid layer $l$, and all $f_l$ features are concatenated along the channel dimension to form the comprehensive grid feature. Additionally, $\omega_{j,l}$ is encoded and concatenated to provide supplementary scale modeling.
Geometric constraints are integral to Structure-NeRF, ensuring that the reconstructed models adhere to real-world structural properties. Normal consistency constraint ensures surface normals align with true geometry, quantified by the angle between generated and true normals. The loss is defined as:
$$ L_{NC} = \frac{1}{N} \sum_{i=1}^{N} \left(1 – \sin\left(\arccos\left(\frac{\nabla I_i \cdot \nabla I_{\text{true}}}{\|\nabla I_i\| \cdot \|\nabla I_{\text{true}}\|}\right)\right)\right) $$
where $N$ is the number of points, $I_i$ is the image intensity at point $i$ in the generated scene, $\nabla I_i$ is its gradient, and $I_{\text{true}}$ and $\nabla I_{\text{true}}$ are the corresponding values in the true geometry. The model optimizes by minimizing $L_{NC}$ to maximize consistency.
Plane fitting constraint ensures generated surfaces closely align with planes by comparing predicted and actual plane parameters:
$$ L_{PF} = \frac{1}{N} \sum_{i=1}^{N} \left(1 – \sin\left(\arccos\left(\frac{P_{\text{gen},i} \cdot P_{\text{real},i}}{\|P_{\text{gen},i}\| \cdot \|P_{\text{real},i}\|}\right)\right)\right) $$
Here, $P_{\text{gen},i}$ and $P_{\text{real},i}$ are the predicted and actual plane parameters for point $i$, respectively. Minimizing $L_{PF}$ enhances surface-plane alignment.
Vertical/horizontal constraint ensures line segments adhere to desired orientations, with loss based on angular differences:
$$ L_{VH} = \frac{1}{N} \sum_{i=1}^{N} \left(1 – \sin\left(\arccos\left(\frac{|v_i \cdot u|}{\|v_i\| \cdot \|u\|}\right)\right)\right) $$
where $N$ is the number of line segments, $v_i$ is the direction vector of segment $i$ in the generated scene, and $u$ is the desired direction vector. The overall loss function combines these constraints with the reconstruction loss:
$$ L_{\text{all}} = L(s, w, \hat{s}, \hat{w}) + \lambda_1 L_{NC} + \lambda_2 L_{PF} + \lambda_3 L_{VH} $$
where $L(s, w, \hat{s}, \hat{w})$ is the standard reconstruction loss, and $\lambda_1$, $\lambda_2$, $\lambda_3$ are weighting coefficients.
The Manhattan World Model (MWM) is utilized to analyze point cloud data in Structure-NeRF, extracting core building structures by assuming urban elements align with three orthogonal directions. The distance function for MWM is defined as:
$$ D(x, y, z) = \sum_{i=1}^{N} \min(d_i(x, y, z), t_i) $$
where $D(x, y, z)$ is the minimum distance to the nearest plane in MWM, $N$ is the number of planes, $d_i(x, y, z)$ computes the distance to the $i$-th plane, and $t_i$ is a threshold for point classification. The implementation involves two stages: initial classification of points into categories like walls, edges, and corners, followed by volumetric characterization to ensure coherent segmentation of urban structures. This process highlights component interconnectedness and enables precise dimension estimation. For example, building height is determined as:
$$ \text{Height} = \max(|Z_{\text{top}} – Z_{\text{bottom}}| \cdot C) $$
where $Z_{\text{top}}$ and $Z_{\text{bottom}}$ are the $z$-coordinates of the top and bottom planes or walls, and $C$ is a correction factor converting pixel size to physical dimensions. Similarly, width and length are quantified using Manhattan distance principles.
In experimental evaluations, we collected 98 images of a library building using a DJI Mavic 3 Pro Unmanned Aerial Vehicle, equipped with a 4/3-inch CMOS sensor and 20-megapixel main camera, providing images at 5280 × 3956 resolution. Data acquisition employed tilt photography mode, and experiments were conducted on a system with an NVIDIA RTX 3090 GPU and 64 GB RAM. Image preprocessing involved estimating camera parameters using COLMAP software. Qualitative analysis shows that Structure-NeRF effectively captures 3D structural features, accurately reconstructing main building elements like windows, roofs, and walls from various viewpoints. The model’s consistency across views indicates global coherence with minimal quality loss, attributed to geometric constraints. Visual outputs exhibit high realism and fidelity, with building elements appearing coherent and undistorted.
Quantitative comparisons with mainstream models demonstrate the superiority of Structure-NeRF. The following table summarizes key training metrics, highlighting improvements in efficiency and quality:
| Metric | Structure-NeRF | mip-NeRF | iNGP |
|---|---|---|---|
| Training Time (hours) | 2.26 | 4.55 | 0.84 |
| SSIM | 0.93 | 0.82 | 0.62 |
| PSNR (dB) | 36.23 | 32.45 | 28.65 |
| LPIPS | 0.26 | 0.08 | 0.03 |
Structure-NeRF reduces training time by approximately 50% compared to mip-NeRF, while achieving higher SSIM (13.41% and 50.00% improvements over mip-NeRF and iNGP, respectively) and PSNR (10.45% and 25.10% improvements). The LPIPS score, which measures perceptual similarity, is also significantly higher, indicating better detail replication. These results underscore the model’s suitability for architectural and cluster modeling, where rendering quality and structural accuracy are paramount.
The integration of Unmanned Aerial Vehicle technology, particularly using systems like the JUYE UAV, provides a robust data foundation for Structure-NeRF. The multi-view sensors on these UAVs capture comprehensive imagery that, when processed through our optimized NeRF framework, yield detailed 3D reconstructions. The geometric constraints and MWM further enhance the practicality of the outputs for real-world applications, such as building inspection and urban development. By leveraging advanced sampling and weight reduction techniques, Structure-NeRF addresses the limitations of traditional NeRF methods, offering a scalable solution for complex architectural environments.
In conclusion, the Structure-NeRF model represents a significant advancement in 3D reconstruction for building models using Unmanned Aerial Vehicle multi-view sensors. By combining the strengths of mip-NeRF and iNGP with geometric constraints and MWM, it achieves high precision, efficiency, and realism. The method’s ability to reduce training time while improving image quality metrics makes it a valuable tool for architects, engineers, and researchers. Future work could explore adaptive constraint weighting and integration with real-time processing for dynamic scenes, further expanding the capabilities of Unmanned Aerial Vehicle-based reconstruction systems like the JUYE UAV.