Studies on Three-Dimensional (3D) Accuracy Optimization and Repeatability of UAV in Complex Pit-Rim Landforms As Assisted by Oblique Imaging and RTK Positioning

Unmanned Aerial Vehicles (UAVs) are a novel technology for landform investigations, monitoring, as well as evolution analyses of long−term repeated observation. However, impacted by the sophisticated topographic environment, fluctuating terrain and incomplete field observations, significant differences have been found between 3D measurement accuracy and the Digital Surface Model (DSM). In this study, the DJI Phantom 4 RTK UAV was adopted to capture images of complex pit-rim landforms with significant elevation undulations. A repeated observation data acquisition scheme was proposed for a small amount of oblique-view imaging, while an ortho-view observation was conducted. Subsequently, the 3D scenes and DSMs were formed by employing Structure from Motion (SfM) and Multi-View Stereo (MVS) algorithms. Moreover, a comparison and 3D measurement accuracy analysis were conducted based on the internal and external precision by exploiting checkpoint and DSM of Difference (DoD) error analysis methods. As indicated by the results, the 3D scene plane for two imaging types could reach an accuracy of centimeters, whereas the elevation accuracy of the orthophoto dataset alone could only reach the decimeters (0.3049 m). However, only 6.30% of the total image number of oblique images was required to improve the elevation accuracy by one order of magnitude (0.0942 m). (2) An insignificant variation in internal accuracy was reported in oblique imaging-assisted datasets. In particular, SfM-MVS technology exhibited high reproducibility for repeated observations. By changing the number and position of oblique images, the external precision was able to increase effectively, the elevation error distribution was improved to become more concentrated and stable. Accordingly, a repeated observation method only including a few oblique images has been proposed and demonstrated in this study, which could optimize the elevation and improve the accuracy. The research results could provide practical and effective technology reference strategies for geomorphological surveys and repeated observation analyses in sophisticated mountain environments.

Moving Car Recognition and Removal for 3D Urban Modelling Using Oblique Images

With the progress of photogrammetry and computer vision technology, three-dimensional (3D) reconstruction using aerial oblique images has been widely applied in urban modelling and smart city applications. However, state-of-the-art image-based automatic 3D reconstruction methods cannot effectively handle the unavoidable geometric deformation and incorrect texture mapping problems caused by moving cars in a city. This paper proposes a method to address this situation and prevent the influence of moving cars on 3D modelling by recognizing moving cars and combining the recognition results with a photogrammetric 3D modelling procedure. Through car detection using a deep learning method and multiview geometry constraints, we can analyse the state of a car’s movement and apply a proper preprocessing method to the geometrically model generation and texture mapping steps of 3D reconstruction pipelines. First, we apply the traditional Mask R-CNN object detection method to detect cars from oblique images. Then, a detected car and its corresponding image patch calculated by the geometry constraints in the other view images are used to identify the moving state of the car. Finally, the geometry and texture information corresponding to the moving car will be processed according to its moving state. Experiments on three different urban datasets demonstrate that the proposed method is effective in recognizing and removing moving cars and can repair the geometric deformation and error texture mapping problems caused by moving cars. In addition, the methods proposed in this paper can be applied to eliminate other moving objects in 3D modelling applications.

Improvement of Workflow for Topographic Surveys in Long Highwalls of Open Pit Mines with an Unmanned Aerial Vehicle and Structure from Motion

Conducting topographic surveys in active mines is challenging due ongoing operations and hazards, particularly in highwalls subject to constant and active mass movements (rock and earth falls, slides and flows). These vertical and long surfaces are the core of most mines, as the mineral feeding mining production originates there. They often lack easy and safe access paths. This framework highlights the importance of accomplishing non-contact high-accuracy and detailed topographies to detect instabilities prior to their occurrence. We have conducted drone flights in search of the best settings in terms of altitude mode and camera angle, to produce digital representation of topographies using Structure from Motion. Identification of discontinuities was evaluated, as they are a reliable indicator of potential failure areas. Natural shapes were used as control/check points and were surveyed using a robotic total station with a coaxial camera. The study was conducted in an active kaolin mine near the Alto Tajo Natural Park of East-Central Spain. Here the 140 m highwall is formed by layers of limestone, marls and sands. We demonstrate that for this vertical landscape, a facade drone flight mode combined with a nadir camera angle, and automatically programmed with a computer-based mission planning software, provides the most accurate and detailed topographies, in the shortest time and with increased flight safety. Contrary to previous reports, adding oblique images does not improve accuracy for this configuration. Moreover, neither extra sets of images nor an expert pilot are required. These topographies allowed the detection of 93.5% more discontinuities than the Above Mean Sea Level surveys, the common approach used in mining areas. Our findings improve the present SfM-UAV survey workflows in long highwalls. The versatile topographies are useful for the management and stabilization of highwalls during phases of operation, as well closure-reclamation.

3D MODELLING USING AERIAL OBLIQUE IMAGES WITH CLOSE RANGE UAV BASED DATA FOR SINGLE OBJECTS

The request for 3D Data for the use of 3D city-models is increasing rapidly. More and more tools are able to deal with data of several sensors, out of video-streams, oblique camera setups with huge overlaps as well as terrestrial data. To achieve high accuracy of the data and a fast processing pipeline, a smart workflow has to be defined and established. However, mixed data sources are still a challenge especially if different sensors with an extremely different GSD are used. This abstracts demonstrates such a workflow, the processing pipeline and the challenges in a mixed data processing. Special calibration and co-calibrating procedures have been applied to get model in model solution managed to solve the dual task of 3D city mapping and cultural heritage conservation. Especially the sensor setup directly influences the geometric accuracy of the product. To do missions for 2–5 cm GSD, metric systems are indispensable while for non-metric applications also simple and cheaper sensors do their job. Besides the different data-sources and sensors, the way of capturing and the related projection is a critical issue. While the classical oblique imaging is a standardized airborne application, captures with UAVs are more like close range photogrammetry on the facades. The combination requests specific pre-processing and definition and transformation steps.

MAGO APPROACH FOR SEMANTIC SEGMENTATION: THE CASE STUDY OF UAVID BENCHMARK DATASET

The present work is focused on a semantic segmentation strategy implemented in the workflow of the tool MAGO (standing for “Adaptive Mesh for Orthophoto Generation”), considering the contribution of the 3D geometry and the colour information, both deriving from the point cloud of the scene. Moreover, the 2D source imagery, previously used to obtain the photogrammetric point cloud, is employed even to enhance the procedure with the recognition of moving objects, comparing the evolution of epochs.The analysed context is an urban scene, deriving from the UAVid dataset proposed for the ISPRS benchmark. In particular, the so-called “seq18”, a set of high-resolution oblique images taken by UAV (Unmanned Aerial Vehicle), has been used to test the semantic segmentation. The workflow includes the production of two Digital Surface Models (DSMs), containing the geometric and radiometric information, respectively, and their processing by means of the Harris corner detector, allowing the understanding of the image variability. Then, starting from the source geometry and colour information and combining them with their variability mapping, a preliminary classification is performed. Further criteria allow the segmentation of the humans and cars present in the scene. In particular, static objects are identified according to the content of the neighbour pixels in a certain kernel, while the evolution in time of moving elements is recognized by means of the comparison of the projected images belonging to the different epochs. The presented preliminary achievements show some criticalities that require further attention and improvement. In particular, the strategy could be enriched getting more information from the source 2D images, which at the moment are directly used only for the comparison of consecutive epochs.

BIDIRECTIONAL MULTI-SCALE ATTENTION NETWORKS FOR SEMANTIC SEGMENTATION OF OBLIQUE UAV IMAGERY

Semantic segmentation for aerial platforms has been one of the fundamental scene understanding task for the earth observation. Most of the semantic segmentation research focused on scenes captured in nadir view, in which objects have relatively smaller scale variation compared with scenes captured in oblique view. The huge scale variation of objects in oblique images limits the performance of deep neural networks (DNN) that process images in a single scale fashion. In order to tackle the scale variation issue, in this paper, we propose the novel bidirectional multi-scale attention networks, which fuse features from multiple scales bidirectionally for more adaptive and effective feature extraction. The experiments are conducted on the UAVid2020 dataset and have shown the effectiveness of our method. Our model achieved the state-of-the-art (SOTA) result with a mean intersection over union (mIoU) score of 70.80%.

Seismic Damage Semantics on Post-Earthquake LOD3 Building Models Generated by UAS

In a post-earthquake scenario, the semantic enrichment of 3D building models with seismic damage is crucial from the perspective of disaster management. This paper aims to present the methodology and the results for the Level of Detail 3 (LOD3) building modelling (after an earthquake) with the enrichment of the semantics of the seismic damage based on the European Macroseismic Scale (EMS-98). The study area is the Vrisa traditional settlement on the island of Lesvos, Greece, which was affected by a devastating earthquake of Mw = 6.3 on 12 June 2017. The applied methodology consists of the following steps: (a) unmanned aircraft systems (UAS) nadir and oblique images are acquired and photogrammetrically processed for 3D point cloud generation, (b) 3D building models are created based on 3D point clouds and (c) 3D building models are transformed into a LOD3 City Geography Markup Language (CityGML) standard with enriched semantics of the related seismic damage of every part of the building (walls, roof, etc.). The results show that in following this methodology, CityGML LOD3 models can be generated and enriched with buildings’ seismic damage. These models can assist in the decision-making process during the recovery phase of a settlement as well as be the basis for its monitoring over time. Finally, these models can contribute to the estimation of the reconstruction cost of the buildings.