Plane Fitting and Depth Variance Based Upsampling for Noisy Depth Map from 3D-ToF Cameras in Real-time

Depth maps obtained through sensors are often unsatisfactory because of their low-resolution and noise interference. In this paper, we propose a real-time depth map enhancement system based on a residual network which uses dual channels to process depth maps and intensity maps respectively and cancels the preprocessing process, and the algorithm proposed can achieve real-time processing speed at more than 30 fps. Furthermore, the FPGA design and implementation for depth sensing is also introduced. In this FPGA design, intensity image and depth image are captured by the dual-camera synchronous acquisition system as the input of neural network. Experiments on various depth map restoration shows our algorithms has better performance than existing LRMC, DE-CNN and DDTF algorithms on standard datasets and has a better depth map super-resolution, and our FPGA completed the test of the system to ensure that the data throughput of the USB 3.0 interface of the acquisition system is stable at 226 Mbps, and support dual-camera to work at full speed, that is, 54 fps@ (1280 × 960 + 328 × 248 × 3).

Download Full-text

Stochastic-Depth Ambient Occlusion

Proceedings of the ACM on Computer Graphics and Interactive Techniques ◽

10.1145/3451268 ◽

2021 ◽

Vol 4 (1) ◽

pp. 1-15

Author(s):

Jop Vermeer ◽

Leonardo Scandolo ◽

Elmar Eisemann

Keyword(s):

Real Time ◽

High Performance ◽

Depth Map ◽

Visual Quality ◽

Ambient Light ◽

Missing Information ◽

Time Performance ◽

Ambient Occlusion ◽

Depth Buffer ◽

Screen Space

Ambient occlusion (AO) is a popular rendering technique that enhances depth perception and realism by darkening locations that are less exposed to ambient light (e.g., corners and creases). In real-time applications, screen-space variants, relying on the depth buffer, are used due to their high performance and good visual quality. However, these only take visible surfaces into account, resulting in inconsistencies, especially during motion. Stochastic-Depth Ambient Occlusion is a novel AO algorithm that accounts for occluded geometry by relying on a stochastic depth map, capturing multiple scene layers per pixel at random. Hereby, we efficiently gather missing information in order to improve upon the accuracy and spatial stability of conventional screen-space approximations, while maintaining real-time performance. Our approach integrates well into existing rendering pipelines and improves the robustness of many different AO techniques, including multi-view solutions.

Download Full-text

Robust hand gesture recognition using multiple shape-oriented visual cues

EURASIP Journal on Image and Video Processing ◽

10.1186/s13640-021-00567-1 ◽

2021 ◽

Vol 2021 (1) ◽

Author(s):

Samy Bakheet ◽

Ayoub Al-Hamadi

Keyword(s):

Real Time ◽

Gesture Recognition ◽

Pose Estimation ◽

Depth Map ◽

Hand Gesture Recognition ◽

Support Vector ◽

Hand Gesture ◽

Hand Pose Estimation ◽

Time Operation ◽

Hand Pose

AbstractRobust vision-based hand pose estimation is highly sought but still remains a challenging task, due to its inherent difficulty partially caused by self-occlusion among hand fingers. In this paper, an innovative framework for real-time static hand gesture recognition is introduced, based on an optimized shape representation build from multiple shape cues. The framework incorporates a specific module for hand pose estimation based on depth map data, where the hand silhouette is first extracted from the extremely detailed and accurate depth map captured by a time-of-flight (ToF) depth sensor. A hybrid multi-modal descriptor that integrates multiple affine-invariant boundary-based and region-based features is created from the hand silhouette to obtain a reliable and representative description of individual gestures. Finally, an ensemble of one-vs.-all support vector machines (SVMs) is independently trained on each of these learned feature representations to perform gesture classification. When evaluated on a publicly available dataset incorporating a relatively large and diverse collection of egocentric hand gestures, the approach yields encouraging results that agree very favorably with those reported in the literature, while maintaining real-time operation.

Download Full-text

Joint upsampling and noise reduction for real-time depth map enhancement

10.1117/12.2039190 ◽

2014 ◽

Cited By ~ 1

Author(s):

Kazuki Matsumoto ◽

Chiyoung Song ◽

Francois de Sorbier ◽

Hideo Saito

Keyword(s):

Noise Reduction ◽

Real Time ◽

Depth Map

Download Full-text

Real-Time Vehicle Detection Framework Based on the Fusion of LiDAR and Camera

Electronics ◽

10.3390/electronics9030451 ◽

2020 ◽

Vol 9 (3) ◽

pp. 451 ◽

Cited By ~ 3

Author(s):

Limin Guan ◽

Yi Chen ◽

Guiping Wang ◽

Xu Lei

Keyword(s):

Real Time ◽

Color Image ◽

Evidence Theory ◽

Depth Map ◽

Vehicle Detection ◽

Detection Accuracy ◽

Distance Information ◽

Detection Model ◽

Detection Mode ◽

Single Sensor

Vehicle detection is essential for driverless systems. However, the current single sensor detection mode is no longer sufficient in complex and changing traffic environments. Therefore, this paper combines camera and light detection and ranging (LiDAR) to build a vehicle-detection framework that has the characteristics of multi adaptability, high real-time capacity, and robustness. First, a multi-adaptive high-precision depth-completion method was proposed to convert the 2D LiDAR sparse depth map into a dense depth map, so that the two sensors are aligned with each other at the data level. Then, the You Only Look Once Version 3 (YOLOv3) real-time object detection model was used to detect the color image and the dense depth map. Finally, a decision-level fusion method based on bounding box fusion and improved Dempster–Shafer (D–S) evidence theory was proposed to merge the two results of the previous step and obtain the final vehicle position and distance information, which not only improves the detection accuracy but also improves the robustness of the whole framework. We evaluated our method using the KITTI dataset and the Waymo Open Dataset, and the results show the effectiveness of the proposed depth completion method and multi-sensor fusion strategy.

Download Full-text