Intelligent System for Estimation of the Spatial Position of Apples Based on YOLOv3 and Real Sense Depth Camera D415

Despite the great possibilities of modern neural network architectures concerning the problems of object detection and recognition, the output of such models is the local (pixel) coordinates of objects bounding boxes in the image and their predicted classes. However, in several practical tasks, it is necessary to obtain more complete information about the object from the image. In particular, for robotic apple picking, it is necessary to clearly understand where and how much to move the grabber. To determine the real position of the apple relative to the source of image registration, it is proposed to use the Intel Real Sense depth camera and aggregate information from its depth and brightness channels. The apples detection is carried out using the YOLOv3 architecture; then, based on the distance to the object and its localization in the image, the relative distances are calculated for all coordinates. In this case, to determine the coordinates of apples, a transition to a symmetric coordinate system takes place by means of simple linear transformations. Estimating the position in a symmetric coordinate system allows estimating not only the magnitude of the shift but also the location of the object relative to the camera. The proposed approach makes it possible to obtain position estimates with high accuracy. The approximate root mean square error is 7–12 mm, depending on the range and axis. As for precision and recall metrics, the first is 100% and the second is 90%.

Human Position Detection Based on Depth Camera Image Information in Mechanical Safety

The devices used for human position detection in mechanical safety mainly include safety light curtain, safety laser scanner, safety pad, and vision system. However, these devices may be bypassed when used, and human or equipment cannot be distinguished. To solve this problem, a depth camera is proposed as a human position detection device in mechanical safety. The process of human position detection based on depth camera image information is given; it mainly includes image information acquisition, human presence detection, and distance measurement. Meanwhile, a human position detection method based on Intel RealSense depth camera and MobileNet-SSD algorithm is proposed and applied to robot safety protection. The result shows that the image information collected by the depth camera can detect the human position in real time, which can replace the existing mechanical safety human position detection device. At the same time, the depth camera can detect only human but not mobile devices and realize the separation and early warning of people and mobile devices.

Hand Gesture of Recognition Pattern Analysis by Image Treatment Techniques

The goal of this project is to write a program in the C++ language that can recognize motions made by a subject in front of a camera. To do this, in the first place, a sequence of distance images has been obtained using a depth camera. Later, these images are processed through a series of blocks into which the program has been divided; each of them will yield a numerical or logical result, which will be used later by the following blocks. The blocks into which the program has been divided are three; the first detects the subject’s hands, the second detects if there has been movement (and therefore a gesture has been made), and the last detects the type of gesture that has been made accomplished. On the other hand, it intends to present to the reader three unique techniques for acquiring 3D images: stereovision, structured light, and flight time, in addition to exposing some of the most used techniques in image processing, such as morphology and segmentation.

ALGORITHMS FOR IMPROVING SPEED AND ACCURACY OF AUTOMATED 3D RECONSTRUCTION WITH A DEPTH CAMERA MOUNTED ON AN INDUSTRIAL ROBOT

3D reconstruction technology is used in a wide variety of applications. Currently, automatically creating accurate pointclouds for large parts requires expensive hardware. We are interested in using low-cost depth cameras mounted on commonly available industrial robots to create accurate pointclouds for large parts automatically. Manufacturing applications require fast cycle times. Therefore, we are interested in speeding up the 3D reconstruction process. We present algorithmic advances in 3D reconstruction that achieve a sub-millimeter accuracy using a low-cost depth camera. Our system can be used to determine a pointcloud model of large and complex parts. Advances in camera calibration, cycle time reduction for pointcloud capturing, and uncertainty estimation are made in this work. We continuously capture pointclouds at an optimal camera location with respect to part distance during robot motion execution. The redundancy in pointclouds achieved by the moving camera significantly reduces errors in measurements without increasing cycle time. Our system produces sub-millimeter accuracy.

COVID surveillance robot: Monitoring social distancing constraints in indoor scenarios

Observing social/physical distancing norms between humans has become an indispensable precaution to slow down the transmission of COVID-19. We present a novel method to automatically detect pairs of humans in a crowded scenario who are not maintaining social distancing, i.e. about 2 meters of space between them using an autonomous mobile robot and existing CCTV (Closed-Circuit TeleVision) cameras. The robot is equipped with commodity sensors, namely an RGB-D (Red Green Blue—Depth) camera and a 2-D lidar to detect social distancing breaches within their sensing range and navigate towards the location of the breach. Moreover, it discreetly alerts the relevant people to move apart by using a mounted display. In addition, we also equip the robot with a thermal camera that transmits thermal images to security/healthcare personnel who monitors COVID symptoms such as a fever. In indoor scenarios, we integrate the mobile robot setup with a static wall-mounted CCTV camera to further improve the number of social distancing breaches detected, accurately pursuing walking groups of people etc. We highlight the performance benefits of our robot + CCTV approach in different static and dynamic indoor scenarios.