Image Processing and Pattern Recognition Group > Research > Autonomus Driving > DESBOR

DESBOR

Dense Stereo-Based Object Recognition system for automatic cruise control (ACC) in urban traffic environments

Project founded by Volkswagen AG, Germany (2005-2007)

Introduction

Automotive companies are investing a lost of interest in intelligent systems which can assist the driving process. Radar and laser sensors based applications are dedicated for precise pose measurements but they cannot give a complete description of the driving environment due to their nature. Vision sensors can compensate the information lack of their measurements. Stereovision has been proved the only method able give reliable results both for measurements and description of the driving environment.

Objectives

Development of a realtime densestereo camera based system able to detect, classify and track relevant objects, detect the lane and estimate the road geometry, detect and recognize traffic signs painted on the roads. Based on the detection parameters the system is able to perform specific urban traffic ACC tasks: stop[[[&]]]go (follow- to-stop and stop-inhibit), vehicle lateral control, set maximum ACC velocity etc.

Marginal conditions:
• detection range: 0 to 50 m
• Field of view: 72 degrees
• Maximum traveling speed: 80 km/h
• Processing rate: 15 frames/second

Methodology

The acquisition system is composed from two digital b/w cameras integrated in a rigid stereo head. The camera parameters are calibrated using a dedicated methodology. Dense stereo reconstruction is performed by a dedicated hardware board (DeepSea by TYZX) or by a software algorithm. Features belonging to lane delimiters are classified and their 3D coordinates are used in the lane detection process. The 3D points above the detected road surface are grouped into objects, taking into account the vicinity criteria, density variation with the distance, and 2D image information such as similar texture and connecting edges. Tracking using Kalman filtering is used both for lane and objects parameters improving. Real time implementation using VisualC framework and MMX instructions for code optimization was used. Detection results can be further fused with information provided by other sensors (radar and laser scanner).

Achievements

Detection results of the DESBOR application are presented in fig. 1:

Fig. 1. DESBOR output.

Dense stereovision

Realtime dense stereo reconstruction is performed using a dedicated board (DeepSea by TYZX) (fig. 2). A software algorithm was also developed for offline validation purposes (fig. 3).

Detection of elevation maps
The elevation map is a mid-level scene description, derived from the raw dense stereo information through a process of filtering and cel-llevel tracking. The resulting map cells are then classified into drivable/non drivable areas, providing a more refined basis for the higher level algorithms. The results of this processing module is used in the higher level algorithms, such as obstacle and lane detection, but they can also provide a crude environment estimation for lateral and longitudinal control in the absence of lane markings and structured (cuboidal) objects.

3D lane detection
An original and robust solution for lane detection using 3D road features, a model driven approach and Kalman filter tracking was developed (fig.2). It uses non-flat road assumption, and works even in extreme conditions (absence of lane markings - fig. 3). Side lanes and driving area (with delimiters) are also detected (fig. 2). Detected lane parameters are: vertical and horizontal curvature, lane width, and lateral position of the ego-car inside the current lane.

Object detection and tracking
The equation of the road surface is used for obstacle/road point separation and therefore object detection works even on non-flat road scenarios. The overrode 3D points are grouped into objects taking into account the vicinity criteria. The 3D density variation with the distance is taken into account. 2D image information such as similar texture and connecting edges are refining the grouping process. The object's position is tracked over successive frames using a mathematical motion model and Kalman filtering. Detected objects are described in terms of 3D position, size and speed (fig. 3 and 4).

Concluding remarks

We have developed a system based on stereovision able to perform in real time many tasks needed for driving environment description and measurements. The system functions are integrated into a dedicated stereovision framework, which can be easily extended with other capabilities (e.g. image analyzes tasks) or to build a specific application for an active security or driving assistance system for automotive industry.

Our services

The obtained expertise of the research team has reached the state-of-the-art level in some specific fields of the computer vision usable in robot or automotive applications: camera calibration, high-resolution stereo reconstruction, stereo measurements, object detection and tracking, 3D lane detection, stereo image acquisition systems (control, programming, design).

Recent publications

S. Nedevschi, R. Danescu, T. Marita, F. Oniga, C. Pocol, S. Sobol, C. Tomiuc, C. Vancea, M. M. Meinecke, T. Graf, T. B. To, M. A. Obojski, "A Sensor for Urban Driving Assistance Systems Based on Dense Stereovision", Proceedings of Intelligent Vehicles 2007, 13-15 June, 2007, Istambul, pp.278-286

R. Danescu, S. Sobol, S. Nedevschi, T. Graf, "Stereovision-Based Side Lane and Guardrail Detection", Proceedings of the IEEE Intelligent Transportation Systems Conference (ITSC 2006), Toronto, Canada, September 17-20, 2006, pp. 1156-1161.

S. Nedevschi, T. Marita, R. Danescu, F. Oniga, C. Pocol, S. Sobol, C. Tomiuc, C. Vancea, S. Bota, "Stereovision Sensor for Driving Assistance", in Proceedings of IEEE International Conference on Intelligent Computer Communication and Processing, 1-2 Sept. 2006, Cluj-Napoca, Romania, pp. 105-112.

T. Marita, F. Oniga, S. Nedevschi, T. Graf, R. Schmidt, "Camera Calibration Method for Far Range Stereovision Sensors Used in Vehicles", Proceedings of IEEE Intelligent Vehicles Symposium, (IV2006) , June 13-15, 2006, Tokyo, Japan, pp. 356-363.

S. Nedevschi, F. Oniga, R. Danescu, T. Graf, R. Schmidt, "Increased Accuracy Stereo Approach for 3D Lane Detection", Proceedings of IEEE Intelligent Vehicles Symposium, (IV2006) , June 13-15, 2006, Tokyo, Japan, pp. 42-49.

Stefan Sobol, Sergiu Nedevschi, "Driving area detection", Poster volume of IEEE 2nd International Conference on Intelligent Computer Communication and Processing, 1-2 September, Cluj-Napoca, Romania, 2006, pp. 89-94

Fig. 2. Hardware dense stereo-reconstruction

Fig. 3. Software dense stereo-reconstruction

Fig. 4. 3D elevation map

Fig.5. Lane detection

Fig. 6 Crossing signs detection