Environment Perception

Road free space detection is one of the very crucial component that plays an important role in the decision making process of an autonomous vehicle. It helps the vehicle in detecting lanes and free space on road that is essential to move vehicle correctly on the street without any collisions. Depending on the type of road we are driving on, it might be of importance to distinguish between multiple lanes that have been detected by our vehicle. For example, on highway scenarios the vehicle is allowed to change lanes, whereas in case of city scenarios there exists the risk of invading lanes of the incoming traffic. In such cases we can instruct our vehicle not to change lane and stay in its current lane. In this work, we are working with raw camera images, processing them with the help of semantic segmentation algorithms that performs pixel level classification by classifying each pixel as ego lane, other lane or background. The developed algorithm is capable of running at a speed of 30 FPS, hence providing the results in real-time.

The main reasons behind road accidents are human factors like fatigue or failure to adjust to changing road conditions. Since humans are much more prone to error than computers, another significant benefit that autonomous cars can bring to authorities is reducing road accidents. To achieve this, autonomous cars in addition to driving on their own also need to correctly identify traffic signs and follow traffic rules.

In this project, we have developed a traffic sign detection system based on deep convolutional neural networks. The model has been trained to detect and classify 43 traffic signs including speed signs, attention signs, warning signs, and priority signs, etc. The developed model was then tested on new data collected by the cameras of our test vehicle as shown in this video. The model is based on a single stage object detection architecture with main focus on fast and real-time processing of image data.

Detecting objects on road is another very important prerequisite for autonomous driving, as it helps the car to see what kind of objects are present near it, for example, vehicles, persons, cyclists, traffic signs, etc., and what is their location. This information plays a critical role for the controller to account for the obstacles when considering possible future trajectories.

In this work, we are using convolutional neural networks that takes raw camera images as input and provides us with two types of information, the 2D bounding boxes for the detected objects and their respective class labels. As these algorithms are supposed to run in real-time, we implemented a single stage object detection algorithm that detects object in the input image using an end-to-end architecture in a single shot. The developed algorithm is capable of running at a speed of 30 FPS, hence providing the results in real-time.

In order to perform dynamic path planning it is not sufficient to only detect nearby objects, but we also need to track them and estimate their trajectories over time. This tracking information can be used to predict their future locations which is very crucial information for obstacle avoidance, path planning and intent recognition. Object tracking is very similar to object detection and involves the accurate identification and localization of dynamic objects in the environment surrounding the vehicle. In addition to detect bounding boxes for the objects, it also assigns them some unique IDs and the goal is to preserve these IDs as long as they don't leave the vehicle's environment.

In this work, we have developed an end-to-end simultaneous object detection and tracking framework that detects objects in the current frame and then performs an offset based association between newly detected objects and detections in the previous frame to generate the track IDs.

The sensor setup of our test vehicle consists of camera, lidar, radar, and wide-angle cameras, etc. Each of these sensors has its own limitations and cannot always provide the complete information about the vehicle surroundings, for example, cameras are affected by bad lighting conditions, LiDARs do not perform good in environmental conditions with bad reflectivity, radars can operate in more varied conditions but are not as accurate as lidars, etc. So, we are combining the input from various sensors to build our environmental model that is not only robust and reliable but also ensures safety for the automated driving functions.

In this work, we have developed an object tracking system based on Kalman filter that tracks objects like vehicles, pedestrians, cyclists, etc. by taking detection results from our 3D object detection algorithm. The 3D detection algorithm is based on a decision level fusion of our lidar and camera detection networks.