639-wpg

CS 639: Road Lane Detection

Anuj Amin (aamin6@wisc.edu) & Prasun Guragain (guragain@wisc.edu)

Problem

The problem statement we are trying to solve is detecting traffic lanes in real-time. With cars being driven every day and engineers trying to make cars safer, many companies are turning to machine learning and self-driving in order to help solve this problem.

This is an essential problem in the real world as there are an increasing number of cars with built-in technologies to do this. For example, Tesla has self-driving cars, and many other big-name automotive companies have technologies such as lane assist, ensuring that the driver is driving in the correct lane and direction. However, this technology is still very new and has many problems. For example, how can the computer detect a lane in poor weather conditions or a faded-out lane? What if there are no street lights at night that prevent the computer from detecting any lanes? What about the effects of lane-like objects on the road such as railways and walkways? These variables affect every driver and can affect traffic lane detection as well. While existing algorithms can have a very high detection rate, generally most of these tests are done using favorable conditions such as a clear road and good lighting (Huang et. al, 2021).

Importance of Lane Detection

Self driving cars and lane detection have the potential to make the world safer. For example, according to the article “Drowsy driving statistics and facts 2022” by Elizabeth Rivelli, there have been over 10,000 fatal crashes due to drowsy or careless driving, including failure to keep in the proper lane, being distracted (by phone, talking eating, etc), and swerving due to weather, objects, etc. These serious fatalities could be easily avoided by equipping programs in modern cars with good traffic lane detections that can account for any, or at least most, situations (Rivelli, 2022).

Current State of Art

The current state of the art of road lane detection is mainly in self-driving cars. The paper "Practical Self-Driving Cars: Survey of the State-of-the-Art" talks about how numerous research studies are being conducted at various stages of Autonomous Driving (AD) systems, but many people rely on Computer Vision techniques similar to our simple one in this project. The AD Pipeline is filled with many important stages, including sensors, planning, control, localization, and more. However, vision is the most significant part of the pipeline. For lane vision, modern AD systems not only focus on detection but also segmentation, classification, and stereo vision, for efficient self-driving system. Another paper "Real-time lane detection for autonomous vehicles" mentions that they have been working on creating high-level road lane detection to account for any conditions, starting with a simple approach like ours, and adding more layered techniques with deep learning (like Convolutional Neural Networks) to create successful algorithms.

Development Steps and Timeline of Project

October 30th: Find dataset, and plan project's technical approach and implementation
November 5th: Begin implementation of script to detect lanes in high resolution static images
November 15th: Testing the accuracy of annotated images from our implementation by manually looking through the images to assure that the lanes are detected effectively, then further refining the process.
November 30th: Implement script to detect lanes in a video streams.
December 5th: Make improvements to implementation and clean up code, and well as more testing.
December 10th: Finalize project and website, and create and record presentation.

Dataset

In order to better grasp the underlying image processing techniques used in lane detection, we opted to use clean and well lit images and video from the Lane Line Detection. Furthermore, in order to further evaluate the strengths and drawbacks of our model, we then decided to use some images from the UNet Lanes v3 dataset in Kaggle. This is a fairly large and robust dataset containing road lane images from simulators to varying real world conditions.

Proposed Solution

Our proposed solution is an algorithm that, given static images or even a video of a car driving on the road, will detect and highlight lanes in real-time. This will done through the image processing techniques learned from class. This approach forms the foundation of many of the state of the art advanced techniques that are currently being implemented in industry and academia.

Method

For this project we used python and openCV in order to process an image i.e. a frame.

Applying Canny Edge Detection

The intuitive step for lane detection is highlighting where potential lanes appear and masking the rest of the image. One can simply choose to select a color that is at least as bright as a lane and ignore the rest of the image. However, what happens if lane colors differ i.e. the left lane is yellow and the right lane is white? Clearly masking in itself will fail here. Therefore, it is critical that we use Canny Edge detection.

However, before we can do that we must turn the image into grayscale in order detect changes in color i.e. detect an edge. Then we follow the standard procedure of applying gaussian smoothing to eliminate any potential noise. We then applied canny edge detection with a proper lower and upper bound value to get the perfect amount of edge detection without having too many false positives.

Basic Color Selection Input

Basic Color Selection Output

Color Selection with Varying Lane Colors Input

Color Selection with Varying Lane Colors Output (Problem!)

Color Selection via Canny Input

Color Selection via Canny Output

Applying Hough Transform

Now that we have applied Canny Edge detection, we need to use the hough transform in order to isolate the lanes. In order to do so, we create a mask over the region where the lanes would lie. We then proceed to apply the hough transform over this region and display the output.

Input 1

Output 1

Input 2

Output 2

Input 3

Output 3

Streamlining the Process

Now that we are able to detect lanes on a static image, we need to detect lanes on a video stream. In order to do so, we just have to read in the image frame by frame and put each frame into our pipeline. We then join the new images with edge detection together and get a video stream of detected lanes.

Problems with Current Approach

While we can clearly see above that the current approach is fairly accurate in detecting lanes. It requires a lot of manual fine tuning and it breaks when we look at different edge cases. We are having trouble detecting lanes when driving alongside a curve as shown above. Furthermore, the model clearly cannot detect lanes correctly with the current pipeline when we extrapolate to nighttime detection or even change the angle. We would have to change the parameters of the canny edge detection and the size of the mask in order to properly detect lanes with these images. Even if we do this there is no guarantee that this will work.

Failed Input 1

Failed Output 1

Failed Input 2

Failed Output 2

Further Exploration

It is evident how complex lane detection truly is. In practice data has to be heavily preprocessed before it can be used. Many researchers have to spend countless hours labelling the lanes and then applying some machine learning model on a training dataset to get better predictions on the test dataset. Therefore, in order to improve our model, we would have to look into machine learning, specifically deep learning. Judging by the size of the original dataset we would need to look into using some form of a neural network like a CNN. This is a very time and computationally expensive task that would require further exploration. Additional direction for this project could be to do manual image processing by finetuning the pipeline to work with night time images and other extreme conditions to better fine tune the algorithm.