Algolux Eos Embedded Perception vs. State-of-the-Art Models

Robust perception under varying illumination and weather is a key requirement for autonomous driving and driver assistance systems. But in practice, these systems often fail under these conditions. This compromises the large number of dependent components such as the planner and controllers for automated emergency braking or automated driving that rely on the perception system output to work effectively. 

Adverse imaging conditions such as low ambient lighting can result in images with a very low signal-to-noise ratio. Vision systems that do not handle suboptimal imaging conditions can easily fail to perform well in such cases. Furthermore, there is a trade-off between robustness and speed. Deeper and more computationally expensive models have higher capacity but operate at very low frame rates.

In this case study, we compare the Algolux Eos real-time robust perception software, with recent real-time object detection models, namely, YOLOv3, RetinaNet-50, and SSD MobileNet V2. 

Results: We determine that Eos is 38 to 55 mAP (Mean Average Precision) points better in adverse imaging conditions and 8 to 27 mAP points better in clean, noise-free conditions and performs 2.4x faster than YOLOv3.

Dataset

In order to evaluate the robustness of different models in adverse imaging conditions, we captured a large dataset under varying imaging conditions. Using camera calibration information, during training, this dataset is augmented to 100x more RAW data than captured, sampling very broad illumination conditions.

The images were captured using a FLIR Blackfly 23S6C-C camera, which uses a Sony IMX249 sensor with 5.86×5.86 micron pixel pitch, and a high-quality machine-vision lens Fujinon CF12.5HA-1. We use a subset of this dataset for training and a separate unseen testing dataset.

The following classes were considered for both training and evaluation:

1. Car/Van/SUV
2. Bus/Truck/Tram/Train
3. Bicycle/Motorcycle
4. Person (Cyclist/Pedestrian)
5. Traffic Light
6. Traffic Sign

To train the different models in good conditions, we combined our training data with publicly available object detection datasets, namely, COCO, Pascal VOC, and BDD100K, resulting in the total number of training images being around 200K. For the public dataset, only the subset of images containing the relevant labels was considered.

For evaluation, we used a joint public validation set consisting of COCO, Pascal VOC, and BDD100K, and our custom validation set consisting of 18 scenes under different lighting and environmental conditions.

Models Compared in this Report

• Eos is a real-time end-to-end robust vision system that operates on a RAW image stream from one or more cameras and outputs task-specific outputs such as bounding boxes with class labels, segmentation masks, lane markings, etc. Eos is designed to be robust to low-light, noise, and in general, adverse imaging conditions while running at minimum 30 FPS on mid-range mobile GPUs (e.g., 1152 CUDA core discrete GPU used in NVidia Drive PX2) and real-time on embedded processors for automotive and security applications.
• YOLOv3 is a single-stage detection built using a 106-layer deep residual network with multiscale feature extraction and detection heads. It has state-of-the-art performance among the publicly available object detection models.
• RetinaNet-50 is another high-performing publicly available single-stage object detection model that is built on top of a 50-layer residual network with a multiresolution Feature Pyramid Network (FPN) architecture.
• Single Shot Multibox Detector (SSD) with MobileNet V2 backbone is one of the fastest single shot detectors designed to run on mobile devices with very little compute. The key components of it are the linear bottleneck layers and usage of depthwise separable convolution to significantly reduce the number of operations.

Number of operations

In terms of floating-point operations (FLOPS), YOLOv3 operates on images of size 608×608 and performs 140 GFLOPS (billions of floating-point operations). Algolux Eos operates on RAW data in 2.4x fewer operations.

Eos is implemented using Tensorflow and trained using a novel Algolux- developed joint optimization framework for 30 epochs. For YOLOv3, we used the implementation from the authors’ GitHub repository and added the necessary configuration files to train and perform inference with 6 labels. The model was trained from pre-trained darknet53 weights and converged after 64 epochs.

For RetinaNet-50 and SSD MobileNet V2, we use the implementation provided in the Tensorflow Object Detection API and trained the models until convergence.

Results

2. COCO, Pascal VOC, and BDD100K Clean Image Validation Dataset