[Case Study] How an Automotive Tier 1 Supplier Used Atlas to Improve Computer Vision Accuracy by up to 48% mAP within Days

The Atlas workflow: 

A small dataset of field RAW images is captured with the target camera module and annotated. It must contain a distribution of images that exercise the operational range of the camera and use case scenarios. The distribution covers the exposure times and gains the sensor is capable of in the expected low to bright light scenes, including High Dynamic Range (HDR), and low to high contrast. It also needs to have good coverage of the target detector classes and object sizes. If IQ metrics are to be evaluated or optimized, a small set of RAW lab chart images would also be captured. 

Since the objective is to optimize an ISP and not train a neural network, this dataset can be orders of magnitude smaller than neural network training datasets, i.e only several hundreds or thousand images rather than hundreds of thousands or millions of annotated images, which is what you’d typically use for training a network.

Each frame is tagged with exposure and gain information from the sensor module so the sensor state is known for each image. That metadata is also used in the optimization process so the ISP parameter modulation functions can be built to control the image quality for the particular capture conditions.

This small RAW image dataset is run through the ISP and into the detector model. Computer vision accuracy metrics, such as average precision and recall of the detected object classes, are then evaluated for each trial ISP configuration. 

Visual image quality KPIs can be directly measured on the image, but for computer vision-only systems, how visually pleasing the images look is unimportant so long as the final ISP parameter configuration enables the vision model to produce more accurate detections. 

This novel optimization framework is applied to find the best ISP parameters to use for a specific pre-trained deep learned or classical computer vision model by minimizing the loss function of those accuracy KPIs.

These ISP pipelines are not differentiable and do not necessarily optimize well with typical gradient-based approaches. There can be null spaces in registers or non-linear stepwise behaviors as an ISP parameter crosses a threshold that is internal to an algorithm, so the optimizer needs to be robust and be able to split up and explore the solution space to escape the massive number of local minima.

Atlas projects require some initial optimizations to ensure proper ISP configuration and convergence, and then a final optimization to maximize results. This typically takes anywhere from 4000 to 8000 trials (or iterations) over a few days to explore the parameter space and converge on an optimized result, depending on factors such as number of parameters and KPI objectives that are being optimized.

This optimization process can also be applied to automate and significantly accelerate visual image quality tuning or optimize for both computer vision and visual image quality goals (Figure 4).

The ISP outputs 14-bit YUV data, which requires 10 bits of range compression from the ISP.

The ISP operation was also subject to some additional constraints. As the image stream was also being used for other computer vision applications, the output image had to be linearizable, which means that transforms have to be global and invertible. 

Due to the Tier 1’s OEM customer requirements, local tone mapping could not be used. Furthermore, unlike most camera configurations, the ISP had to run in a steady state for system stability, meaning adaptive gains could not be set automatically based on image statistics. 

As such, the ISP needed to be optimized for a static configuration for all images that cover the full dynamic range. 

The optimal tone mapping or range compression to apply in the pipeline for the detector was unknown. So, the Atlas optimization process had to determine the best transform to use for mapping the input data range to the most useful range for the detector model. 

A YOLOv4 object detector that had been trained with the COCO dataset was used and only the car and pedestrian detections were considered for this application.

The ISP was parameterized in such a way that ensured the output image met the system constraints while being as flexible as possible with regard to the image processing functions. 

A specific group of blocks in the ISP and the parameters that control the way the algorithms manipulate the images were selected. 

6 lookup tables and 27 scalar parameters were validated and co-optimized through the ISP. These controlled range compression and expansion functions, color accuracy & saturation, spatial filtering, demosaicing, denoising, edge enhancement, sharpening, and contrast (Figure 5).

Some of the compression and expansion functions were optimized using separate optimization loops that used test patterns exercising the full signal range.  

This was done using a measure of quantization error introduced in the ISP to remove biases caused by the luminance distribution of the objects in the optimization set. This enabled analysis of any precision bottlenecks to ensure any data coming into the ISP was also present in the output.

Per the program requirements, Atlas was configured to optimize computer vision accuracy with a single objective loss function, which was mean average precision (mAP).  

30 instances of a bit accurate software model of the ISP were run in parallel and consistently reached convergence after about 1000 ISP configuration trials over a runtime of about 36 hours (Figure 6). 

The speed of each trial was constrained by the performance of the ISP model itself and processing time for the 5-megapixel images. 

Each trial represents a unique ISP configuration, and the configuration settings and image results for each trial were stored by Atlas for later reference. 

The spikes and drops in accuracy seen in Figure 6 as the optimizer is moving toward convergence come from a continued intelligent sampling of parameters. This allows the optimizer to avoid getting trapped in the many local minima in the ISP’s parameter space.

Specific lab charts and test patterns were included in the optimization dataset for each trialed configuration to measure:

• Color Accuracy / Saturation
• Tonal Response
• SNR
• MTF
• Overshoot / Undershoot

The customer also evaluated and tracked various image quality factors during optimization and convergence. Lab-captured chart images were included in the RAW optimization dataset for reporting.

While these could have also been used by Atlas for optimization, this was not an objective of the program. Instead, the intent was to gain insights into the way the algorithms were configured and to do sensitivity analysis with the data.

When the full dataset is analyzed across the different lighting conditions, there are substantial accuracy improvements in low light conditions over the baseline settings provided by the ISP vendor. 

These improvements ranged from a very significant 28% mAP in low light to smaller but meaningful increases for well-illuminated daytime conditions.

Here are some example images from the ISP as they were passed into the YOLOv4 model.

• The original configuration appears dark as it wasn’t necessarily tuned to render the HDR images for a standard dynamic range color space.
• The converged global tone mapping produced low contrast images for the daytime scenes but this did not degrade the detector accuracy. 
• Local contrast could be improved using a local tone mapper for dynamic range compression but that would have violated one of the customer’s systems constraints.

Note: In order to better visualize and compare the example results below for this Case Study, some basic brightness and global contrast stretching were applied to roughly equalize the images.

Despite lower contrast initially, the information required for object detection is still present in the baseline tuned images. The banding produced by the vendor-tuned ISP settings seen below indicates that the tone mapping was not well configured for the output bit depth, even at 14 bit, as there is a massive 140 dB range to cover on the input. 

There is also a massive loss of data in low light conditions as the ISP had not initially been tuned well by the provider for the system constraints and the signal ranges. Clearly, much better detector results were achieved by Atlas optimization of the mAP accuracy metric.

Examples enhanced for visualization: 

In addition to optimizing the Renesas V3H ISP for the customer’s YOLOv4 model, Atlas was also applied to the Algolux Eos detector backend to explore how much further performance could be improved when a much more accurate object detection model was used. 

Here are some example images from the ISP as they were passed into the Eos Perception model.

This case study demonstrated that it is possible to significantly improve computer vision accuracy by optimizing ISPs for pre-trained object detection models. The Atlas optimization framework is sensor and ISP-agnostic. It is not necessary to know the internal workings of the ISP algorithms, but choosing the right subset of registers and effective parameter ranges will speed up convergence time. 

Over the course of the program with this automotive Tier 1 provider, Atlas improved their configuration of the widely used Renesas V3H ISP. This was done in a way that allows their camera system to run without any dynamic controls to ensure system stability, a critical requirement by the Tier 1’s OEM customer, while maximizing computer vision accuracy. 

This was done in a matter of days, without any retraining of the vision model, by using the Atlas automated workflow to optimize the ISP parameters rather than attempt any additional lengthy expert-intensive manual tuning.