PrimeEnhance and PrimeLocate

Introduction

There is currently a revolution in computational imaging, pairing the power of a cutting-edge scientific camera, such as the Prime Family of CMOS cameras from Teledyne Photometrics, with powerful graphics processors from the latest generation of GPUs. Machine learning, data processing and the camera-PC interface are all going to help imaging evolve, through methods such as denoising and localisation.

In this article, we explore two pieces of software in the Prime Family of CMOS cameras, namely PrimeEnhance™, used for denoising, and PrimeLocate™, used for localisation. Prime’s computational power matches the latest generation of GP GPU’s while making the technology easy to deploy and accessible by embedding computational power inside the camera.

PrimeEnhance™

While camera and sensor manufacturers have made steady progress towards perfecting photometric imaging (such as minimizing read noise with technology such as Correlated Multi Sampling), low light imaging remains difficult due to the presence of another source of noise, photon shot noise.

Photon shot noise is an inherent property of light. There is always a statistical variation in the number of photons (or photoelectrons) detected in a given time period. This uncertainty is dependent on the number of signal photoelectrons being measured and has the statistical property of a Poisson distribution. For N signal, there will be √N shot noise, meaning that while shot noise increases as signal increases, it does so more slowly (due to the square root), meaning that shot noise has a more significant impact at low light levels, as outlined with Table 1.

Signal Level (e-)Shot Noise (e- RMS)Percent of signal (%)
52.2344.8
103.1631.6
507.0714.1
1001010
50022.364.5
100031.623.2
Table 1: The effect of photon shot noise at varying signal levels.

In order to improve the signal-to-noise ratio (SNR) at lower signal levels, researchers could increase exposure or light level, but the former affects the camera acquisition speed, and both affect the sample. A remaining technique to reduce the effects of photon shot noise is the use of a denoising algorithm.

When processing data to reduce noise, there are many challenges to consider, such as preserving the quantitative nature of the recorded pixel intensities, preserving key features (edges, textures, details with low contrast), and processing has to be accomplished without introducing new image artifacts like ringing, aliasing or blurring. Many denoising algorithms are currently available, and some are incompatible with certain image types, resulting in artifacts. Additionally, because noise tends to vary with the level of signal, it is difficult for many denoising algorithms to distinguish signal from noise, and as a consequence, small details tend to be removed.

For PrimeEnhance, we have used an algorithm invented at the French National Institute for Research in Computer Science and Automation (INRIA) by Jérôme Boulanger et al. This algorithm has been optimized for fluorescence microscopy in collaboration with the Institute Curie, and works with the fact that while shot noise is a fundamental property of light, fluctuations caused by shot noise behave in a predictable fashion governed by Poisson statistics. PrimeEnhance implements a 2D denoising process that evaluates and processes incoming images in real-time to reduce the effects of photon shot noise at low signal levels. The algorithm also preserves the finer details and features of biological samples and does not introduce image artifacts. In addition, PrimeEnhance is a quantitative algorithm, ensuring that intensity values remain unchanged. The implementation of PrimeEnhance has a significant impact on photon shot noise, winning the Vision Systems Innovator Award 2016.

PrimeEnhance works by being aware of each camera’s characteristics and specifications. It uses this knowledge to first evaluate the image data and perform a variance stabilization transform, which removes the dependency between the mean intensities and their noise characteristics. Then a small patch of pixels is compared to similar sized patches in iteratively increasing areas of surrounding pixels (neighborhoods). The pixels within the neighborhood are selectively weighted based on their similarity to the intensity values of the original patch and using these weighted corrections, the original patch is updated. This process is repeated throughout the entire image, updating each patch and reducing the impact of shot noise. Once this process has been completed, the inverse variance stabilizing transform is applied to ensure that the quantitative nature of the pixel values is maintained.

PrimeEnhance Evaluation

In Fig.1 two images are shown, acquired without (left) and with (right) PrimeEnhance. Both images are of the same sample and with the same exposure. The mean, minimum, and maximum intensity values remain unchanged between both images, demonstrating that all measurements made remain quantitative and are relatable to each other. In the PrimeEnchance image, the standard deviation has been reduced, indicating the removal of noise. Within the PrimeEnhance image, features have been preserved while no artifacts have been generated, shot noise in the image has been reduced, extracting features that were previously indistinguishable from the noise.

Figure 1: The effect of PrimeEnhance. The top image is raw, the bottom image has PrimeEnhance activated.

By increasing the effective signal to noise in each frame, it is possible to acquire high-quality images at lower exposure times, reducing the effects of phototoxicity and photobleaching on samples. By way of demonstration, Fig.2 shows three imaging conditions: raw images taken with 100 ms and 800 ms exposure, and a PrimeEnhance image taken with 100 ms exposure. All images are accompanied by line profiles to show the presence of noise and how clear the signals are. The raw 100 ms image contains considerable noise and features are unclear, increasing the exposure to 800 ms results in 8x more signal and a clearer image. However, increasing the exposure limits the camera speed and exposes the sample to more light, which can lead to bleaching and photodamage, and also requires more time per image acquisition, adding inefficiency to experimentation. The alternative is keeping the exposure at 100 ms and using PrimeEnhance, which also removes noise, as seen from the line profile and clearer features, resulting in high image quality while maintaining low exposures.

Figure 2: PrimeEnhance and exposure time. The raw image at 100 ms has a low signal and is more significantly impacted by photon shot noise, this can be improved by either increasing the exposure by 8x (see 800 ms raw image) or by using PrimeEnhance.

Reference

Jérôme Boulanger, Charles Kervrann, Patrick Bouthemy, Peter Elbau, Jean-Baptiste Sibarita, Jean Salamero (2010) Patch-Based Nonlocal Functional for Denoising Fluorescence Microscopy Image Sequences, IEEE Transactions on Medical Imaging, Institute of Electrical and Electronics Engineers, 29 (2), pp.442-453. ⟨10.1109/TMI.2009.2033991⟩. ⟨inria-00541082⟩

PrimeLocate™

Advanced microscopy applications such as super-resolution and single-molecule often require localization, such as single-molecule localization microscopy (SMLM) for STORM or PALM imaging. With the high image quality of the Prime family, identifying and localizing these molecules is a simple matter, but the full image frame is transferred to the host for analysis when the molecules of interest may only occupy a tiny percentage of the frame. This results in a large amount of data transfer and processing time without adding useful information.

PrimeLocate dynamically evaluates image data and locates up to 500 regions per frame containing single-molecule data relevant for localization (especially for super-resolution applications). Only the selected regions are transferred to the host computer, drastically reducing the amount of data and time required for analysis. By transferring only the relevant raw data, users have the freedom to use their preferred localization algorithm to generate super-resolution images.

PrimeLocate allows for full speed 100 fps acquisition of super-resolution data over USB 3.0 interfaces, reducing full-frame acquisitions from ~800 MB per second to ~12 MB per second. This simplifies image capture, as well as reducing processing time and storage requirements, making your experiments more efficient!

Figure 3: PrimeLocate workflow. Molecules in an image are located by software, and only these three molecule co-ordinates are read out, rather than the whole frame.

Summary

Computational imaging methods are powerful pieces of software that can greatly assist specific needs, and the Prime camera family comes packed with several pieces of software to ease and improve quantitative scientific imaging. PrimeEnhance quantitatively increases the peak signal to noise ratio by ~3-5x by reducing photon shot noise, increasing the clarity and quality of images. PrimeLocate dynamically evaluates acquired images and reduces the surplus of data generated during high-speed super-resolution imaging, well suited for localization microscopy.