Deep Learning detection of nanoparticles and multiple object tracking of their dynamic evolution during in situ ETEM studies

A new article by the Laboratoire Hubert Curien’s Image Science and Computer Vision Team regarding multiple object tracking has recently been published in the multi-disciplinary journal Scientific Reports, from the publishers of Nature.

This work is the result of a collaboration with the MATEIS, CREATIS and IRCELYON laboratories, and presents an approach combining deep learning and computer vision for the detection and in situ monitoring of nano-objects’ populations. This approach allows the extraction of small objects’ complex trajectories on a noisy and non-uniform background. Its application to the analysis of palladium nanoparticles on an alumina substrate during their calcination under oxygen by ETEM electron microscopy has provided an opportunity to highlight coalescence and fusion phenomena, which are very important for catalytic properties.

This study has benefited from the support of the CNRS IngéLySE Federation’s scientific program and EUR SLEIGHT (Université de Lyon - St-Etienne) 

Abstract

In situ transmission electron microscopy (TEM) studies of dynamic events produce large quantities of data especially under the form of images. In the important case of heterogeneous catalysis, environmental TEM (ETEM) under gas and temperature allows to follow a large population of supported nanoparticles (NPs) evolving under reactive conditions. Interpreting properly large image sequences gives precious information on the catalytic properties of the active phase by identifying causes for its deactivation. To perform a quantitative, objective and robust treatment, we propose an automatic procedure to track nanoparticles observed in Scanning ETEM (STEM in ETEM). Our approach involves deep learning and computer vision developments in multiple object tracking. At first, a registration step corrects the image displacements and misalignment inherent to the in situ acquisition. Then, a deep learning approach detects the nanoparticles on all frames of video sequences. Finally, an iterative tracking algorithm reconstructs their trajectories. This treatment allows to deduce quantitative and statistical features about their evolution or motion, such as a Brownian behavior and merging or crossing events. We treat the case of in situ calcination of palladium (oxide) / delta-alumina, where the present approach allows a discussion of operating processes such as Ostwald ripening or NP aggregative coalescence.

Read the full article at: https://www.nature.com/articles/s41598-022-06308-2

Contact: Christophe Ducottet ducottet @ univ-st-etienne.fr (ducottet @ univ-st-etienne.fr)

Illustration above: Synopsis of the algorithm. (1) Registration of the real input sequence, (2) training of a U-Net model with simulated images, (2’) detection and segmentation of particles in each frame using the model, (3) determination of trajectories using the dedicated NP-Tracker algorithm

Animated illustration above: Example of simulated sequence (field of view of 100x100 𝑛𝑚2) and the corresponding result of detection (yellow circles) and tracking (blue for ongoing trajectories and red for ended ones)