Instruction Manual for the ParticlesNN Web Service:
searching particles on scanning probe microscopy images
using deep learning
Okunev
A.G.1, 2, 3, Mashukov M.Yu.3, Nartova A.V.1, 2, Matveev
A.V.1, 2*
1Novosibirsk
State University,
Pirogova Str. 1, 630090, Novosibirsk, Russia
2Boreskov Institute of
Catalysis SB RAS, pr. Acad. Lavrentieva, 5, 630090,
Novosibirsk, Russia
3Novosibirsk State
University Higher College of Informatics, Russkaja
Str. 35, 630058, Novosibirsk, Russia
The ParticlesNN web service was
created on the basis of the CascadeRCNN neural network
that was trained on nanoparticle images obtained using scanning tunneling
microscopy [1]. A trained neural network is able to recognize nanoparticles as legible
contours. The Service processes results by determining the size of the contours
and their statistical parameters (surface coverage, concentration, average
size, particle size distribution).
ParticlesNN has the following features that differ from other
software products:
1) It is possible to process images containing
noise, artefacts that are typical for probe microscopy images, without
additional processing.
2) A user can adjust automatically determined
contours with the help of external software products.
3) Joining statistical processing of image sets
is available.
4) Processing results are displayed in the form
of a histogram and tables where information on all identified objects is
available.
Particle
recognition with neural networks allows us to improve the quality of
recognition over time by accumulating marked data. In addition, the web service
is able to process images of any objects represented as intensity spots such as
particles (for example, images obtained by different types of microscopy),
biological cells, objects in images from quadrocopters, etc.
In
general, our web service is a platform for processing images with neural
networks. This is why there is one image processing mode not related to
nanoparticles. And several more are on their way.
After
you open http://particlesnn.nsu.ru/files
and sign in to it, you will see a list of previously processed files.
We
recommend uploading a .txt file created in the WSxM
program [2] and have information on the scale of images in nanometers (ASCII
Matrix Files or ASCII Matrix Files with labels) in the header. Then, the output will show the results in
pixels and nanometers.
ParticlesNN also supports uploading
files in the usual graphic file formats (.bmp, .png
and others) and the .json format created in the Labelme program [3]. When upwnloading in these formats, the results are only presented
in pixels.
The neural
network is less effective with small particles and / or if there is a large number of particles. The software requires no more
than 500 particles in the image. The recommended image resolution is from 250 *
250 to 3000 * 3000 pixels.
To upload
a file at the bottom of the page, click on the "Browse" button and
select the required file. We also recommend filling out the “Description”
field. After that, click on the “Upload” button, a new page will open where the
data is processed.
You
select a data processing method in the “Processing ML model” field. There are five
of them:
· MMDetection NanoPart
v. 3.0 – latest version of nanoparticles counting
method, generally we recommend to leave this default value,
· MMDetection NanoPart v. 2.5 + contours
correction – nanoparticles counting method with previous version of neural
network, but improved with additional “algorithmic” contours correction;
generally version 3.0 provides better results,
· MMDetection NanoPart v. 2.6 – previous
version of neural network, but with additional particles filtering options
possibilities (we plan to add these options to v. 3.0 too)
· People recognition – counting of people on photos,
·
dummy
model – service method, for official use.
In the future,
the number of models will increase and comments will be provided for each
model. You can fill in the “Comments” field if you want. The comments will
relate to this new run. Next, click on the "Process" button to start
processing. The image is processed in about one minute. To get the results you
need to refresh the browser page (as a rule, it looks like a green arrow
pointing in a circle on the browser panel).
After
data processing, recognition results will appear: file list, processed image,
statistical results, histogram of the distribution of
the found particles by size.
One of ParticlesNN
processing algorithms (the “MMDetection NanoPart v. 2.5 + countours corrections”
“model”) provides two versions of results for the particles found: 1) found by
the neural network; 2) refined contours obtained after applying the profile
fitting procedure for each particle with a Gaussian 2D distribution. Other
algorithms only produce the contours, predicted by neural network, and
calculate statistics on them.
1. File List:
· xxx.log: file of the report on the progress of
processing (technical file);
· xxx.bmp: image with contours (shown to the left of the
file list), found by the neural network);
· xxx_Img_contours_predicted_fitted.bmp: image with
contours (shown to the left of the file list), “predicted” (found by the neural
network, green) and fit (fitting with a Gaussian 2D distribution, red); only
generated by the model with contours refinement;
· xxx_Img_masks_predicted.bmp: image with masks of
particles found by a neural network;
· xxx_Stat_bbox_predicted.csv: table (opens in Excel)
with detected particle data containing
the columns: index (particle number; bbox_center_x, bbox_center_y - x and y coordinates of the particle), area pixels (particle area in pixels), diameter_pixels (the
designed diameter, statistical diameter, of the particle calculated on the
assumption that the particle is represented by a circle whose area is equal to
the area of the particle), area_nm2 (particle
area in nm if a .txt file was loaded with information about the image scale), diameter_nm (particle diameter in nm);
· xxx_Stat_bbox_predicted_commas.csv: similar to the
previous file, but with comma decimal separators instead of period;
depending on your computer’s locale, Excel can open files with semicolons and
report error on comma ones or vice versa;
· xxx_Stat_contours_predicted_fitted.csv: table (opens in Excel) with data on detected
particles and refined contours in pixels. The table contains additional
columns: d_min_pred
(minimum particle diameter drawn through the center of mass), d_max_pred (maximum diameter; area_fit
— area of a particle refined by fitting the profile of each particle with a
Gaussian 2D distribution), d_fit, d_min_fit,
d_max_fit (minimum and maximum diameters of the
refined particles drawn through the center of mass);
· xxx_Vis_Labelme_fitted.json: a file in the Labelme program format with countours, found by the neural network and refined by
fitting the profile of each particle with a Gaussian 2D distribution; it is
possible to load this file into the Labelme program
and to correct the contours;
· xxx_Vis_Labelme_predicted.json: a file in the Labelme
program format in which it is possible to correct the contours of particles
found by a neural network;
· xxx_Vis_VGG_predicted_fitted.json: a file in the format of the resource VGG Image
Annotator (VIA, [4]) where you can correct online the contours of particles
found by the neural network and refined by fitting the profile of each particle
with a Gaussian 2D distribution;
· xxx_Vis_VGG_predicted.json, xxx_Vis_Labelme_predicted_Coco.json,
xxx_Vis_Labelme_predicted_Via.json: similar to the
previous files, but for the contours, predicted by neural network.
2. Statistical Results:
· Unique masks – statistics on the contours of particles found by a
neural network.
· Total particle area – sum of all particles’ areas, S.
· Detected particles – the number of particles found by the neural network,
N.
· Average particle area S0 = S / N.
· Overlapping area – the area of particle overlapping.
· Particles area – fraction of the image occupied by particles (in %).
· Mid-surface diameter d = (4S0 / π) 0.5.
· Mid-surface diameter std.
dev. – standard deviation of
the average surface diameter.
· Average projected area
diameter – the average value of
the projected diameter (statistical diameter).
· Average projected area
diameter std. dev. – standard deviation of
the average value of the designed diameter.
· Density – the number of particles per 1000 pixels.
Similar results are presented for particle contours
refined by fitting the profile of each particle with a Gaussian 2D distribution
(Fit masks statistics).
3. Histogram of the Particle Size Distribution
For a more detailed
statistical data analysis, we recommend using more professional software
products, for example, (5) is a resource.
Our web
service is under active development, so small discrepancies can arise between
actually produced files and the file list above.
If the
user is not completely satisfied with the neural network recognition results,
they can run another model on the same data, adjust the contours, adding or
removing erroneously found circuits using external software products such as Labelme, [3]. The former can be made by selecting the
different model at the bottom of the uploaded file’s page and pressing the
Process button. New results of processing will be added below the previous
ones. Histogram will depict results for the latest processing.
After ParticlesNN
file adjustments, you can conduct statistical processing on refined contours. To
do this, you must upload the file in xxx.json format
on the main page to get the processing result.
You can
work on the site without registration, as a guest. In that case please note
what appears at the top of the recognition results page, “Save the link to this
page. It is only accessible through this link". For further work, you need
to save the link on your computer and it will serve as your password to this
page. Like shared links in Google Drive or Dropbox. If necessary, you can give
the link to colleagues and open it on any computer. It is also possible for the
user to contact the administrator and make the recognition result “open”. In this
case, the link will appear on the main page of the site for any user and you
will not need to save it.
Nevertheless
we recommend you to register on the site. In that case you will see all files
you have uploaded without additional efforts. You will also be able to open
access to your files to your colleagues registered under different accounts. To
do so write us (contacts).
Statistical Processing of Multiple Images
Excluding of Borders From Statistics
[1] A.G.
Okunev, A.V. Nartova, A.V. Matveev, "Recognition of Nanoparticles on Scanning
Probe Microscopy Images using Computer Vision and Deep Machine Learning," 2019
IEEE International Multi-Conference on Engineering, Computer and Information
Sciences (SIBIRCON), Novosibirsk, Russia, 2019, pp. 0940-0943
doi:10.1109/SIBIRCON48586.2019.8958363
[2] I. Horcas, R. Fernandez, J.M. Gomez-Rodriguez, J. Colchero, J. Gomez-Herrero, A.M. Baro.
WSXM: a software for scanning probe
microscopy and a tool for nanotechnology, Rev. Sci. Instrum.,
78 (2007), 013705-1–013705-8, https://doi.org/10.1063/1.2432410
http://www.wsxm.es/download.html
[3] Ketaro Wada, labelme: Image Polygonal Annotation with Python
(2016), https://github.com/wkentaro/labelme
particlesnn.nsu.ru/data/static/tools/labelme.exe