English / Russian version

Instruction Manual for the ParticlesNN Web Service:

 searching particles on scanning probe microscopy images using deep learning

Okunev A.G.1, 2, 3, Mashukov M.Yu.3, Nartova A.V.1, 2, Matveev A.V.1, 2*

1Novosibirsk State University, Pirogova Str. 1, 630090, Novosibirsk, Russia

2Boreskov Institute of Catalysis SB RAS, pr. Acad. Lavrentieva, 5, 630090, Novosibirsk, Russia

3Novosibirsk State University Higher College of Informatics, Russkaja Str. 35, 630058, Novosibirsk, Russia

Video

download it

Overview

The ParticlesNN web service was created on the basis of the CascadeRCNN neural network that was trained on nanoparticle images obtained using scanning tunneling microscopy [1]. A trained neural network is able to recognize nanoparticles as legible contours. The Service processes results by determining the size of the contours and their statistical parameters (surface coverage, concentration, average size, particle size distribution).

ParticlesNN has the following features that differ from other software products:

1) It is possible to process images containing noise, artefacts that are typical for probe microscopy images, without additional processing.

2) A user can adjust automatically determined contours with the help of external software products.

3) Joining statistical processing of image sets is available.

4) Processing results are displayed in the form of a histogram and tables where information on all identified objects is available.

Particle recognition with neural networks allows us to improve the quality of recognition over time by accumulating marked data. In addition, the web service is able to process images of any objects represented as intensity spots such as particles (for example, images obtained by different types of microscopy), biological cells, objects in images from quadrocopters, etc.

In general, our web service is a platform for processing images with neural networks. This is why there is one image processing mode not related to nanoparticles. And several more are on their way.

Data Loading

After you open http://particlesnn.nsu.ru/files and sign in to it, you will see a list of previously processed files.

We recommend uploading a .txt file created in the WSxM program [2] and have information on the scale of images in nanometers (ASCII Matrix Files or ASCII Matrix Files with labels) in the header.  Then, the output will show the results in pixels and nanometers.

ParticlesNN also supports uploading files in the usual graphic file formats (.bmp, .png and others) and the .json format created in the Labelme program [3]. When upwnloading in these formats, the results are only presented in pixels.

The neural network is less effective with small particles and / or if there is a large number of particles. The software requires no more than 500 particles in the image. The recommended image resolution is from 250 * 250 to 3000 * 3000 pixels.

To upload a file at the bottom of the page, click on the "Browse" button and select the required file. We also recommend filling out the “Description” field. After that, click on the “Upload” button, a new page will open where the data is processed.

Data Processing

You select a data processing method in the “Processing ML model” field. There are five of them:

·      MMDetection NanoPart v. 3.0 – latest version of nanoparticles counting method, generally we recommend to leave this default value,

·      MMDetection NanoPart v. 2.5 + contours correction – nanoparticles counting method with previous version of neural network, but improved with additional “algorithmic” contours correction; generally version 3.0 provides better results,

·      MMDetection NanoPart v. 2.6 – previous version of neural network, but with additional particles filtering options possibilities (we plan to add these options to v. 3.0 too)

·      People recognition – counting of people on photos,

·      dummy model – service method, for official use.

In the future, the number of models will increase and comments will be provided for each model. You can fill in the “Comments” field if you want. The comments will relate to this new run. Next, click on the "Process" button to start processing. The image is processed in about one minute. To get the results you need to refresh the browser page (as a rule, it looks like a green arrow pointing in a circle on the browser panel).

After data processing, recognition results will appear: file list, processed image, statistical results, histogram of the distribution of the found particles by size.

One of ParticlesNN processing algorithms (the “MMDetection NanoPart v. 2.5 + countours corrections” “model”) provides two versions of results for the particles found: 1) found by the neural network; 2) refined contours obtained after applying the profile fitting procedure for each particle with a Gaussian 2D distribution. Other algorithms only produce the contours, predicted by neural network, and calculate statistics on them.

1. File List:

·      xxx.log: file of the report on the progress of processing (technical file);

·      xxx.bmp: image with contours (shown to the left of the file list), found by the neural network);

·      xxx_Img_contours_predicted_fitted.bmp: image with contours (shown to the left of the file list), “predicted” (found by the neural network, green) and fit (fitting with a Gaussian 2D distribution, red); only generated by the model with contours refinement;

·      xxx_Img_masks_predicted.bmp: image with masks of particles found by a neural network;

·      xxx_Stat_bbox_predicted.csv: table (opens in Excel) with detected particle data  containing the columns: index (particle number; bbox_center_x, bbox_center_y - x and y coordinates of the particle),  area pixels (particle area in pixels),  diameter_pixels (the designed diameter, statistical diameter, of the particle calculated on the assumption that the particle is represented by a circle whose area is equal to the area of the particle), area_nm2  (particle area in nm if a .txt file was loaded with information about the image scale), diameter_nm (particle diameter in nm);

·      xxx_Stat_bbox_predicted_Commas.csv: similar to the previous file, but the table fields are separated by a comma, not a semicolon; depending on your computer’s locale, Excel can open files with semicolons and report error on comma ones or vice versa;

·      xxx_Stat_contours_predicted_fitted.csv:  table (opens in Excel) with data on detected particles and refined contours in pixels. The table contains additional columns: d_min_pred  (minimum particle diameter drawn through the center of mass), d_max_pred (maximum diameter; area_fit — area of a particle refined by fitting the profile of each particle with a Gaussian 2D distribution), d_fit, d_min_fit, d_max_fit (minimum and maximum diameters of the refined particles drawn through the center of mass);

·      xxx_Vis_Labelme_fitted.json: a file in the Labelme program format with countours, found by the neural network and refined by fitting the profile of each particle with a Gaussian 2D distribution; it is possible to load this file into the Labelme program and to correct the contours;

·      xxx_Vis_Labelme_predicted.json: a file in the Labelme program format in which it is possible to correct the contours of particles found by a neural network;

·      xxx_Vis_VGG_predicted_fitted.json: a file in the format of the resource VGG Image Annotator (VIA, [4]) where you can correct online the contours of particles found by the neural network and refined by fitting the profile of each particle with a Gaussian 2D distribution;

·      xxx_Vis_VGG_predicted.json, xxx_Vis_Labelme_predicted_Coco.json, xxx_Vis_Labelme_predicted_Via.json: similar to the previous files, but for the contours, predicted by neural network.

 

2. Statistical Results:

·       Unique masks – statistics on the contours of particles found by a neural network.

·       Total particle area – sum of ​​all particles’ areas, S.

·       Detected particles – the number of particles found by the neural network, N.

·       Average particle area S0 = S / N.

·       Overlapping area – the area of ​​particle overlapping.

·       Particles area – fraction of the image occupied by particles (in %).

·       Mid-surface diameter d = (4S0 / π) 0.5.

·       Mid-surface diameter std. dev. – standard deviation of the average surface diameter.

·       Average projected area diameter – the average value of the projected diameter (statistical diameter).

·       Average projected area diameter std. dev. – standard deviation of the average value of the designed diameter.

·       Density – the number of particles per 1000 pixels.

 

Similar results are presented for particle contours refined by fitting the profile of each particle with a Gaussian 2D distribution (Fit masks statistics).

 

3. Histogram of the Particle Size Distribution

For a more detailed statistical data analysis, we recommend using more professional software products, for example, (5) is a resource.

 

Our web service is under active development, so small discrepancies can arise between actually produced files and the file list above.

If the user is not completely satisfied with the neural network recognition results, they can run another model on the same data, adjust the contours, adding or removing erroneously found circuits using external software products such as Labelme, [3]. The former can be made by selecting the different model at the bottom of the uploaded file’s page and pressing the Process button. New results of processing will be added below the previous ones. Histogram will depict results for the latest processing.

After ParticlesNN file adjustments, you can conduct statistical processing on refined contours. To do this, you must upload the file in xxx.json format on the main page to get the processing result.

Confidentiality

You can work on the site without registration, as a guest. In that case please note what appears at the top of the recognition results page, “Save the link to this page. It is only accessible through this link". For further work, you need to save the link on your computer and it will serve as your password to this page. Like shared links in Google Drive or Dropbox. If necessary, you can give the link to colleagues and open it on any computer. It is also possible for the user to contact the administrator and make the recognition result “open”. In this case, the link will appear on the main page of the site for any user and you will not need to save it.

Nevertheless we recommend you to register on the site. In that case you will see all files you have uploaded without additional efforts. You will also be able to open access to your files to your colleagues registered under different accounts. To do so write us (contacts).

Additional Capabilities

Statistical Processing of Multiple Images

Excluding of Borders From Statistics

Literature

[1] A.G. Okunev, A.V. Nartova, A.V. Matveev, "Recognition of Nanoparticles on Scanning Probe Microscopy Images using Computer Vision and Deep Machine Learning," 2019 IEEE International Multi-Conference on Engineering, Computer and Information Sciences (SIBIRCON), Novosibirsk, Russia, 2019, pp. 0940-0943 doi:10.1109/SIBIRCON48586.2019.8958363

[2] I. Horcas, R. Fernandez, J.M. Gomez-Rodriguez, J. Colchero, J. Gomez-Herrero, A.M. Baro. WSXM: a software for scanning probe microscopy and a tool for nanotechnology, Rev. Sci. Instrum., 78 (2007), 013705-1–013705-8, https://doi.org/10.1063/1.2432410

http://www.wsxm.es/download.html

[3] Ketaro Wada, labelme: Image Polygonal Annotation with Python (2016), https://github.com/wkentaro/labelme

particlesnn.nsu.ru/data/static/tools/labelme.exe

[4] http://www.robots.ox.ac.uk/~vgg/software/via/via.html

[5] https://www.wessa.net/histo.wasp