Automatic classification of spread‐F types in ionogram images using support vector machine and convolutional neural network

Abstract

An ionogram image serves as a valuable data for examining the ionospheric bottom side characteristics and variabilities. Spread-F is indicated or identified by plasma irregularity in the ionospheric region. Diffused echo in the ionogram images particularly pose challenges for efficient interpretation required in further applications. An automatic classification of spread-F is presented in this study. Ionogram images are automatically classified using preprocessing techniques to improve the classification performance. In this study, the classification is designed by two machine learning algorithms, including support vector machine (SVM) and convolutional neural network (CNN). The CNN model with preprocessing technique outperforms the SVM alternative based on 4,692 labelled ionogram images from the FMCW-type ionosonde at Chumphon station, Thailand. The model successfully classified clear, frequency spread-F (FSF), range spread-F (RSF), strong spread-F (SSF), and unidentified class with an accuracy of 98.0%, 85.1%, 90.7%, 66.7%, and 99.2%, respectively. The proposed automatic classification models achieved to classify classes of ionogram images. In addition, the image filtering and data preprocessing are useful with ionogram images for improving the model classification performance.

Graphical Abstract

Introduction

The ionospheric layer is the upper part of the atmosphere that is ionized by solar radiation and forms a dense of charged particles changing over time (Bowman 1960). The ionospheric bottomside characteristics can be observed by different ionosonde systems, such as frequency-modulated continuous wave (FMCW) ionosonde (Yao et al. 2012) and digital ionosonde (Rao et al. 2022a). The FMCW-type ionosonde with pulse compression technique used at Chumphon station, Thailand transmits continuous pulses of high frequency range of 2–30 MHz toward the ionosphere layers. The signals are reflected by the ionospheric plasma resulting in the echoes being recorded and then the ionosonde displays the recorded data on a graph called the ionogram (Nozaki 2009; Thammavongsy et al. 2020). An ionospheric disturbance in the F layer appears on the ionogram as spread-F traces often characterize the presence of plasma irregularities, typically observed during night-time when the ionosphere is less influenced by solar radiation. In addition, equatorial spread-F specifically occurs in the equatorial region. Furthermore, there are several types of spread-F phenomenon, each exhibiting distinct characteristics, including frequency spread-F (FSF), range spread-F (RSF), mixed spread-F (MSF), and strong spread-F (SSF) (Wang et al. 2008). Figure 1 illustrates the characteristics of ionogram images, including a typical ionogram, as shown in Fig. 1a, and ionograms with spread-F types. FSF in Fig. 1b represents a horizontal spread of ionogram traces around the F region critical frequency, which can be observed in various patterns. RSF in Fig. 1c is a vertical spread of ionogram traces over the wide range of frequency. MSF in Fig. 1d can be observed as a combination of spreading characteristics between FSF and RSF. SSF in Fig. 1e is an intensified version of RSF, where the spreading conditions are significantly expanded. In addition, if the FMCW-type ionosonde fails to receive a reflected signal or encounters an error during the process, it plots an ionogram with no trace, classified as an unidentified class in this study, as shown in Fig. 1f.

Fig. 1

Ionogram sample of a clear class, b FSF class, c RSF class, d SSF class, e MSF class, f unidentified class

The presence of this phenomenon causes the ionization density in ionospheric layer to become irregular and the plasma depletion in some areas has relatively lower density than the surrounding areas, hence, impacting the transmission of radio waves, leading to an unclear, poor quality, delays, and errors in recorded data (Thammavongsy et al. 2020). Consequently, it is essential to conduct ionospheric observations for various applications, such as developing an alert system to detect irregularities in radio communication systems and global positioning systems (GPS).

These observations are crucial for ensuring reliable and accurate performance in these systems. Nowadays, the ionosondes operators are required to manage the processing of ionogram data, involving scaling task and categorizing thousands of ionogram images. To solve this problem, an application of machine learning is introduced. Machine learning has a powerful ability to manage the large amount of data which can be well-applied for automatic ionogram image classification tasks. In general, machine learning can be classified into three types which are supervised learning, unsupervised learning, and reinforcement learning. The supervised learning is a learning of the machine by feeding labelled data directly. The unsupervised learning is learning by feeding raw data or unlabeled for training, and the reinforcement learning is learning through decision making to optimize the outcome (Janiesch et al. 2021). According to types of machine learning, supervised learning is the most suitable to be used with ionogram images due to large amounts of available data (Luwanga et al. 2022).

In data management of ionogram, there was a method proposed by Xiao et al. (2020) using deep learning method for ionogram automatic scaling (DIAS) to scale a large amount of ionosonde data. DIAS model consists of encoder and decoder networks. The encoder network model was evaluated and compared using VGG16, ResNet50 and Efficient-b5 as backbones. While the decoder network has applied the feature pyramid network (FPN) module to enhance the scaling accuracy. The results showed that the DIAS model with ResNet50 backbone and FPN module scaled ionograms with the precision of 95.79%, while the traditional method achieved only 88.67%. In 2021, De La Jara and Olivares presented a method of using CNN to detect ionospheric echo in digital ionograms with three different models, evaluated by Intersection over Union (IoU) to measure the accuracy between manual and automatic trace detector. The first model was fed by filtered images. The second model trained by adding manually extracted images after the first model. The last model was trained by feeding manually extracted data only which achieved the IoU value of 0.174, 0.602, and 0.569, respectively. This model also showed that extracted data can highly improve the IoU value in this work. Xue et al. (2022) presented an echo extraction for three types of ionograms (vertical, oblique, and backscatter ionograms) using CNN as classification and extraction model with residual learning and skip connection structure which improved the model performance compared with the traditional method by 22.18%, 22.56%, and 6.67%, respectively. In the same year, Luwanga et al. proposed a method for spread-F detection on digital ionogram image through SVM and CNN models with three different based models which are VGG16, InceptionV3, and ResNet50. The results showed that SVM model achieved the precision score of 77.00% and CNN model with ResNet50 achieved the precision score of 95.00%. However, the SVM model in this research showed poor performance and then was abandoned for the further evaluation. Therefore, the performance of SVM model on ionogram data remains inconclusive. In 2022, the auto-detection method for ionospheric irregularity in digital ionogram proposed by Rao et al. (2022b) A tool based on fuzzy relation that detects the height and frequency points in the denoised digital ionogram images to identify them into classes. The proposed method was able to detect ionograms with an efficiency of 96.71%, 97.83%, 89.71%, 68.32%, and 93.39% for sporadic-E, FSF, RSF, MSF, and SSF events, respectively. Recently, Wang et al. (2023) presented a deep learning model for spread-F detection and classification. The digital ionogram over 100,000 images were used for training, evaluating, and testing on various models. The ionogram had been cropped, resized and added with the simulated noise into the original images. The results indicated that ResNet50 achieved a test accuracy of 92.36% to detect and classify the ionogram into FSF, RSF, MSF, SSF, and no spread-F. Accordingly, the most used ionogram image in the mentioned works are digital ionograms which are different from the used ionogram images in this work as recorded by the FMCW system which contains more variation of noises. Additional image preprocessing methods are introduced to deal with all unwanted noise and improve the data quality in the models for classification and further applications.

Therefore, the main purpose of this work is to evaluate model performance between shallow and deep machine learning structures which are SVM and CNN models for operating the automatic classification task with ionogram data set. To alleviate the manual tasks associated with ionogram data for the FMCW-type ionosonde. In addition, the preprocessing methods are also investigated to improve the model performance using different techniques such as image filtering with image sharpening, image thresholding, median blur, gamma correction, fast non-local means, and bilateral filters. These techniques are utilized in training, validation, and testing processes for both models.

Experiment setup

This section provides information on the experimental design, including data set preparation, image preprocessing methods, model details, and evaluation process to determine the most effective approach for improving the model performance.

Data set preparation

The ionogram data set used in this study are obtained from the FMCW ionosonde at Chumphon station, Thailand under the administration of the Southeast Asia Low-latitude Ionospheric Network (SEALION) Project (Maruyama et al. 2007). The data collection period in this study spans from March to May and August to October in 2014–2016 and 2018–2020, respectively. The data was collected during specific periods to consider seasonal changes in the phenomena. The data set of 4693 ionogram images were manually classified by expert inspector into clear class, ionogram with no spread-F, (1320 images), FSF class (760 images), RSF class (1620 images), SSF class (126 images), and unidentified class (866 images). The sample of each ionogram classes are shown in Fig. 1.

For the spread-F occurrence, RSF events frequently occur near the Chumphon station, (Rungraengwajiake et al. 2013), resulting in the RSF class having the highest number of samples in spread-F classes. Conversely, SSF events rarely occur in the equatorial region, resulting in the lowest number of samples. Subsequently, the data set is divided into three subsets which are training set, validation set, and test set. The data set is split into three sets, including 70% of training set, 15% of validation set and 15% of test set, (Razzano and Cuoco 2018). From the total image, the training set is 3,285 images, the validation, and test set are 704 images. When splitting the data set, it is important to ensure that there are sufficient samples for training while also avoid having too few samples for validation and testing. It is noticed that the data classes in this work are severely imbalanced, particularly within the SSF class, there is a risk of having an insufficient number of samples available for evaluating the model during validating and testing.

Image preprocessing methods

The manipulation of the data set is considered before analyzing or using in other processes of machine learning tasks including image preprocessing and image filtering with the purpose to filter out noise and enhance the features in ionograms. Image filtering technique can be used to reduce noise, distortion and improve the overall information of the ionogram images (De La Jara and Olivares 2021). With proper image preprocessing, it can directly improve the performance and efficiency of the models in terms of computer vision (Xiao et al. 2020). In this work, there are seven preprocessing techniques to be evaluated which are described below.

• 1) Image cropping: a technique to specifically crop the image to obtain only the necessary part of the ionogram. By cropping the F layer region from the full ionogram image, the image contains only the important information.

• 2) Image thresholding: a thresholding technique to separate the region in an image by determining the threshold value.

• 3) Image sharpening: a filter that passes a specific kernel matrix through image pixels, resulting the sharpened image. The kernel matrix is defined as $\left[\begin{array}{ccc}0& -1& 0\\ -1& 5& -1\\ 0& -1& 0\end{array}\right]$.

• 4) Median blur: a technique for image denoising by replacing the value of each pixel with the median value of the neighboring pixels within a defined window.

• 5) Gamma correction filter: a technique for adjusting the overall brightness and contrast of the image by determining the gamma value. By increasing gamma value, the pixel intensity in image will become brighter while decreasing gamma value resulting darker output.

• 6) Fast non-local means filter (fast-NLmeans): a denoising technique with ability to preserve the structure of the image and denoise by comparing the similarity between nearby image area and calculate the average normal value (without noise) of each area.

• 7) Bilateral filter: a filter for noise reduction while preserving important image structure by considering both spatial proximity and intensity similarity between pixels.

From the presented preprocessing methods, the image cropping is used as a preliminary process for both SVM and CNN models to crop out the unwanted data from the full size ionogram images. While other preprocessing will be applied and evaluated on the ionogram images separately to compare the classification performance.

Model details

This study presents the application of SVM and CNN models to evaluate the classification performance on this ionogram data set. Typically, the SVM has a simple architecture to determine the hyperplane and classify the data into classes using the kernel trick. While the CNN has a more complex architecture with multiple convolutional layers and neurons to analyze various features from the input data. It should be noted that this experiment is conducted on different types of ML model to provide the performance comparison between two different model structures on classifying this ionogram data set and examine the impacts of the proposed preprocessing methods on both models. Therefore, the best way to evaluate the improvement of the proposed method must be examined for both SVM and CNN algorithms on the specific data set then compare the results. The structure of the image classification methodology of each model is shown in Fig. 2.

Fig. 2

Flowchart of ionogram classification of a SVM model, b CNN model

Parameters for model evaluation

The metrics for model evaluation in this study are utilized as follows,

• 1. Precision represents the accuracy of positive predictions by calculating the ratio of true positives (ionogram with spread-F) with the sum of true positives and false positives as shown in following formular

  $${\text{Precision}}=\frac{{\text{True Positive}} }{{\text{True Positive}} +{\text{False Positive}} }$$

• 2. Recall represents the performance to identify all positive instances by calculating true positive with the sum of true positives and false negatives as shown in following formular

  $${\text{Recall}}=\frac{{\text{True Positive}} }{{\text{True Positive}} +{\text{False Negative}} }$$

• 3. F1-score calculates the balanced measurement of the model performance between the ratio of precision and recall as shown in following formular

  $${\text{F1 score}}= 2\times \frac{{\text{Precision}}\times {\text{Recall}}}{{\text{Precision}}+{\text{Recall}}}$$

• 4. Support represents the number of tested samples for each ionogram class.

• 5. Accuracy represents the overall model performance by calculating the ratio between correct prediction and total number of tested samples as shown in following formular.

  $${\text{Accuracy}}=\frac{{\text{True Positive}} +{\text{True Negative}} }{{\text{Total number of predictions}} }$$

SVM model for image classification

SVM is a supervised machine learning algorithm used for data classification and regression. The algorithm determines the best hyperplane that can divide the data into separated categories by maximizing the margin width, which is the gap between the closest data points and the hyperplane. The SVM can handle low- and high-dimensional data and can be used for managing both linear and non-linear data using different kernel functions to transform the input data into a higher-dimensional space for data separation (Brereton and Lloyd 2010; Raghavendra and Deka 2014). A linear kernel creates a linear boundary in the original feature space, while polynomial and Radial Basis Function (RBF) kernels use nonlinear transformations to achieve nonlinear boundaries in higher-dimensional spaces. Thus, these mentioned kernel tricks were evaluated to find out the most suitable kernel for this ionogram data set.

Table 1 defines the parameters for tuning the SVM classification model in this work. All input data are cropped into three-dimensional tensor, and a depth of one channel in grayscale images. The three main variables in SVM are C parameter, gamma value, and kernel function. C parameter is the regularization variable that adjusts the cost of misclassifying in training sample. Gamma value is used to determine a shape of the decision boundary, and kernel function is a mathematical function to transforms data into a higher-dimensional space for data separation. As previously stated, the way to obtain the optimized parameter is to utilize a hyperparameter tuning process. Hyperparameter tuning is used to determine the optimal values for the model to improve performance on a specific data which typically done through a combination of parameters and variables, then sort out the best parameters to optimize the SVM model. Another preprocessing method for SVM classification model in this work is K-means clustering for data pre-separation. This clustering technique is a method of grouping samples based on their characteristics, aiming to place similar samples into the same group and distinguish them from other groups. The objective is to group the similar image based on their intensities together into clusters while maximizing differences between groups and minimizing variations within groups (Pham et al. 2005). In this step, K-means clustering is tested to separate the ionogram into three different size of clusters which are 2, 3, and 4. Then, each cluster will be applied with image filtering to examine how image filter affects the training and evaluation of SVM model.

Table 1 SVM model architecture for ionogram image classification task

CNN model for image classification

CNN is a type of neural networks commonly used in deep learning for image recognition and classification tasks using a series of convolutional layers to extract the features from the input image which need the pooling layers to reduce their dimensionality and resulting the extracted features of the input data. After, the model is trained by feeding the network with labelled input image and adjusting the network weights to minimize the difference between the predicted output (Chollet 2022). The last is classification process in fully connected (FC) layer consisting of neurons that applies the trained weight and connected to every neuron from the previous layer and apply activation functions to the sum of weighted values into the final output of image classification (Yamashita et al. 2018).

In this work, a VGG16 pretrained model is used as a base model for feature extraction, as defined in Table 2. Originally, the VGG16 model was developed by the Visual Geometry Group at the University of Oxford. The "16" in VGG16 refers to the number of layers in the network and has been trained on a large data set of images for computer vision and image classification tasks (Simonyan and Zisserman 2014). As determined in SVM model, a primary setup such as, the input image size, the number of samples and image cropping for VGG16 are the same.

Table 2 VGG16 model architecture for ionogram image classification task

On the other hand, CNN model will be applied with ImageDataGenerator as data augmentation besides applying on SVM model due to the process can be done only on CNN model. ImageDataGenerator is a utility function in Keras library works by generating batches of image with real-time augmentation during the training process of CNN model. It applies data transformation (image filtering) during the training process. ImageDataGenerator can add the additional training samples in CNN model. The primary objective of the optimized model is to minimize training loss by adjusting the basis model parameters. This involves an optimizer, which determines how the model parameters are updated. Learning rate is a hyperparameter controlling the step size during learning process. Batch size represents the number of training samples processed in each iteration. Epoch indicates the number of training processes, allowing the model to learn and refine its performance. The number of dense layers defines the model capacity and complexity. The dropout is a model regularization technique by randomly dropping the neuron units during the training process to prevent the overfitting problem. Lastly, the model performance is evaluated by classifying unlabeled ionogram images.

Results and discussions

This section provides the experiment result on preprocessing methods on ionogram image and both classification models, also the parameter determination for both SVM and CNN models. Following this, the performance of both models for ionogram image classification is also evaluated.

Image preprocessing method

For both SVM and CNN models, the input images are cropped according to the F layer region of the ionosphere. As shown in Fig. 3, the original ionogram image size of 738 by 500 pixels is cropped into image shape of 224 by 224 pixels with 1 greyscale channel to select the useful part of ionogram data. Furthermore, utilizing a down scaling with the input data can result in a faster training process.

Fig. 3

Ionogram image with a original size, b cropped size

SVM model performance

In this work, the parameters of SVM model are first tuned using hyperparameter tuning technique. Hyperparameter tuning refers to finding the optimal values that maximized model performance by evaluating each combination of C parameter, gamma value, and kernel trick on the test data set, as shown in Table 3.

Table 3 Hyperparameter tuning: the highest accuracy results of each kernel trick on SVM model

From Table 3, the tuned hyperparameters consist of C parameter equals to 0.1, Gamma value as auto and kernel trick as polynomial. The next step is to apply these parameters to the SVM model and utilize the clustering technique to group up the data set with similar image intensity. K-mean clustering technique has been applied to separate ionogram images, as shown in Table 4.

Table 4 Results of applying clustering technique to ionogram by descending the mean value of image intensity

Table 4 shows examples of ionogram images from different clustering size. The size of clusters determines the number of groups to be separated as indicated by the number of subclusters within 2, 3, and 4 clustering. If the cluster size is not aligned with the characteristics of the data set, it can lead to suboptimal separation. Thus, the clusters are varied and partitioned to examine the benefits of the clustering technique across various cluster sizes.

The specification of the number of clusters is required before grouping into subclusters. Thus, the number of clusters varies, ranging from 2, 3, to 4 clusters. To determine the proper clustering size, all three clustering sizes must be evaluated. The experiment is then applied with 6 different image filters on each cluster to examine the impact of image filters on different clustering sizes.

Following this, a proposed image filtering technique is introduced to improve the training and learning process of the SVM model. The purpose of image filtering is to transform and extract the information from the original data, enhancing its utility and features for the model. Figure 4 shows an ionogram image with each of applied image filter, in Fig. 4b, sharpening filter sharpens the overall components in the image making it more noticeable. In Fig. 4c, thresholding filter results in a significant separation between low and high image intensity. In Fig. 4d, median filter slightly blurs out grain noise in the ionogram image. In Fig. 4e, gamma correction emphasizes the overall image intensity similar to the thresholding filter. In Fig. 4f, fast-NLmeans filter denoises the area with low intensity and remained the ionogram trace. In Fig. 4g, bilateral filter fades the background noise, while the ionogram trace still remained.

Fig. 4

Samples of ionogram image with a no image filter, b image sharpening filter, c image thresholding filter, d median blur filter, e gamma correction filter, f fast-NLmeans filter, g bilateral filter

Then, the SVM model is evaluated for ionogram image classification utilizing the clustering technique with 2, 3, and 4 clusters, and subsequently tested each scenario by applying each image filters to every possible subcluster aiming to thoroughly examine the outcomes, as listed in Tables 5, 6, 7.

Table 5 SVM model classification accuracy with image filters and clustering size of 2

Table 6 SVM model classification accuracy with image filters and clustering size of 3

Table 7 Results of applying image filters with clustering size of 4 on SVM model

From Table 5, each image filter is utilized with all clusters separately. Meanwhile, when referring to “All clusters”, it means that the filter is applied on every sample during each evaluation. The results of applying the different image filters showed that the image sharpening filter achieved the highest classification accuracy of 86.5% when applied on the 1st cluster. While thresholding filter on all clusters and fast-NLmeans filter on the 1st cluster, both accuracies achieved 86.4%. Utilizing clustering technique leads to improved model performance compared to not using it. However, the achieved highest accuracy from 2 clustering is insignificant. Hence, testing more clustering size is evaluated.

Since, there are multiple subclusters, a comprehensive evaluation is conducted, considering all possible cluster separation scenarios. This allows for a detailed examination of the image filter impact on these subclusters, as illustrated in Table 6. The results showed that applying image sharpening filter on the 1st and 2nd clusters achieved the accuracy of 87.0% followed by fast-NLmeans filter on the same clusters achieved the accuracy of 86.5%. In this regard, enhancing the image quality on high image intensity ionogram can improve the SVM model accuracy with 3 clustering efficiently.

Table 7 shows the model classification accuracy on all possible cluster separations utilizing the image sharpening filter can increase the model accuracy to be 86.8% when applied on the 1st and 3rd clusters and on only the 3rd cluster. Distinctly, applying median blur filter on the 4th cluster which contains low image intensity ionograms can also improve the accuracy to 86.8% as same as image sharpening filter.

After applied image filters and evaluated on 2, 3, and 4 clustering sizes, the SVM model with 3 clustering and image sharpening filter applied on the 1st and 2nd cluster successfully classified with the highest accuracy of 87.0%, as shown in Table 8.

Table 8 Results of SVM model with 3 clustering and image sharpening filter

From Table 8, FSF and SSF classes achieved lower accuracies among the others because FSF phenomenon can be observed in various characteristics. Therefore, the model needs more of FSF samples with higher diversity of data set for the learning and training process, while SSF phenomenon significantly rare to be observed causing the data set is lack of an effective quantity of SSF samples. Moreover, image sharpening filter on 2 and 4 clustering sizes also achieving the improved accuracy means that SVM model performance had improved by utilizing a filter to emphasize the overall image features.

CNN model performance

Since CNN model generally has more parameters and more complex structure comparing to SVM model, each parameter can be adjusted or calibrated to obtain the best setup for each task. In this work, VGG16 pretrained model is selected to be used as a base model for the CNN classification model. VGG16 has been well performed in various classification competitions and has been used to detect the spread-F occurrence on the digital ionogram image (Luwanga et al. 2022). Thus, VGG16 is a reliable pretrained model with suitable size of model structure to be evaluated in this work.

Table 9 shows the obtained results of using VGG16 to classify the ionogram data. The results indicate that the model can classify ionogram images with the accuracy of 89.3%. The SSF class achieved quite low accuracy due to insufficient training and validation data.

Table 9 CNN model performance with original VGG16 structure

CNN model parameter tuning

Parameter tuning is used to observe the impact of different parameter settings on the model such as adjusting the model optimizer, learning rate and varying the size of fully connected layer. Tuning can help the model to gain more efficiency for capturing and extracting image features improving the model learning effectiveness, reduce overfitting, and enhance computational efficiency (Kandel and Castelli 2020). The model structure was first modified by adjusting the optimizer and learning rate to observe the impacts of each configuration and find the most suitable parameter setup. The optimizers were evaluated using Adam, Adagard, and Stochastic Gradient Descent (SGD), while the learning rate was evaluated across the range of 0.01, 0.001, 0.0001, 0.00001, and 0.000001 (Isa et al. 2022), as shown in Table 10.

Table 10 Evaluation results of model optimizer and learning rate on CNN model

Table 10 shows that Adam optimizer with a learning rate of 0.00001 achieved the highest accuracy of 89.3%, while Adagard and SGD achieved 86.8% and 88.8%, respectively. Therefore, a further experiment was conducted using the Adam optimizer with a learning rate of 0.00001, as it was the most suitable setup for the CNN model.

Fully connected layer modification

The modification can be done by varying the size of the fully connected layer from the shallower to deeper structure. The shallower model structure requires more nodes, number of neurons and layers in fully connected layer to process data features as to obtain the similar performance. Compared to the deeper architecture, it requires a smaller number of neurons while costing more computational units (Basha et al. 2020). Thus, the experiment was conducted to test on the different configurations of fully connected layer on CNN model with VGG16 to observe its impact on the model performance.

The model was designed to vary the configuration from 4 to 9 fully connected layers by the bottom layer of the model corresponding to the classification for five classes of ionogram data, as shown in Table 11.

Table 11 Evaluation results of different sizes of fully connected layer on CNN model

After adjusting and tuning model structures, the results showed that the deeper model architectures achieved higher accuracy score (Zhong et al. 2019) while slightly decrease upon the shallower architectures. From the experiment results, with each training iteration of the CNN model, the results demonstrate the variability with different accuracy scores. This means that more experiments with these configurations are needed as these modifications improved the CNN model to acquire a higher accuracy for ionogram classification task as in comparison to the original architecture.

Data augmentation

This method provides various properties to configure the data augmentation process. This augmentation method makes the model to be trained with more diversity of ionogram data with the ability to determine the possibility of applying the filter. This function could be evaluated across the possibility range of 20%, 40%, 60%, 80%, and 100%. The experiment with ImageDataGenerator was conducted through the CNN model with 4 to 9 layers as the fully connected layer optimization process to observe its impact of each image filter precisely as the outcomes of ImageDataGenerator listed in Table 12.

Table 12 Results of the CNN model with ImageDataGenerator with filters on different size of fully connected layer

Table 12 shows that the model with 7, 8, and 9 fully connected layers achieved the same accuracy of 92.0% on fast-NLmeans, bilateral, and median blur filter, respectively with 20% of filter utilization. Then, the models are applied with generalization technique to assess the improvement using dropout technique.

Model regularization technique

After training and testing CNN model with ImageDataGenerator, the model regularization technique called dropout technique is utilized. Dropout is mainly used to improve generalization and prevent overfitting of the model. During training process, dropout will randomly select a subset of neurons of each layer and temporarily drop the operation of those neurons by setting their outputs to zero with a probability of P as the remaining neurons are kept with a probability Q. Dropout helps CNN model to learn more robust and generalizable representations (Mhaskar and Poggio 2016; Khan et al. 2019). The next experiment is to evaluate the impact of dropout technique on the chosen model by testing across the value of 10 to 90% with step size of 10% on the selected CNN model with ImageDataGenerator, as shown in Table 13.

Table 13 Performance of CNN model with dropout on the selected models

After evaluating the model with the dropout technique from Table 13, the results showed that the 7-layer model with 30% of dropout achieved the highest accuracy score at 92.9% along with 98.0 of F1-score for clear ionogram, 85.1 for FSF, 90.7 for RSF, 66.7 for SSF, and 99.2 for unidentified class, as shown in Table 14. Followed by the 8-layer model with 10% of dropout achieved the accuracy of 92.2%. However, the dropout technique with 9-layer model fails to improve the model accuracy resulting in a lower accuracy score of 91.8% which is lower than the 9-layer model without dropout technique.

Table 14 Results of 7-layer CNN model with ImageDataGenerator and dropout (30%)

After the dropout test, the dropout technique improved the performance on the 7- and 8-layer models in this work. On the other hand, the 9-layer model achieved a lower accuracy. This means that the model requires no model regularization as it might lead to significant information loss during training process and hindering learning resulting in lower model accuracy and performance.

As the tuned CNN model has been augmented with fast-NLmeans filter for feature extraction and dropout technique for model regularization, the model performance efficiently improved as it can achieve the highest accuracy score of 92.9% to classify ionogram images. Nonetheless, the model struggled with effectively classifying SSF class. Because it rarely observed at the station causing the extremely low number of samples for training and validation, severely impacting the model performance. Subsequently, FSF class also achieves low accuracy, possibly caused by the same reason as SSF class.

SVM and CNN model performances comparison

After tuning and applying the data preprocessing methods into SVM and CNN model, the CNN model performed better than the SVM model with the model accuracy of 92.9% and 87.0%, respectively, as shown in Table 15. Both achieved high accuracies on classifying clear class and unidentified class because not only these classes have the most simple characteristic and pattern but also have a high number of samples making both models learned and performed very well. CNN model exhibited higher performance for both clear and unidentified class than SVM model.

Table 15 Model performance comparison between SVM and CNN models

Based on the results, the complexity of each spread-F characteristic presents a significant challenge for obtaining the precise classification. The CNN model can classify FSF class better than SVM model by 13.2%. As mentioned, FSF class has many varieties of characteristics which is why it has a lower accuracy compared to other spreading pattern. Conversely, the SVM model classified RSF and SSF class better than CNN model by 3.2% and 7.6%. In the case of RSF class, the CNN might has suffered from overfitting to noisy data leading to false classification. For SSF class, it basically contains a lot of noises making both models cannot learn the patterns and features enough. While SSF class has the fewest training samples among others resulting in low accuracy. However, the CNN model managed to classify all ionogram with better accuracy than SVM model.

The SVM and CNN classification model in this work can classify five types of ionogram images, while the SVM and CNN with VGG16 models by Luwanga et al. (2022) can classify two types of digital ionogram which are ionogram with and without spread-F. In addition, the digital ionogram images recorded by the FMCW-type ionosonde at Cumphon station have lower data resolution and require the manual interpretation introducing the human error in some cases. Then, it is more complicated to compare the data and results between the FMCW-type and digital ionosondes. Moreover, if the alternative ionogram types are employed beyond those used in this research, it becomes imperative to train new models with specific data to those ionograms. The other kinds of ionogram images might have similar characteristics by the trace and spread-F patterns. However, the background noise and other components of plotting characteristic in the ionogram images might be differed due to the different places of record station and the used of different ionosonde type. Nonetheless, utilizing the obtained weights from this research which trained for ionogram classification, can potentially enhance the learning efficiency and classification accuracy of the model on other kinds of ionogram images.

Conclusions

In this work, SVM and CNN models were evaluated as an automatic ionogram image classification model to classify the total of 4,963 ionogram images into five classes: clear (1,320 images), FSF (760 images), RSF (1,620 images), SSF (126 images), and unidentified class (866 images). With the use of image filtering which are image sharpening, image thresholding, median blur, gamma correction, fast-NLmeans, and bilateral filters for the feature extraction. In SVM model, it was firstly tuned using hyperparameter tuning to determine the appropriate parameter for the model then K-means clustering technique was utilized before proceeding image filters. As the result showed that this method improved the classification performance. In CNN model, the model optimizer, learning rate, and the size of dense layer were tuned before utilizing ImageDataGenerator as the data augmentation technique which improved the classification performance satisfactorily. From the analysis, the performances of CNN and SVM are satisfactory in classifying ionogram images. The CNN model achieved the classification accuracy of 92.9% which outperformed SVM model with accuracy of 87.0%. However, SVM outperforms CNN in classifying RSF and SSF classes, especially in SSF class, due to the limited data set of SSF samples for CNN model. This is because finding a hyperplane with a maximum margin prioritizes the characteristics of the data over the quantity of samples, enabling SVM to generalize effectively, especially with limited data sets. On the other hand, CNN outperforms in classifying clear, FSF, and unidentified classes benefiting from its more complex algorithm and larger sample sizes for each class compared to the SSF class. Despite the models and preprocessing technique, data set quantity remains crucial for model performance. Obtaining an ideal data set is challenging due to due to varying spread-F occurrence probabilities, increasing samples will certainly improve the classification performance. From this research, the classification models can be utilized for the FMCW-type ionosondes in the SEALION Project (Maruyama et al. 2007) or other stations employing the same system. In cases where ionograms obtained from other ionosonde types, the obtained weights from this work can be used and applied for further study. In summary, the ionogram image classification model using SVM and CNN model can classify ionogram images within the given data set admirably. While SVM model performs well in certain scenarios and CNN model exhibited a stronger overall performance. In addition, these models will be useful to the automatic scaling of spread-F for the detection and classification of Equatorial Plasma Bubbles (EPB), particularly during the approaching solar maximum. This has practical implications in the field of ionospheric science and applications, enhancing the ability to understand the ionospheric phenomena.