Automatic detection of sudden commencements using neural networks

Abstract

The aim of this work is to develop an automatic system to detect sudden commencements (SCs). SCs are produced by a sudden increase of solar wind dynamic pressure and are detected simultaneously everywhere on the ground (Araki, 1994). Since 1975, Ebro Observatory is responsible to elaborate the list of SC, following the morphological rules given by Mayaud (1973). Nowadays, this task is still done manually and presents some difficulties; the most worrying one is the decreasing number of observatories who collaborate with this task because most of them opted for the installation of unattended observatories. Hence, the necessity of an alternative method to continue the service becomes a urgency. The automatic method presented in this work is based on neural network analysis. To succeed with neural networks, we did a previous work of characterization and parameterization of SCs by statistical analysis. In this way, we focused on the determination of the appropriate parameters to be used as the inputs of the network which resulted to be: slope, change of magnetic activity and difference of the levels before and after the jump. We worked with X component and also with Y component. An important criteria introduced in this work is the necessary coherence of the results obtained with this new automatic method with those obtained with the manual method and reported in the old list of SC. Finally, the neural network is able to recognize the SC pattern successfully, but now this is achieved in a non-manned way. A robust quasi-real-time detection can be undertaken in the future.

1. Introduction

Geomagnetic sudden commencements (SCs) are produced by sudden increases of solar wind dynamic pressure and are observed globally everywhere on the ground. The amplitude and the waveform greatly change depending upon latitude and local time (Araki, 1994). Since the global simultaneous occurrence of the SC phenomena with a clear onset time and the well identified sources are the main characteristics distinguishing SCs from other magnetic field disturbances such as substorms and storms, SCs provide us with a fundamental understanding of a transient response of the magnetosphere and ionosphere to the solar wind.

SCs are phenomena which have been studied from the early years of geomagnetic field. In 1973, P. N. Mayaud (Mayaud, 1973) introduced a lot of changes and improvements in the SC definition and its identification criteria. In fact, his new definition is already used nowadays: “sudden commencements followed by a magnetic storm or by an increase in activity lasting at least one hour”. In this new definition more importance was given to the change of rhythm in the magnetic activity, than to the amplitude of the magnetic storm, and therefore some SCs included in the list are not followed by a magnetic storm.

Since 1975, Ebro Observatory houses the International Service on Rapid Magnetic Variations, responsible for creating and publishing the lists of rapid variations. This task was entrusted by the International Association of Geomagnetism and Aeronomy (IAGA), resolution number 6 of the XVI IUGG General Assembly, Grenoble, 1975 (Curto et al., 2007). One of these rapid variations is the SC phenomenon. Traditionally, many observatories from a worldwide network send possible candidates to be SC to Ebro Observatory. From this list of candidates, Ebro Observatory checks solar wind data and magnetic data from low-latitude stations (Alibag, Honolulu, Kanoya, M’Bour and San Juan) and elaborates a definitive list of SC events. This task presents some difficulties: the different interpretation of the Mayaud’s rules by the collaborating observers and the decreasing number of observatories who collaborate with this task because most of them opted for unattended observatories.

After IAGA meeting in Sopron (Hungry) in 2009, and taking in consideration some previous works as (Joselyn, 1985), and (Curto et al., 2007), more objective criteria to select events were introduced, e.g., at least a slope of 3 nT/min at minimum in 3 of 5 low-latitude stations. These criteria were tested in advance in order the new lists be coherent with the oldest lists (Mayaud, 1973). Figure 1 shows an example of a rejected candidate, and it shows that this is not an easy task because there are rejected candidates which at first glance are very similar to a true SC.

Fig. 1.

Example of one candidate rejected because there were only two (of five) low observatories presenting a slope greater than 3 nT/min: ABG 2.7 nT/min; HON 2.4 nT/min; KNY 3.3 nT/min; MBO 3.3 nT/min; SJG 1.6 nT/min.

There are some authors interested in an automatic detection of SC (for example, Takano et al., 1999; Shinohara et al., 2005; or Hafez and Ghamry, 2011). Generally, they excluded from their data set the events with small amplitude and they focused their detection system in the amplitude and onset time. With a more realistic approach, we used a complete data set, including events with small amplitude, and we took in consideration the state of the geomagnetic field behind and ahead of SC, and we took as a main constrain to maintain coherence with the older lists.

We proposed a new method of SC detection based on the automatic analysis of digital geomagnetic field data. It detects SC analyzing morphological parameters of the magnetic variation. We used neural networks (NN) for this automatic process because neural networks are able to recognize hidden patters in data sets and NN learns from cases, thus it is a way to incorporate all the knowledge about SC detection accumulated in the old method to be used in the new method.

2. Methodology. Neural Networks

Neural Networks with different architectures had been proved to solve complex problems in the field of geomagnetism or space weather (for example, Lundstedt, 1991; Lundstedt and Wintoft, 1994; Calvo et al., 1995). This is because NN have the property of learning from cases and thus they can tackle difficult problems which cannot be solved with mathematical algorithm based on rules.

2.1 Data set

We used 1 minute data of the magnetic elements from 2000 to 2007. In our first approach, we worked with data from Ebro Observatory (EBR), and in our second approach with data from 5 Low-latitude Stations (Honolulu, San Juan, M’Bour, Alibag and Kanoya) provided by INTERMAGNET network, Table 1 shows their geographic and geomagnetic coordinates and also the institutions that support the 6 observatories used in this study. As we commented above, Ebro Observatory is responsible to elaborate the list of SC, and then we are also interested to know possibilities to be self-sufficient in a future service of real time detection regarding the worst case with no availability of external data.

Table 1. Geographic and geomagnetic coordinates from the 6 observatories used in this study. In the last column, the supporting organization is indicated.

Methodologically, we adopted a two-step approach. Firstly we trained the neural network, and secondly we put the NN into detection operative stage. For the training stage we needed a data set with an accurate number of events; if the number of events is too small, perhaps this data set was not enough representative of all solutions to the problem, and if the number of events is too large, the NN could be confused or biased (Hagan et al., 1996).

Our first step to prepare data set for our training stage was to make a parameterization. We used 1 minute data and we selected possible candidates to be SC. Before that, we needed a collection of characteristics of the events and a collection of threshold values to decide which event could be a potential SC, that is, a candidate. With this aim, we analyzed principal characteristics of SC: amplitude, change of rhythm and slope. SC is a global phenomenon but presents an important variability depending on latitude and local time. We calculated amplitude, change of rhythm and slope for the 188 SC events from the official list for this period, Fig. 2, as recorded at Ebro Observatory. Instead of amplitude we worked with change of levels with the aim to avoid some uncertainty introduced by possible preliminary reverse impulses (PRI). The change of levels was calculated with the difference ahead and behind SC onset time, magnetic field data averaged over 10 minutes. Change of rhythm was calculated with the difference of the standard deviation of the derivate ahead and behind the SC, averaged 60 minutes. There are no clear thresholds for change of level either for change of rhythm; so we will focus on a threshold on the slope. If we take the recommended value for low-latitude stations, 3 nT/min, we will exclude 13% of events. As the amplitude decreases with latitude, for a mid-latitude observatory as Ebro, it is reasonable to take a threshold value of 2 nT/min. With this threshold value, we only exclude the 1.6% of events. If we took a minor threshold the number of candidates would increase too much and the posterior training process would have been not effective. Then, our initial threshold is based on the slope, with this selection candidates process we have a data set with 586 candidates which contains 185 events of 188 listed in the official list of SC. Figure 3 shows an example of 3 candidates included in the selection; in this case, the first candidate is a very clear SC and the another two candidates are not so clear because they appear during a magnetic storm time.

Fig. 2.

Top panel, amplitude of all SC listed measured at Ebro. Mid panel change of rhythm of all SC listed measured at Ebro. Bottom panel slope of all SC listed measured at Ebro. With a thin line is showed typical threshold value for low-latitude stations, 3 nT/min, we found 13% of cases under this threshold. However, there are only 1.6% of cases under the threshold of 2 nT/min.

Fig. 3.

Example of 3 potential events to be a SC after candidate selection.

2.2 Neural Networks

There are lots of different architectures of neural networks. We used a feed-forward network because this kind of NN is able to recognize hidden patterns from a complex data set (Hertz et al., 1991; Caudill and Butler, 1992). A feed-forward network is arranged in layers of nodes. The input to the nodes in one layer is the sum of the weighted outputs from the nodes in the previous layer. The output from a node is given by the input to the node and the nodal transfer function. Usually all nodes in one layer are connected to all nodes in the next layer. There are no connections between nodes in the same layer (Rojas, 1996). The neural network is trained by adjusting the weights until the average error on a set of known training examples are minimized. The training algorithm is a modified form of gradient descent called error back-propagation (Rumelhart et al., 1986; Vogl et al., 1988; Hagan et al., 1996).

The network architecture has been found through a standard training and validation procedure using two separate data sets. We used data set from year 2000 as a validation data set, and data set from 2001 to 2007 as a training data set. We took in consideration the difference between period 2000–2002 with an important number of SC occurrence, and the period 2003–2007 with a minor SC events. A large number of networks, with different architectures, were trained on data from the training set. Using validation data set, the network with the smallest validation error was selected as the optimal network. Our final architecture was a 3 layer network: an input layer, a hidden layer and an output layer, with 5, 10 and 2 neurons in each layer respectively.

2.3 Training the neural network

Training a network means finding a set of weights that minimizes the average error on the training set. The training is done iteratively by showing the network known input-output pairs, calculating the error and updating the weights accordingly. The weights are not updated after every input-output pair but after a number of examples, an epoch.

In relation to our input-output pairs, as inputs we used SC parameterization, slope, change level in X and Y components (both in nT) and change of rhythm; and as output we had a digit with value “1” (hit) if the candidate was in the official SC list and “0” (fail) if the candidate was not in the list. Figure 4 shows a schematic representation of our network. After the training process, to check the capability of the network, we only used, as input of the network, a data set which had been excluded in the training process. Then, we compared its response (output) with the official list.

Fig. 4.

Schematic representation of our network. As inputs, we used Date, Slope, Change level in components X and Y, and Rhythm of each event. The output assets the probability of this event to be a true SC. The target is that this qualification coincides with the manual list.

3. Results

Once the training process was finished, we went on with the operative detection process. Here we present the results of our validation test. Using only the Ebro magnetic data corresponding to the year 2000 as a test, and after thresholding we obtained 139 candidates to be an SC. For this period and according to the official list only 41 of them were true SC. Our automatic system based on a network produced a list of SC with the following results: 35 events were detected correctly, 6 events were non-detected, and 14 came out to be false positives. Left panel in Fig. 5, compare the two different lists: the number of cases in list obtained with the traditional and manual method appears as the area inside the solid line, and that corresponding to the new list—produced by the automatic system based on a neural network—appears as that area inside the dashed line.

Fig. 5.

Comparison between manual list (area in the solid line) and neural network list (area in the dashed line). The area non-covered by that associated to the neural network list corresponds to non-detected events, and the area that exceeds manual list corresponds to false positive events. Left panel shows results using only data from Ebro Observatory, the areas that correspond to non-detected and false positive are significant. Right panel, shows results when using data from 5 low-latitude stations, in this case, the areas for non-detected and false positive are tiny.

Figure 6 shows an example of false positive and Fig. 7 shows an example of a non-detected event. The high number of events detected is remarkable if we take into consideration that we only used data from one station, and this station was located in a mid-latitude area.

Fig. 6.

Example of a false positive. At Ebro it looks like a SC with a large slope, but in the 5 low-latitude stations, its slope is smaller than 3 nT/min.

Fig. 7.

Example of a non-detected SC. This event is in the manual list of SC, but at Ebro it shows a very small slope.

On the other hand, we are interested in creating an automatic method to detect SC but its results should maintain coherence with those identified by the traditional method used to elaborate the old list, according with Mayaud’s rules. By working with data from only our observatory, Ebro, we benefit from the fact that we also control the data acquisition and their availability in real-time, but our network system presents a number of false positives which are not appropriate if what we want is to have definitive data coherent with the old list. In order to avoid this inconvenience and taking advantage of the facilities from INTERMAGNET network we took data from 5 low-latitude observatories, Alibag, Honolulu, Kanoya, M’Bour and San Juan, where the SC present clearer features as a big step function and no primary reverse impulses (Araki, 1994). We repeated the whole process over this new data set: running the candidate selection process, training again the network with this new candidate data set and performing the same test. At this point, we imposed another criterion for the selection: the simultaneity in the movement at least in 3 of the 5 observatories. With these constrains, our network had higher rate of goal, as expected. Now, our network presents 38 events correctly detected; 3 events non-detected; and 5 false positive. Right panel in Fig. 5 compares the results of automatic method (from our neural network) and the results from the traditional method (old lists).

4. Discussion

This new method even obtains satisfactory results when data from few observatories are available. But, if we work with enough observatories, we also can ensure that the produced lists will be coherent with the traditional lists. This was one of our conditions to validate the capacity of the network to solve properly our problem. Using this new method we could elaborate SC lists much quicker than with the traditional method. In this sense they could be delivered to users without delay. SC events and also SC lists are used and will be used as indicators of one of the more interesting events for Space Weather researchers.

We could expect that the major number of false positives would occur during storm time, but the network is able to reject these candidates, due to the fact that it takes the change of rhythm as a variable. We want to remark that the change of rhythm is fundamental to make an accurate classification of SC events.

The number of SC detected correctly using only data from one mid-latitude station, EBR, indicates that the NN is able to recognize SC pattern and also is able to reject other phenomena with similar appearance. However, the number of false positives using only data from EBR is not appropriate for our purpose, and then we need to add data from other observatories, as when we work using manual detection.

Of course, the NN works better when the SC has a waveform more like stepwise function and has high amplitude. But, in the case of SC with small amplitude and large rise time, to decide if this event is or not a SC is not an easy task even for a manual detection, and then we have to check additional data from other observatories and also from satellites.

As a future works we are planning an upgrade of the neural network with the aim to work with 1 second data. According IAGA recommendations, we are now sampling our magnetometers to obtain 1 second data.

In the future, we plan to make quasi-real time detection.

5. Conclusions

The aim of this work was to produce an automatic method of SC detection. Ebro Observatory is responsible to elaborate the official lists of SC events. Until now this work has still been done manually by using provisional lists from reports of possible candidates from other observatories which collaborate on this task. The whole process, from receiving the reports to checking every candidate manually, takes too much time. Moreover, the number of observatories which collaborate with the service is decreasing at the same rhythm as the number of people who manually take care of the magnetic data. Consequently, as the future of the service was in danger. We tried to devise an automatic method to speed up this process. Neural networks are helpful to solve our problems because they are able to find the hidden patterns of a data set. Moreover, NN present the capacity to learn from cases, and they have an adaptive learning useful to afford the detection of natural phenomena.

In this work, we have shown the ability of neural networks to detect SC events. Almost a complete success in identification is achieved when we use additionally data from low latitude observatories and imposing their simultaneity. Even with one station the method has acceptable results. In consequence, a robust quasi-real-time detection can be undertaken in the future because it will work even reasonably well under very adverse conditions.