Automatic detection of sudden commencements using neural networks
© The Society of Geomagnetism and Earth, Planetary and Space Sciences (SGEPSS); The Seismological Society of Japan; The Volcanological Society of Japan; The Geodetic Society of Japan; The Japanese Society for Planetary Sciences; TERRAPUB. 2012
Received: 23 August 2012
Accepted: 18 December 2012
Published: 23 August 2013
The aim of this work is to develop an automatic system to detect sudden commencements (SCs). SCs are produced by a sudden increase of solar wind dynamic pressure and are detected simultaneously everywhere on the ground (Araki, 1994). Since 1975, Ebro Observatory is responsible to elaborate the list of SC, following the morphological rules given by Mayaud (1973). Nowadays, this task is still done manually and presents some difficulties; the most worrying one is the decreasing number of observatories who collaborate with this task because most of them opted for the installation of unattended observatories. Hence, the necessity of an alternative method to continue the service becomes a urgency. The automatic method presented in this work is based on neural network analysis. To succeed with neural networks, we did a previous work of characterization and parameterization of SCs by statistical analysis. In this way, we focused on the determination of the appropriate parameters to be used as the inputs of the network which resulted to be: slope, change of magnetic activity and difference of the levels before and after the jump. We worked with X component and also with Y component. An important criteria introduced in this work is the necessary coherence of the results obtained with this new automatic method with those obtained with the manual method and reported in the old list of SC. Finally, the neural network is able to recognize the SC pattern successfully, but now this is achieved in a non-manned way. A robust quasi-real-time detection can be undertaken in the future.
Geomagnetic sudden commencements (SCs) are produced by sudden increases of solar wind dynamic pressure and are observed globally everywhere on the ground. The amplitude and the waveform greatly change depending upon latitude and local time (Araki, 1994). Since the global simultaneous occurrence of the SC phenomena with a clear onset time and the well identified sources are the main characteristics distinguishing SCs from other magnetic field disturbances such as substorms and storms, SCs provide us with a fundamental understanding of a transient response of the magnetosphere and ionosphere to the solar wind.
SCs are phenomena which have been studied from the early years of geomagnetic field. In 1973, P. N. Mayaud (Mayaud, 1973) introduced a lot of changes and improvements in the SC definition and its identification criteria. In fact, his new definition is already used nowadays: “sudden commencements followed by a magnetic storm or by an increase in activity lasting at least one hour”. In this new definition more importance was given to the change of rhythm in the magnetic activity, than to the amplitude of the magnetic storm, and therefore some SCs included in the list are not followed by a magnetic storm.
Since 1975, Ebro Observatory houses the International Service on Rapid Magnetic Variations, responsible for creating and publishing the lists of rapid variations. This task was entrusted by the International Association of Geomagnetism and Aeronomy (IAGA), resolution number 6 of the XVI IUGG General Assembly, Grenoble, 1975 (Curto et al., 2007). One of these rapid variations is the SC phenomenon. Traditionally, many observatories from a worldwide network send possible candidates to be SC to Ebro Observatory. From this list of candidates, Ebro Observatory checks solar wind data and magnetic data from low-latitude stations (Alibag, Honolulu, Kanoya, M’Bour and San Juan) and elaborates a definitive list of SC events. This task presents some difficulties: the different interpretation of the Mayaud’s rules by the collaborating observers and the decreasing number of observatories who collaborate with this task because most of them opted for unattended observatories.
There are some authors interested in an automatic detection of SC (for example, Takano et al., 1999; Shinohara et al., 2005; or Hafez and Ghamry, 2011). Generally, they excluded from their data set the events with small amplitude and they focused their detection system in the amplitude and onset time. With a more realistic approach, we used a complete data set, including events with small amplitude, and we took in consideration the state of the geomagnetic field behind and ahead of SC, and we took as a main constrain to maintain coherence with the older lists.
We proposed a new method of SC detection based on the automatic analysis of digital geomagnetic field data. It detects SC analyzing morphological parameters of the magnetic variation. We used neural networks (NN) for this automatic process because neural networks are able to recognize hidden patters in data sets and NN learns from cases, thus it is a way to incorporate all the knowledge about SC detection accumulated in the old method to be used in the new method.
2. Methodology. Neural Networks
Neural Networks with different architectures had been proved to solve complex problems in the field of geomagnetism or space weather (for example, Lundstedt, 1991; Lundstedt and Wintoft, 1994; Calvo et al., 1995). This is because NN have the property of learning from cases and thus they can tackle difficult problems which cannot be solved with mathematical algorithm based on rules.
2.1 Data set
Geographic and geomagnetic coordinates from the 6 observatories used in this study. In the last column, the supporting organization is indicated.
Name (IAGA code)
Geographic latitude (°)
Geographic longitude (°)
Geomagnetic latitude (°) (11th IGRF, year 2011)
Geomagnetic longitude (°) (11th IGRF, year 2011)
Indian Institute of Geomagnetism
Observatori de l’Ebre
United States Geological Survey
Japan Meteorological Agency
Institut de Recherche pour le Développment
San Juan (SJG)
United States Geological Survey
Methodologically, we adopted a two-step approach. Firstly we trained the neural network, and secondly we put the NN into detection operative stage. For the training stage we needed a data set with an accurate number of events; if the number of events is too small, perhaps this data set was not enough representative of all solutions to the problem, and if the number of events is too large, the NN could be confused or biased (Hagan et al., 1996).
2.2 Neural Networks
There are lots of different architectures of neural networks. We used a feed-forward network because this kind of NN is able to recognize hidden patterns from a complex data set (Hertz et al., 1991; Caudill and Butler, 1992). A feed-forward network is arranged in layers of nodes. The input to the nodes in one layer is the sum of the weighted outputs from the nodes in the previous layer. The output from a node is given by the input to the node and the nodal transfer function. Usually all nodes in one layer are connected to all nodes in the next layer. There are no connections between nodes in the same layer (Rojas, 1996). The neural network is trained by adjusting the weights until the average error on a set of known training examples are minimized. The training algorithm is a modified form of gradient descent called error back-propagation (Rumelhart et al., 1986; Vogl et al., 1988; Hagan et al., 1996).
The network architecture has been found through a standard training and validation procedure using two separate data sets. We used data set from year 2000 as a validation data set, and data set from 2001 to 2007 as a training data set. We took in consideration the difference between period 2000–2002 with an important number of SC occurrence, and the period 2003–2007 with a minor SC events. A large number of networks, with different architectures, were trained on data from the training set. Using validation data set, the network with the smallest validation error was selected as the optimal network. Our final architecture was a 3 layer network: an input layer, a hidden layer and an output layer, with 5, 10 and 2 neurons in each layer respectively.
2.3 Training the neural network
Training a network means finding a set of weights that minimizes the average error on the training set. The training is done iteratively by showing the network known input-output pairs, calculating the error and updating the weights accordingly. The weights are not updated after every input-output pair but after a number of examples, an epoch.
On the other hand, we are interested in creating an automatic method to detect SC but its results should maintain coherence with those identified by the traditional method used to elaborate the old list, according with Mayaud’s rules. By working with data from only our observatory, Ebro, we benefit from the fact that we also control the data acquisition and their availability in real-time, but our network system presents a number of false positives which are not appropriate if what we want is to have definitive data coherent with the old list. In order to avoid this inconvenience and taking advantage of the facilities from INTERMAGNET network we took data from 5 low-latitude observatories, Alibag, Honolulu, Kanoya, M’Bour and San Juan, where the SC present clearer features as a big step function and no primary reverse impulses (Araki, 1994). We repeated the whole process over this new data set: running the candidate selection process, training again the network with this new candidate data set and performing the same test. At this point, we imposed another criterion for the selection: the simultaneity in the movement at least in 3 of the 5 observatories. With these constrains, our network had higher rate of goal, as expected. Now, our network presents 38 events correctly detected; 3 events non-detected; and 5 false positive. Right panel in Fig. 5 compares the results of automatic method (from our neural network) and the results from the traditional method (old lists).
This new method even obtains satisfactory results when data from few observatories are available. But, if we work with enough observatories, we also can ensure that the produced lists will be coherent with the traditional lists. This was one of our conditions to validate the capacity of the network to solve properly our problem. Using this new method we could elaborate SC lists much quicker than with the traditional method. In this sense they could be delivered to users without delay. SC events and also SC lists are used and will be used as indicators of one of the more interesting events for Space Weather researchers.
We could expect that the major number of false positives would occur during storm time, but the network is able to reject these candidates, due to the fact that it takes the change of rhythm as a variable. We want to remark that the change of rhythm is fundamental to make an accurate classification of SC events.
The number of SC detected correctly using only data from one mid-latitude station, EBR, indicates that the NN is able to recognize SC pattern and also is able to reject other phenomena with similar appearance. However, the number of false positives using only data from EBR is not appropriate for our purpose, and then we need to add data from other observatories, as when we work using manual detection.
Of course, the NN works better when the SC has a waveform more like stepwise function and has high amplitude. But, in the case of SC with small amplitude and large rise time, to decide if this event is or not a SC is not an easy task even for a manual detection, and then we have to check additional data from other observatories and also from satellites.
As a future works we are planning an upgrade of the neural network with the aim to work with 1 second data. According IAGA recommendations, we are now sampling our magnetometers to obtain 1 second data.
In the future, we plan to make quasi-real time detection.
The aim of this work was to produce an automatic method of SC detection. Ebro Observatory is responsible to elaborate the official lists of SC events. Until now this work has still been done manually by using provisional lists from reports of possible candidates from other observatories which collaborate on this task. The whole process, from receiving the reports to checking every candidate manually, takes too much time. Moreover, the number of observatories which collaborate with the service is decreasing at the same rhythm as the number of people who manually take care of the magnetic data. Consequently, as the future of the service was in danger. We tried to devise an automatic method to speed up this process. Neural networks are helpful to solve our problems because they are able to find the hidden patterns of a data set. Moreover, NN present the capacity to learn from cases, and they have an adaptive learning useful to afford the detection of natural phenomena.
In this work, we have shown the ability of neural networks to detect SC events. Almost a complete success in identification is achieved when we use additionally data from low latitude observatories and imposing their simultaneity. Even with one station the method has acceptable results. In consequence, a robust quasi-real-time detection can be undertaken in the future because it will work even reasonably well under very adverse conditions.
This research has been supported by Spanish projects CTM2008-03033-E and CTM2009-13843-C02-01 of MICINN. The results presented in this paper rely on data collected at magnetic observatories. We thank the national institutes that support them and INTERMAGNET for promoting high standards of magnetic observatory practice. (www.intermagnet.org).
- Araki, T., A physical model of the geomagnetic sudden commencement, Geophys. Monogr., 81, 183–200, 1994.Google Scholar
- Calvo, R. A., H. A. Ceccatto, and R. D. Piacentini, Neural network prediction of solar activity, Astrophys. J., 444, 916–921, 1995.View ArticleGoogle Scholar
- Caudill, M. and C. Butler, Understanding Neural Networks: Computer Explorations, vols. 1 and 2, The MIT Press, Cambridge, MA, 1992.Google Scholar
- Curto, J. J., J. O. Cardús, L. F. Alberca, and E. Blanch, Milestones of the IAGA international service of rapid magnetic variations and its contribution to geomagnetic field knowledge, Earth Planets Space, 59, 463–471, 2007.View ArticleGoogle Scholar
- Hafez, A. G. and E. Ghamry, Automatic detection of geomagnetic sudden commencements via time-frequency clusters, Adv. Space Res., 48, 1537–1544, 2011.View ArticleGoogle Scholar
- Hagan, M. T., H. B. Demuth, and M. H. Beale, Neural Network Design, Boston, PWS Publishing, MA, 1996.Google Scholar
- Hertz, J. A., A. Krogh, and R. G. Palmer, Introduction to the Theory of Neural Computation, lecture notes vol. 1, Santa Fe Institute Studies in the science of complexity, 327 pp., Addison-Wesley, Redwood City, CA 94065, 1991.Google Scholar
- Joselyn, J. A., The automatic detection of geomagnetic-storm sudden commencements, Adv. Space Res., 5, 193–197, 1985.View ArticleGoogle Scholar
- Lundstedt, H., Neural network predictions of geomagnetic activity, in IAGA Programs and Abstracts, XX General Assembly, IUGG, Vienna, 1991.Google Scholar
- Lundstedt, H. and P. Wintoft, Prediction of geomagnetic storms from solar wind data with the use of a neural network, Ann. Geophys., 12, 19–24, 1994.View ArticleGoogle Scholar
- Mayaud, P. N., A hundred years series of Geomagnetic data 1868–1967: Indices aa and storm sudden commencements, IAGA Bull., 33, 1973.Google Scholar
- Rojas, R., Neural Networks—A Systematic Introcuction, chapter 7 The backpropagation algorithm, Springer-Verlag, Berlin, 1996.Google Scholar
- Rumelhart, D. E., G. Hinton, and R. Williams, Learning representations by back-propagation errors, Nature, 323, 533–536, 1986.View ArticleGoogle Scholar
- Shinohara, M., T. Kikuchi, and K. Nozaki, Automatic realtime detection of sudden commencements of geomagnetic storms, J. NICT, 52(3/4), 197–205, 2005.Google Scholar
- Takano, S., T. Minamoto, H. Arimura, K. Niijima, T. Iyemori, and T. Araki, Automatic detection of geomagnetic sudden commencements using lifting wavelet filters, Lecture Notes in Computer Science, vol. 1721, Springer, Berlin, Heidelberg, 1999.Google Scholar
- Vogl, T. P., J. K. Mangis, A. K. Rigler, W. T. Zink, and D. L. Alkon, Accelerating the convergence of the backpropagation method, Biological Cybernetics, 59, 256–264, 1988.View ArticleGoogle Scholar