Automated detection of geomagnetic storms with heightened risk of GIC

Abstract

Automated detection of geomagnetic storms is of growing importance to operators of technical infrastructure (e.g., power grids, satellites), which is susceptible to damage caused by the consequences of geomagnetic storms. In this study, we compare three methods for automated geomagnetic storm detection: a method analyzing the first derivative of the geomagnetic variations, another looking at the Akaike information criterion, and a third using multi-resolution analysis of the maximal overlap discrete wavelet transform of the variations. These detection methods are used in combination with an algorithm for the detection of coronal mass ejection shock fronts in ACE solar wind data prior to the storm arrival on Earth as an additional constraint for possible storm detection. The maximal overlap discrete wavelet transform is found to be the most accurate of the detection methods. The final storm detection software, implementing analysis of both satellite solar wind and geomagnetic ground data, detects 14 of 15 more powerful geomagnetic storms over a period of 2 years.

Background

Geomagnetic storms are rapid variations in the Earth’s geomagnetic field. They are the result of clouds of charged particles from the sun interacting with our magnetosphere. These storms are commonly split into three phases (Tsurutani and Gonzalez 1997): an initial phase with a sudden increase in field strength (sudden storm commencement or SSC), the main phase with a decrease in field strength and periods of rapid field variations, followed by a recovery phase as the geomagnetic field gradually returns to its normal strength. The rapid variations that occur during the initial and main phases of storms can have a great impact on our modern infrastructure through currents induced in power lines and pipelines. These geomagnetically induced currents (GIC) have caused blackouts in the past, an example being the complete collapse of the Hydro-Quebec power grid in March 1989 (Allen et al. 1989) during a particularly powerful storm. The quick and automated detection of SSCs is therefore of great interest for those involved in monitoring and maintaining power grid infrastructure (see, e.g., Joselyn 1985; Molinski 2002). The aim of this study is to determine how to best detect SSCs.

The SSC most closely resembles a step function in the horizontal component of the geomagnetic field H at the time of storm initiation, but the exact shape and size of the SSC depend strongly on the location and local time at the place of measurement (Araki 1977). A solar wind shock front arriving at the Earth’s magnetic field exerts a rapid increase in ram pressure, which leads to a compression of the magnetic field and a short-lived increase in field strength in the horizontal component (see, e.g., Akasofu 1981; Araki 1994 for details).

SSCs have historically been picked out manually by an expert, but this is time-consuming and can only occur in hindsight, which does not allow for up-to-date warnings. This problem has been tackled many times before (Hafez and Ghamry 2011; Joselyn 1985; Mendes et al. 2005; Shinohara et al. 2005; Takano et al. 1999), most recently and extensively by Hafez et al. (2013a), where methods were developed for automated SSC detection in real-time geomagnetic observatory data with very promising results. These methods naturally have their limitations; the earlier studies used 1-min data, which do not have high enough resolution for very detailed or rapid detection. Furthermore, all attempts, besides those in Hafez et al. (2013a), cited difficulty in separating SSC signals from other quickly varying geomagnetic signals, and all had unsatisfactorily high levels of false positives. These methods were based on analysis of geomagnetic data alone, which limits knowledge of the geomagnetic conditions on the Earth’s surface. We wish to address these problems with a new method, which combines the use of ground geomagnetic observatory data with solar wind data measured at the Advanced Composition Explorer (ACE, NASA) satellite, 1.5 million km from Earth at Lagrange point 1 on the Sun–Earth line. Similar systems have been developed in individual observatories. However, at the time of writing, no paper has been written on such a finished combined system.

In this study, we compare multiple methods of automated SSC detection and suggest a new system that utilizes both ground-based geomagnetic measurements and satellite observations. Section “Approach” presents the methods compared and the approach taken in our attempt to automatically detect geomagnetic storm commencements. In section “Results” we will then present the results from our comparisons and propose a system that can automatically detect geomagnetic storms in near real-time using data from a single observatory in combination with satellite data. The last section “Discussion” discusses and summarizes the results.

Approach

In order to develop and test an automated geomagnetic storm detector, we first need a definition of what exactly constitutes a geomagnetic storm; this topic is discussed in the first section. With a working definition of the signals we intend to detect, we then give an overview of which methods would be best suited as automated storm onset time pickers for SSCs and CME shocks. Our approach in comparing the methods is described along with the data sets used.

Defining geomagnetic storms and stormy conditions

In order to test and validate the methods used in this study, we picked out storms from recent years (solar cycle 24) and used data measured at the Conrad Observatory during this time. This timespan covered February 2013 through April 2015, which included the solar activity peak reached in early 2014. Although this period was quieter compared to preceding solar cycles, there were still a number of geomagnetic storms.

A database was built from the preliminary list of SSCs within the International Service of Geomagnetic Indices (ISGI) monthly bulletins (http://isgi.unistra.fr/monthly_bulletin.php), which came to a total of 57 SSCs. We did not attempt to test the detection methods on all SSCs, many of which resulted in weak geomagnetic storms, but instead picked storms according to certain criteria. Our main priority is to detect storms that are likely to lead to notable consequences on Earth, such as GIC. In this study, we consider only storms that fulfill the following criteria [similar to those suggested by Joselyn and Tsurutani (1990) and Curto et al. (2007) in their quantitative definitions of SSCs]:

• 1. The rapid increase in the horizontal component of the geomagnetic field at storm commencement must exceed a value of $\Delta H \ge 10$ nT within a 3-min period after initiation.

• 2. The geomagnetic storm in the immediate 24 h following an SSC must have a value of $k_p \ge 5$.

While not directly linked to the strength of storms or the likelihood of resulting in GIC, we define a minimum SSC amplitude for purposes of identification and to set a minimum limit for computational purposes.

The planetary k _p index (Bartels et al. 1939; Rostoker 1972) is preferentially chosen over other geomagnetic indices, as it is a good representation of the agitated state of the magnetic field at mid-latitudes, which is our area of interest. In their study of the top 40 GIC days in Europe, Juusola et al. (2015) noted that the standard index commonly used to determine the strength of storms, the Dst index, was not a comprehensive indicator for days with large GIC, but the k _p index was.

After application of these two criteria, the list of 57 SSCs was reduced to a total of 15. These are listed (alongside the analysis results) in Table 1. Of the storms that fulfill these criteria, it was observed that, with a few exceptions, they all possessed the following solar wind properties preceding the storm on Earth:

• 1. The arrival of a cloud in the solar wind with wind speed v _sw > 400 km/s at the ACE satellite (at the L1 point, or Sun–Earth Lagrange point 1) shortly precedes an SSC. This arrival appears as a discontinuity in the wind speed, suggesting an arriving coronal mass ejection (CME) shock front (as detailed in, e.g., Huttunen et al. 2002). The amount of time the shock at the ACE satellite precedes the SSC on Earth varies between 0.5 and 1.5 h and depends on the solar wind speed at the shock front.

• 2. The proton flux j _sw in the range 47–68 keV exceeds 50,000 particles/(cm⁻² s⁻¹ ster⁻¹ MeV⁻¹) at the time of shock arrival at ACE.

The physics behind geomagnetic storms and the correlation between the solar wind properties and storms has been studied extensively [for a summary see Tsurutani and Gonzalez (1997) and references therein]. In addition to the methods used to detect SSCs in geomagnetic data, we also describe the method used to detect shock arrival fronts and determine stormy conditions using ACE satellite data. There have been many past papers on the geoeffectiveness of various solar wind parameters (Gonzalez and Tsurutani 1987; Richardson and Cane 2011; Tsurutani and Gonzalez 1997) alongside developments in predicting geomagnetic conditions on Earth using solar wind data (Boberg et al. 2000; Lotz and Cilliers 2015; Pulkkinen et al. 2010; Wintoft 2005). In similar fashion to this project, a CME shock detector was developed by the British Geological Survey (BGS) (Thomson et al. 2005) to work in conjunction with a GIC model. There have been other studies on developing warning systems based on solar wind data, a recent example being Kubo et al. (2015), in which a warning system for radiation exposure in aviation was proposed.

Table 1 List of storms

For the purposes of our storm detector, storms are assumed to be triggered by the arrival of a CME shock on Earth. This assumption is supported by the findings of Borovsky and Denton (2006), namely that all storms with GIC events on record were connected to CME-driven storms. Many solar wind parameters were evaluated in our approach of finding the parameters most indicative of arriving storms. Geomagnetic storms and their connection to solar wind have been studied in detail in the past, and it is commonly accepted that the geoeffectiveness of a storm depends on the interplanetary magnetic field B _z (particularly a southward directed field) value (Hirshberg and Colburn 1969). The amplitude of the SSC, on the other hand, is largely dependent on the solar wind ram pressure (Gonzalez et al. 1989), a product of the solar wind density and velocity, given by $P = \frac{1}{2} \rho v^2$. We evaluated the correlation of all solar wind parameters with the selection of 15 storms and looked for behavior unique to these storms. This led to the conclusion that v _sw and j _sw were the most useful parameters, with the highest correlation with k _p and storm events, i.e., the patterns seen in the solar wind in these two parameters prior to a storm would not be seen in other circumstances. The magnetic field component B _z would usually only come into play in the later development of the storm, and the magnitude and direction of the magnetic cloud were generally not visible at the shock front. Conversely, an evaluation of the ram pressure, which we would expect to be a useful parameter, as it is linked to the CME shock front and to SSC generation, could not be implemented successfully. This is mainly linked to ACE data reliability; the particle density data were often simply missing or so noisy that the signal of a shock front arrival would be less clear in the ram pressure than in the solar wind measurements.

These characteristics were built into the evaluation method as the definition for “stormy conditions,” which is a measure of the likelihood of conditions in the solar wind leading to a geomagnetic storm on Earth. If a sudden rise in solar wind speed and a sufficient increase in the proton flux are observed in the solar wind data, the event is ascribed a probability factor depending on the exact behavior of the solar wind and how well it matches the ideal CME behavior stated above. An example of a typical geomagnetic storm preceded by a CME shock is shown in Fig. 1. In this plot, the CME shock is clearly distinguishable from the normal solar wind speed variation, even though the data become much noisier after the shock front has passed. After a period of 45 min, the time it takes for the cloud to travel the distance between the ACE satellite and Earth, the storm begins on Earth with an SSC, seen as a sudden increase in the H-component.

Fig. 1

A typical geomagnetic storm onset as a CME shock front seen in the solar wind speed data (thin gray line, measured by ACE at a distance of 1.5 × 10⁶ km) and an SSC on Earth (thick blue line, measured at the Conrad Observatory). Both the CME shock and SSC are easily identifiable

Automated detection of SSCs

With a working definition of geomagnetic storms and the signals we are trying to detect, we now describe the computational approach. As the SSC is the clearest sign representing the start of a geomagnetic storm, we base our storm detector on the detection of SSCs.

In order to find the best approach in detecting SSCs, we compare three methods. This approach was chosen because of the variety of methods applied in the literature. While the DWT methods had been applied with good results (Hafez et al. 2013a), we had also had reasonable success in the Conrad Observatory applying a hitherto untested method (AIC) in smaller studies in the past. To ensure that these methods do in fact offer a benefit over a more basic approach, we compared them to an analysis using the first derivative. These methods are described below.

First derivative (FD) method

The first time derivative of the magnetic time series is the simplest and most intuitive to understand of the methods we tested. The first derivative should highlight areas of increased slope, for which there are few other natural causes beyond geomagnetic storms and SSCs, as the rapid increase at the time of an SSC will show up as a peak in the first derivative. A storm is detected by this method when a peak in the first derivative exceeds the limit a _FD for a period of time exceeding the minimum duration p _FD. An example is shown in Fig. 2.

Fig. 2

Squared first time derivative (FD) of the time series. The form is similar to the MODWT of the function, but the signal is noisier. This could sometimes make it more difficult to clearly define the edges of a peak. The high-frequency part of the SSC is marked on either side by the two dashed gray lines

Maximal overlap discrete wavelet transform (DWT)

A wavelet analysis such as that used in Hafez et al. (2013a), which essentially separates the signal into frequency bands, has been shown to be a useful approach in detecting SSCs. The discrete wavelet transform (DWT) of a set of data allows us to look in particular at different scales of frequencies within the data using multi-resolution analysis (MRA). In this study, we apply the Maximal Overlap DWT, also known as the undecimated DWT and the stationary DWT, among other names. The benefit of using MODWT over DWT is that it is insensitive to the choice of start time in the time series. The application of MRA of MODWT to a time series results in two types of functions, with their corresponding coefficients: the approximation of the signal followed by varying degrees of details, which characterize the higher frequency content of the signal. By considering individual details, we can effectively look at the specific high frequency content of a signal as if it had been filtered out, thereby homing in on signals and frequencies unique to SSCs. This method has been applied extensively before in varying conditions; Hafez et al. (2012) used the MODWT to detect SSCs in 3-s resolution data, while Hafez et al. (2013a) used the DWT on 1-s resolution data and Hafez et al. (2013b) used the MODWT on 1-s resolution data. Hafez and Ghamry (2013) used MODWT on 1-min resolution data, and Ghamry et al. (2013) applied the DWT to 3-s resolution data.

Our method of using the MODWT as an analysis tool follows closely the method used in Hafez et al. (2013b) and Hafez and Ghamry (2013). The authors used a Haar wavelet in the latter analysis and looked at the first detail in particular, and we have adopted the same approach. A plot of the first two details of the MODWT using a Haar wavelet is plotted in Fig. 3. In their approach, false triggers were eliminated as best as possible using data from multiple observatories; a trigger from the algorithm would only be considered a correct detection if it was also present at the other observatories, which requires real-time access to these data. In this study, we use data from only one observatory.

Fig. 3

MRA of the squared MODWT details with a Haar wavelet applied to geomagnetic data. Two details are visible; with each higher number of detail, higher-frequency parts of the signal are reproduced. The high-frequency part of the SSC is marked on either side by the two dashed gray lines

A similar method for picking peaks as used in the first derivative method is applied here, where a detection is defined as a peak in the MODWT first detail that exceeds a minimum value a _DWT for a period of time exceeding the minimum duration p _DWT.

Akaike information theoretic criterion (AIC)

The Akaike information criterion (AIC) described in Akaike (1974) has been used successfully in seismology in the past (Sleeman and Eck 1999; St-Onge et al. 2011) for automated detection of P-wave onsets and may be able to accurately pick out geomagnetic storm commencements, and has been used at the Conrad Observatory with reasonable success in the past. The AIC function essentially compares the information before and after a point in a time series (x), and a change in information between these two parts results in a peak in the first derivative of the function over time. The AIC is determined using the following formula on the time series:

$$\begin{aligned} \hbox {AIC}(x) &=(x - M) \cdot \log \left( \sigma _{1,\mathrm{max}}^{2}\right) \\&\quad + (N - x - M) \cdot \log \left( \sigma _{2,\mathrm{max}}^{2}\right) + C_{2} \end{aligned}$$

(1)

Here, x is the point in the time series around which the signal information is evaluated, $\sigma _{1,{\mathrm{max}}}$ is the maximum variance below the point, and $\sigma _{2,{\mathrm{max}}}$ is the maximum variance above the point. M is the model autorecursive filter order (set to 1), and N is the number of data points in the time series. C ₂ is a constant, which we leave out of our calculations as we are only considering the first derivative.

In this study, we evaluate the first time derivative of the AIC of the magnetic field signal (see Fig. 4 for an example). A peak in these data that rises above a multiple of the standard deviation of the first derivative is considered a detection. For our algorithm, we find an ideal time series length of 30 min using 1-s data.

Fig. 4

Time derivative of the Akaike information criterion (AIC). There is a definite peak at the time of SSC. The peak is picked out as an SSC detection if its maximum exceeds a multiple of the variance of the whole function. Note that both edges of the function show noisy behavior. This method looks at windows of data, and the timing is important, making it effective to use overlapping windows when evaluating larger data sets. The high-frequency part of the SSC is marked on either side by the two dashed gray lines

Automated detection of CMEs

As detailed in section “Defining geomagnetic storms and stormy conditions,” the arrival of a CME shock front is assumed to preclude a geomagnetic storm for the cases we are considering. For the purpose of separating rapid geomagnetic signals from true SSCs, we include satellite measurements of solar wind properties in our analysis and use these to determine the current “stormy conditions.”

This is achieved, first, by detecting a shock front (discontinuity) in the solar wind speed using what is essentially a low-pass filter to reduce noise. Around any given point in the time series of solar wind data, the average of a window of data below the point is subtracted from the average of a window of data above the point to find a simple change in wind speed level. The length of these windows was set to 20 min (or 20 samples), as below this value true step functions in the v _sw could often be lost to noise. The point in time of the shock front arrival t _CME was taken from the maximum of the resulting function. This method, though simple, proved reliable at both identifying CMEs and picking out the exact time of CME arrival at the ACE satellite, while separating the CMEs from signals resulting from noise or more active solar wind periods.

An analysis of the various properties of the solar wind at the timestamp t _CME then leads to a cumulative probability of whether or not the signal that triggered the detection was a genuine CME shock front likely to cause a storm. The probabilities depend on the following conditions, each of which results in an individual probability P _i :

• 1. Change in solar wind speed, e.g., $\Delta v_{\mathrm{sw}} > 50$ km/s. The value of the change in solar wind speed was taken as the maximum value in the function used to detect the CME front.

• 2. Solar wind speed of clouds, e.g., v _sw > 400 km/s. This value was obtained by averaging the solar wind speed in the 10 min following t _CME.

• 3. Change of variance in wind speed after the discontinuity or $\sigma _{2} > \sigma _{1}$, where $\sigma _{1}$ is the standard deviation in the solar wind in the 20 min window before t _CME, and $\sigma _{2}$ is the standard deviation in the same window following t _CME. It was observed that the solar wind speed would generally become more disturbed within the cloud behind the shock front, leading to a larger standard deviation.

• 4. Value of proton flux in the 47–68 keV range, e.g., j _sw > 50,000 particles/(cm² s ster MeV). The level of proton flux was calculated from the average of the flux in the 10 min following t _CME. It was observed to rise to an elevated level either slightly prior to or simultaneously with the observable CME shock front in the solar wind speed.

These variables were chosen after analyzing various properties of the solar wind measured by the ACE satellite for the 15 example storms. The rise in solar wind speed, along with the proton density and ram pressure, have often been considered good parameters to observe with regard to the geoeffectiveness of a solar wind cloud (Gonzalez and Tsurutani 1987; Richardson and Cane 2011). The proton flux, which has not been extensively covered in past studies but is related to the proton density, proved a useful parameter in this study.

The final probability of “stormy conditions” is then taken as a sum of all of the individual probabilities P _i (total number of m) after application of corresponding weights a _i :

$$P_{\mathrm{CME}} = \frac{ \sum \nolimits _{i}^m a_{i} P_{i} }{\sum \nolimits _i^m a_i}$$

(2)

The respective weights were selected empirically while applying the method. The values for each weight were adjusted through trial and error until the maximum number of storms was correctly detected with probabilities correspondingly larger than noise-related signals. The greatest weight is given to the elevated proton flux (3, while all other weights are equal to 1).

Application to data

As described in each method, the criteria that lead to a possible SSC detection are different for each method. Beyond that, however, the storm detection methods remain the same regardless of SSC identification algorithm.

All the proposed methods are applied to measurements of the H-component of the geomagnetic field. The time of SSC initiation, t _SSC, is selected as the initiation time of each respective algorithm peak (this being the first point at which the minimum level is exceeded), and the amplitude of the SSC ($\Delta H$) is determined by subtracting the field strength at t _SSC from the field strength at the end of the peak or period exceeding the minimum level. The duration of the SSC $\Delta t$, which is of interest in determining the variation rate of the SSC, is taken as the time difference between the end of the peak and t _SSC. In summary, the storm detection algorithm delivers three variables when applied to data containing an SSC: t _SSC, $\Delta H$ and $\Delta t$.

As noted by Shinohara et al. (2005), although it was possible for them to correctly detect rapid increases in magnetic field strength with an automated detection algorithm, the system resulted in many false triggers from similar signals that did not result in storms. In order to improve the rate of correct detections, they would also need to determine whether stormy conditions apply to correctly identify a trigger as commencement of a geomagnetic storm.

To combine “stormy conditions” from satellite data analysis with detections of SSCs in the magnetic field, a two-step approach is implemented. First, the satellite data are analyzed for potential shock fronts. These are given a probability factor dependent on various parameters of the shock. In addition, an estimated SSC time on Earth, $t_\mathrm{SSC,Est}$, is calculated using the arrival time of the CME shock at the satellite, t _CME, and the bulk solar wind speed at the shock front, v _sw. This is assumed to have a simple $t \propto v^{-1}$ relationship, and we use the following equation:

$$t_{\mathrm{SSC, Est}} = t_{\mathrm{CME}} + c_{\mathrm{arr}} v_{\mathrm{sw}}^{-1} + d_{\mathrm{arr}}$$

(3)

The variables $c_{\mathrm{arr}}$ and $d_{\mathrm{arr}}$ (storm “arrival” coefficients) are calculated from a fit to data taken from the 15 storms examined in this study. A plot of the fit can be seen in Fig. 5.

Fig. 5

Correlation between the solar wind speed at the shock front and the time taken between the arrival of the CME shock front at the ACE satellite and the SSC initiation on Earth

Second, the Earth magnetic field data are analyzed using one of the methods listed above. The two detections are then combined using considerations of the estimated SSC arrival time. Only those detections of SSCs in the magnetic data that fall within a period of $t_{\mathrm{SSC,Est}} \pm 30$ min are considered detections, and the probability of the detection is determined using a normal distribution of the result of $t_{\mathrm{SSC}} - t_{\mathrm{SSC,Est}}$. The larger window of ±30 min was chosen to cover the full range of travel times (between 20 and 70 min) and to allow for errors within the ACE shock arrival time $t_{\mathrm{CME}}$, which may be over- or underestimated because of loss of satellite data as the shock front arrives.

The probability assigned to the preceding shock at the ACE satellite is also included in a final probability factor for the SSC detection:

$$P_{\mathrm{SSC}} = \frac{a_{\mathrm{CME}} \cdot P_{\mathrm{CME}} + a_{\Delta H} \cdot P_{\Delta H} + a_{t_{\mathrm{arr}}} \cdot P_{t_{\mathrm{arr}}}}{a_{\mathrm{CME}} + a_{\Delta H} + a_{t_{\mathrm{arr}}}}$$

(4)

$P_{\Delta H}$ is a probability awarded according to the size of the detected SSC, as a change in the geomagnetic field of insignificant size (<5 nT) (Joselyn and Tsurutani 1990) is unlikely to be an SSC. The weighting parameters here were also adjusted through trial and error; the largest weight was given to $P_{t_{\mathrm{arr}}}$, which had a weight of 2, while the others were given a weight of 1.

The method described above shows how we get from data to detections, but we have included three different methods to compare. To achieve a quantitative comparison of the methods, we look at three properties (in order of decreasing importance):

• 1. Reliability: how many of the known SSCs does the method correctly detect? How many false positives does the method produce? To determine the reliability of each method, we ran the data of geomagnetic field measurements from the period with known storms through a test storm detector and iterated through a range of parameters unique to each method. For example, with the FD method, these parameters would be the amplitude of the time derivative of the time series that must be exceeded, a _FD, and p _FD, the length of time it exceeds it. Within this two-dimensional parameter space, we determined the number of correct detections (detections within ±4 min range of SSC time as defined by the ISGI) and number of false positives per pair of parameters. The best parameters for the method were those that would result in the maximized number of correct detections and minimized number of false positives. A representation of this method is shown in Fig. 6. As can be seen in the figure, the highest sensitivity parameters (low amplitude a, short period p) result in the largest number of correct detections, but they are greatly outnumbered by the number of false positives. Note that these are the results for the method alone, without combination with satellite data, which leads to a great number of false positives.

• 2. Speed: how soon after an SSC can the event be detected by each method? To evaluate this we determined, through iteration of increasing time, how long each method takes to trigger a correct detection after an SSC has started. (Note: this is not the computation time required by the computer for a detection, but the delay after the initiation of an SSC before the signal can be interpreted as such.)

• 3. Accuracy: how accurately are the SSC time, t _SSC, and change in the magnitude of the H-component of the geomagnetic field, $\Delta H$, picked out? Once the optimized parameters have been determined according to point 1, we run the data from stormy days containing SSCs through the test detector with each method and compare the results to values picked out by eye.

Data sets

All geomagnetic data used in this study were measured at the Conrad Observatory (47.9°N, 15.8°E in geographic coordinates, 47.1°N, 99.1°E in geomagnetic coordinates) in Austria. The resolution is 1 s, which is an improvement over most former storm detection methods that used data with only a 1-min resolution. We hope to determine the best way to detect storms in near real-time, and this requires as high a data resolution as possible. The period of evaluation covered February 2013 (2013-02-01 00:00) until April 2015 (2015-04-01 00:00) with a total of 741 days. (This includes a break in the summer of 2013 during additional building work at the observatory.)

Fig. 6

Representation of the method used to determine the ideal storm detection parameters for a given method. In this case when using the MODWT, a _MODWT is the amplitude a peak in the squared MODWT needs to exceed to trigger a potential detection, and p _MODWT is the period it needs to exceed this amplitude to be a proper detection. The test “detector” was tested on the geomagnetic data using each pair of values within a certain range ($0.00 < a_{\mathrm{MODWT}} \le 0.01\,{\hbox {nT}}^2$ and $20 < p_{\mathrm{MODWT}} \le 90$ s), and the numbers of correct detections and false positives were recorded. The best parameters were those that had the highest percentage of correct detections out of the total possible (left) and the correspondingly highest percentage of correct detections out of all detections made, including false positives (right)

Values for the k _p index were taken from the Geoforschungszentrum Potsdam (GFZ).

Satellite data on solar wind properties used in this study were obtained exclusively from the NASA Advanced Composition Explorer (ACE) satellite, which is situated at the L1 point at a distance of 1.5 million km from the Earth in the direction of the Sun. These data are publicly available. For real-time detection purposes, we used data available from the NOAA Web site (ftp://ftp.swpc.noaa.gov/pub/lists/ace/), which has 1-min solar wind data (bulk solar wind speed, proton density) from the ACE SWEPAM instrument available with a delay of 3–4 min, and 5-min resolution data from the EPAM instrument on solar wind particle fluxes available with a delay of 7–12 min. The data are mostly unprocessed and contain a considerable amount of noise.

For the evaluation of detections of past storms, we used level 2 ACE data from the OMNI database (http://omniweb.gsfc.nasa.gov/).

Results

The results from our test storm detector comprise a list of detections that would have been made over the past 2 years and are detailed in Table 2 alongside results from evaluating the speed and accuracy of each method. Satellite CME shock front and geomagnetic SSC detections are initially treated as separate detections.

Table 2 Summary of results

Reliability of ACE storm evaluation

Of the storms we looked at with the ACE CME detection method, a total of more than 500 shocks were detected within the solar wind data, out of which 14 of our 15 official shock fronts were correctly detected. (The single false negative here represents a calculated shock arrival time that was 30 min off the actual CME arrival time, which meant it was outside of our acceptance range but could factor into later correct SSC detections.) However, the number of false positives in this case far outweighs the number of correct detections. By applying a simple condition of only considering storms with $P_{\mathrm{CME}} \ge 75$ (see Eq. 2), we reduce the number of detected shock fronts to a total of 15, of which 12 are correctly detected CME shocks and 3 are false positives. This does not mean that the two lower-probability false negatives were then ignored in further analysis. These could later factor into a correct SSC detection with the corresponding probability $P_{\mathrm{CME}}$ factoring into a sufficiently high $P_{\mathrm{SSC}}$.

In Thomson et al. (2005), in which a CME shock detector that also uses ACE solar wind data was developed, similar results were found, namely that the number of false positives rises with the sensitivity of the parameters and number of correct detections. Their final algorithm resulted in around one false alarm per month. Comparatively, our system represents a significant improvement at a total of three false alarms over a span of 26 months.

Reliability of magnetic data evaluation

To assess the reliability of each method, we split all detections made by our test storm detector over the data period into five groups:

• Storm: the detection was of an SSC of one of the storms that fulfilled the criteria listed in section “Defining geomagnetic storms and stormy conditions.”

• SSC: the detection was of an SSC from the remaining 42 SSCs (as defined by the ISGI in the period of analysis) that did not fulfill our criteria.

• Stormy period (<24 h): the detection was of a non-SSC signal within 24 h of a storm or SSC.

• False positive: this constitutes a false detection where the software detected a signal that had no connection to a confirmed geomagnetic storm.

• False negative: this is a missed detection.

We have also included a quantitative factor for evaluating the reliability, R:

$$R = \frac{N_{\mathrm{correct}}}{N_{{\mathrm{false}}\,{\mathrm{positives}}} + 1} = \frac{N_{\mathrm{Storms}} + N_{\mathrm{SSCs}}}{N_{{\mathrm{false}}\,{\mathrm{positives}}}+ 1}$$

(5)

As can be seen in the first section of Table 2, in which all detections made by the algorithms without application of stormy conditions are included, both the MODWT and FD methods achieved 15 out of 15 correct detections, while the number of false positives rises into the hundreds. The AIC method, with 13 out of 15 detections, showed less sensitivity to SSC signals and also resulted in fewer false positives. This method also had the highest percentage of all non-SSC detections outside of the stormy periods at 73 %, whereas the MODWT and FD methods have 42 and 53 %, respectively. This suggests that compared to the AIC method, both the MODWT and FD methods have higher sensitivity to the kinds of signals caused by storms over other kinds of geomagnetic signals.

As is obvious from these results, the use of any of these methods alone with no further constraints would not constitute a useful storm detector because the false positives greatly outnumber the correct detections. These results echo the findings of Shinohara et al. (2005), who found that an automated detection system detects all rapid increases in the geomagnetic field, not all of which are caused by geomagnetic storms. These results are nevertheless useful to keep in mind for a greater overview of the types of signals detected by each method and for comparison later.

With the combination of results from magnetic and ACE data analysis, the results show a massive reduction in the number of false positives compared to the methods without stormy conditions, as is to be expected. The combined method with the most correct detections was the FD method, with 15 out of 15. The MODWT method lost one detection to the stricter conditions with 14 detections, but the storm in question was at the bottom end of our criteria.

We evaluated the false positives (seven different detections in total) found by all methods; three of these could be described as sudden impulses, rapid increases in field strength resulting from a minor CME shock in the solar wind that did not result in a storm. Two were identified as substorms, and two others were signals that should not have been classified as SSCs. The MODWT method, although resulting in four false positives, did not detect anything other than SIs and substorms. Although these are false triggers, they do represent the detection of geomagnetic stormy conditions, and so in the test of reliability, the MODWT method offers the most benefits. This is likely because it is the most apt at picking out signals unique to storms. The MODWT method also had the highest value of R, and we would therefore label this method the most reliable of the three.

Detection speed

The method able to detect an SSC the fastest was the AIC method, which needed an average of 55 s for a detection with a minimum of 12 s. The detection speeds for other methods range on average from 100 to 150 s, with minima of 50–90 s. The computation times for each method were calculated for the evaluation of a 2-h period, which are all in the range of 0.4 s. The computations were carried out on a PC running Ubuntu 12.04 with an Intel Core i3-2120 (3.30 GHz) processor and 4 GB of RAM.

Accuracy

The FD method is the best in terms of timing accuracy, and all methods slightly overestimate t _SSC. The MODWT shows a particularly high standard deviation and spread in the values (due mainly to one detection deviating by 3 min) and the FD method slightly underestimating t _SSC but with a lower standard deviation. The value for t _SSC gained from the AIC method could be minutes off the actual value, and this method proved the least accurate by for picking out timing values.

In the determination of the magnitude (in nT) of $\Delta H$, both the FDM and AIC methods showed fairly high accuracy with no skewing to over- or underestimate. The MODWT produced slightly more inaccurate results in comparison and did tend to underestimate the size of the SSC. This is likely due to the discontinuous shape of many SSCs, and the MODWT method preferentially picks out continuous slopes.

When looking at the accuracy of both picked parameters, the FD method provides the most accurate results.

Automated storm detection software

We now present the prototype currently running at the Conrad Observatory using a data set covering the past 2 h of near real-time ACE solar wind data and geomagnetic data measured (with a few ms delay) at the observatory. The software was put together after analyzing the various methods in this study and considering how best to implement these in a continuous detection software. The method determined to bring the best of all components (reliability, speed, and accuracy) was the MODWT method.

The software is split into two routines, one for the evaluation of ACE data and one for the analysis of geomagnetic data. As a bridge between the two routines there is a status log with the latest detections, and here the stormy condition status is set to “quiet” when there have been no detections. Any positive results from the routine analyses are split into two groups per type:

• Satellite CME shock detections

  • Satellite detection: this is the detection of a possible shock front within the ACE satellite data. We do not know whether it is a storm or not.

  • CME: this is a detection that the software has categorized as a CME shock front powerful enough to induce a geomagnetic storm.

• Geomagnetic SSC detections

  • Magnetic detection: this is the detection of an SSC-like signal in the geomagnetic data within the expected time range defined by a satellite detection/CME. We do not know whether it is an SSC or not.

  • Storm: this is an SSC detection that the software has categorized as an SSC at the commencement of a geomagnetic storm.

The routine for ACE satellite data runs every 10 min; this period is limited because of data upload times. If a shock front with a high enough probability ($P_{\mathrm{CME}} > 75$) is detected, a summary of the properties of the solar wind and the estimated SSC time on Earth will be compiled into a report (which can subsequently be sent as a warning e-mail to all subscribers). The status of the stormy conditions in the status log will be updated from “quiet” to “stormapproach,” and the detection will also be logged as a CME rather than a basic detection. This change in the status of stormy conditions means that the routine for magnetic data, which runs every minute, will begin putting the magnetic data through the MODWT analysis to look for SSCs within the estimated $t_{\mathrm{SSC}} \pm 30$ min range. If an SSC with a high enough probability ($P_{\mathrm{SSC}} > 80$) is detected, a further report is compiled, and the storm status is upgraded to “stormy.” This SSC will then be logged under the category Storm instead of a simple magnetic detection

On the other hand, if a CME shock front with a lower probability ($P_{\mathrm{CME}} \le 75$) is detected, the detection will be logged as a simple satellite detection. Conditions will remain “quiet,” but the magnetic routine will still search for SSCs within the estimated $t_{\mathrm{SSC}} \pm 30$ min range. In some cases, a detected SSC with a high enough probability will upgrade the satellite detection of a low-probability solar wind shock front from a satellite detection to a CME. In this case, a report for only the SSC will be processed, and the status will be upgraded from “quiet” to “stormy.”

After a Storm detection, the status will remain “stormy” for 24 h following the detection, and there will be no further reports or warnings within this period. This is to eliminate the majority of false positives caused by substorms and similar rapid signals. There is an exception to this rule: if a further shock is detected in the satellite data, the detection will be logged as a satellite detection and the routine for magnetic data will continue to search for SSCs within the expected period, and if one with an exceptionally high probability ($P_{\mathrm{SSC}} > 95$) is detected, a further storm report will be produced announcing a new Storm, and the “stormy” status will be extended for a further 24 h. This probability was chosen after analysis of the results from our test storm detector, where there were many SSCs that followed initial SSCs less than 24 h after. These were found to always have a probability greater than 95 %, a boundary that all other non-SSC signals remained beneath.

Below is an automatically created report of a geomagnetic storm from a June 22, 2015, SSC at the Conrad Observatory as an example of the information provided by a final storm detector and how it can be useful as a warning system:

• Results from magnetic evaluation:

• Time of SSC (UTC): 2015-03-17 04:45:32

• Magnitude of SSC: 42.09 nT

• Duration of SSC: 120 s

• Estimated storm strength: G3-Strong

• Time of detection (UTC): 2015-03-17 04:48:03

• Results from ACE evaluation:

• Time of CME shock arrival at ACE (UTC): 2015-03-17 04:00:00

• Solar wind bulk speed: 501.62 km/s

• Solar wind proton flux (47–68 keV): 455,500 p/(cm² s ster MeV)

Within the SSC reports, we also include an estimated storm strength using the NOAA storm scales. This calculation is based solely on the level of particle flux at the shock front at the time it reaches the ACE satellite, as this was shown to have the highest correlation (R = 0.63) with storm max(k _p ) from all the ACE satellite parameters. This allows for an accuracy of $k_{p,{\mathrm{est}}} \pm 1$ to be achieved based on the limited number of example storms. This procedure can be refined at a later time with an expanded data set.

Discussion

With a detection rate of 93 %, we have achieved a satisfactory level of storm detection for stronger storms as defined by our criteria. However, there are certain assumptions made over the course of the development that may limit the abilities of the system and should be addressed.

Detection of CIR-initiated storms

Co-rotating interaction regions (CIRs) are radially aligned solar wind flows of varying speeds with forward and reverse shocks bordering the flows with greater velocities (Gosling and Pizzo 1999). CMEs, which are more common during solar maximum, typically cause stronger, shorter duration storms with more rapid variations, whereas storms caused by CIRs are slower and calmer. This method of storm detection, which requires CME shocks and SSCs as prerequisites for a successful storm warning, does not detect slow-onset storms, which are generally categorized as having neither a discontinuity from a CME shock front nor a definitive SSC. This represents the difference between CME-initiated and CIR-initiated storms, which has been studied in detail in Borovsky and Denton (2006). In their study, the authors came to the conclusion that 69 % of CME-driven storms are connected to an SSC, and all GIC events on record were connected to CME-driven storms. We can therefore argue that, for the purpose of predicting and detecting storms with heightened risk of GIC, this method is well suited.

Latitudinal bias

A further limitation arises from looking at only one observatory, making this system geographically restricted. SSCs are more pronounced at higher latitudes, meaning that at lower latitudes they may not register and may be missed. In this case, an approach such as that used by Hafez et al. (2013a) with measurements from multiple observatories being analyzed and compared is beneficial. For nowcasting and prediction services, however, high-resolution (1-s) real-time data are required. This is not available for most observatories, and thus, we focused on what can be achieved by one observatory.

Availability of ACE data

One potential problem is the availability of real-time ACE satellite data. The satellite data in very stormy periods are often fragmented, leaving perhaps 5 % of useful data over a 2-h period. This restricts the usefulness of the method, which needs a certain percentage to be able to provide results. To combat this problem, the information on the solar wind should ideally come from multiple satellite sources between the Sun and Earth, such as the recently launched DSCOVR (NASA/NOAA) satellite or a future ESA space weather satellite.

Comparison of two data sets

In addition to evaluating the properties listed in section “Application to data,” we also compared two sets of data with a smaller overlapping time range of 141 days. The first data set, on which all the other results are based, was data measured using an older FGM-FGE standard observatory fluxgate variometer from 2004 with a quiet day noise level of 0.1 nT/$\sqrt{\hbox {Hz}}$ at 0.3 Hz. The second data set was taken from comparatively newer (acquired in 2009) LEMI-025 fluxgate magnetometer measurements, which have a lower noise level of 0.01 nT/$\sqrt{\hbox {Hz}}$ at 0.3 Hz. It was found that all the methods provided better results when using data from the newer instrument; the parameters required for a high percentage of correct detections were less sensitive, which resulted in fewer false triggers (reduced by a factor of 2–7) when using the methods alone without stormy conditions. This shows that a low noise level provides optimal results. It also means that the parameters used in each method should be chosen according to noise level.

Comparison of MODWT with Hafez et al. (2013b)

We lastly compare our results to those of Hafez et al. (2013b), which is the most recent paper on the matter. In their study, the authors use the undecimated DWT method (also an MODWT method) on 1-s resolution data.

Hafez et al. (2013b) report an error ranging between −30 and 180 s, and our error ranges between −25 and 195 s. Their average and standard deviation of detection error were 35 and 44 s, respectively, very similar to our values of 17 and 53 s. In the algorithm of Hafez et al. (2013b), detection takes 50–180 s with the hardware conditions described in section “Detection speed.” Our algorithm requires, on average, 160 s after storm onset before an SSC is detected, so it is comparable in terms of detection delay. As can be seen, both studies result in very similar levels of accuracy with different data sets and programming approaches, underlining the reliability of this method.

Hafez et al. (2013b) attempted to detect all SSCs within their time period, and achieved detection of 122 of 133 storms, or a correct detection rate of 91 %. We did not attempt detection of every SSC in our analysis, making a comparison with our rate of detection difficult. For comparison purposes, we ran the iteration using the MODWT method for all SSCs in our period of analysis and achieved a detection percentage of 79 % with a maximum of 44 of 57 SSCs detected. As we are using a single mid-latitude observatory in contrast to a wide spread of observatories, it is very likely that some SSCs detectable in higher latitudes fall below our detection threshold, leading to a lower percentage of detection.

Conclusion

To summarize, we took three different computational methods that can detect rapid changes in signals and applied them to past geomagnetic data to determine which method could best pick out sudden storm commencements. It was found that the accuracy of these methods was greatly improved when we applied “stormy conditions” describing the likelihood that an SSC detection was caused by a CME. The method that proved to be the best, the same multi-resolution analysis of the MODWT as used in Hafez and Ghamry (2013), was then implemented in a routine that checks the solar wind and geomagnetic observatory data constantly for storm commencements. This storm detector, tested on data from the past 2 years, correctly detected 14 of the 15 storms looked at in the study, but also resulted in four false positives. All of these false positives, however, were detections of signals related to geomagnetic storms.

This is the first paper to suggest a storm detection system using analysis of geomagnetic field data in conjunction with solar wind data. A project similar to this is NASA’s Solar Shield project (Pulkkinen et al. 2010), which is a program for GIC forecasting that also utilizes ACE satellite data to recognize and detect approaching storms. The Solar Shield project is based on far more complex MHD modeling of the solar wind and its impact on the Earth’s magnetic field. Here we describe a comparatively simple but effective method to detect the approach of strong geomagnetic storms for purposes of giving advanced warning of GIC risk using data measured at the L1 point and then to determine automatically the time of storm commencement on Earth.

All storm detection algorithms used in this study are available in the free GeomagPy Python software package (https://pypi.python.org/pypi/GeomagPy/). Automated geomagnetic storm detection warnings are also being sent out in e-mail form by the software described.