On the feasibility of promptly producing quasi-definitive magnetic observatory data

Abstract

Magnetic observatories currently distribute two types of data: preliminary data, available in less than 72 hrs in the case of INTERMAGNET observatories, and definitive baseline-corrected data, produced only once a year. Several users and groups of users have expressed the need for baseline-corrected observatory data produced in a continuous manner. The main applications for such quasi-definitive data include geomagnetic field modeling and the calculation of geomagnetic activity indices. We present an original method for producing quasi-definitive data at the end of each calendar month using temporary baselines. Preliminary and definitive data at nine INTERMAGNET observatories are used to test this method, simulating the production of quasi-definitive data throughout the year 2008. The temporary baselines obtained are very close to the definitive ones, except during the last few days of each time interval. The means and standard deviations of the differences between quasi-definitive and definitive data do not exceed 0.3 nT, well below the current INTERMAGNET standard of accuracy. This result demonstrates the feasibility of promptly producing quasi-definitive data at most magnetic observatories of INTERMAGNET type.

1. Introduction

Magnetic observatories continuously measure the Earth’s magnetic field at fixed locations at the Earth’s surface. They are designed to operate for several decades to record the geomagnetic secular variation. Their data are widely used for calculating time-varying core field models, often in combination with satellite magnetic data or historical data (e.g., Hulot et al., 2007; Jackson and Finlay, 2007), and for studying rapid processes in the core, including core flows (Holme, 2007) and so-called geomagnetic jerks (Courtillot et al., 1978). Magnetic observatory data are also used for studying the various electrical current systems in the Earth’s ionosphere and magnetosphere, both during short events such as magnetic storms and substorms, and on longer time scales (e.g., McPherron, 2009).

Magnetic observatories belonging to the global INTERMAGNET (International Real-time Magnetic Observatory Network; http://www.intermagnet.org) network provide one-minute values of the three components of the geomagnetic field. However, vector magnetometers used in magnetic observatories, often of fluxgate-type, usually drift in time due to several factors such as temperature variations and ageing of the electronics. In order to periodically recalibrate the vector magnetometer and, if needed, correct for the tilt of the pillar on which the magnetometer is mounted, absolute measurements are performed on a regular basis (typically once a week). Such measurements are made by a trained observer using a single-axis fluxgate magnetometer mounted onto a non-magnetic theodolite, and a scalar magnetometer (see, e.g., Jankowski and Sucksdorff, 1996). Absolute measurements are used to calculate a calibration curve, called a baseline. INTERMAGNET currently distributes two types of data products: preliminary data, which are made available in quasi-real time (less than 72 hours), and definitive data, which are produced only once a year, typically a few months after the end of the civil year. Definitive data are corrected for baseline variations, whereas preliminary data may not have any baseline corrections applied and can thus only be used for studying rapid variations of external origin.

The need for baseline-corrected observatory data released on a shorter time scale has recently emerged within the community of global modeling. Global spherical harmonic models of the recent geomagnetic field usually rely on a combination of observatory and satellite data (e.g., Lesur et al., 2008; Olsen et al., 2009), except for their most recent part as definitive observatory data are only available after a one year delay. This is to be contrasted with fully calibrated data from magnetic satellites such as Oersted and CHAMP, which are available within only a few days after the commissioning phase. For the same reason, the usefulness of standard observatory data products for calculating and validating the International Geomagnetic Reference Field (IGRF) model every five years (Macmillan and Maus, 2005) is limited.

In order to address this need, INTERMAGNET is currently preparing to distribute a new data product called quasi-definitive data, defined as data corrected using temporary baselines shortly after their acquisition and very near to being the final data of the observatory (Baillie et al., 2009; Chulliat et al., 2009). Quasi-definitive data will thus fill the gap between preliminary and definitive data. They should prove particularly useful for calculating the Level 2 products (field models of the various field sources) of the upcoming ESA Swarm mission (Friis-Christensen et al., 2006). Using quasi-definitive data is also expected to improve the quality of quick-look versions of geomagnetic indices such as the D_st.

The aim of this paper is to discuss the feasibility of the prompt production of quasi-definitive data. We present an original method for producing such data and the first results of its implementation in nine magnetic observatories belonging to the Bureau Central de Magnétisme Terrestre (BCMT) network (Fig. 1). After having assessed the quality of quasi-definitive data based on a systematic comparison with definitive data in 2008, we propose a quality standard for this new data type.

Fig. 1

Locations of the nine magnetic observatories considered in this study.

2. Method

We propose a method where quasi-definitive data are calculated every month. Shortly after the end of month M of year Y (noted M/Y hereafter), the following data processing steps are performed: (1) preprocessing of the variation data of M/Y; (2) calculation of a temporary baseline from 1st December of year Y-1 to the last day of M/Y; (3) calculation and validation of baseline-corrected data from 1st January of year Y to the last day of M/Y. Using this method, twelve quasi-definitive datasets are successively produced during year Y; each new dataset replacing the previous one. However, as we will see later, at most observatories it is possible to implement a variant of this method where step (3) is only applied to data from M/Y.

Step (1) involves removing spikes and correcting for jumps in the raw one-second or one-minute variation data recorded by the vector magnetometer, following standard observatory practice. Most spikes or disturbances generated by nearby human activities (e.g. vehicles), have typical signatures distinct from those of magnetic pulsations and can thus be quickly identified and removed. Spikes of unknown origin are left unprocessed provided their amplitude is smaller than 5 nT. Jumps caused by planned movements of the magnetometer or changes to the electronics are corrected for by adding a constant offset to the data.

In step (2), we calculate a temporary baseline based on absolute measurements, made at least once a week according to INTERMAGNET recommendations, but sometimes less often due to staff shortage or other organizational constraints in some observatories. We use cubic smoothing splines with a constant smoothing parameter (csaps function of the Matlab™ software, which relies on the algorithm of de Boor, 1978) to approximate the differences H_o, D_o, and Z_o between absolute measurements and variometer measurements. The choice of the smoothing parameter P determines the trade-off between the roughness of the baseline and its distance to H_o, D_o or Z_o: if P = 0, the spline is the least squares straight line fit to the data; if P = 1, the spline is the variational cubic spline interpolant. P is selected for each component and each observatory based upon the value used for calculating the definitive data of the previous year (when available), assuming that the main properties of the baseline do not change from one year to the other.

In step (3), quasi-definitive data are generated from the baseline obtained in step (2) and then reviewed by a qualified observer for final validation. The quality control procedure consists of visual inspection of temporary baselines (outlier removals), checking the continuity between the quasi-definitive data and the definitive data of the previous year, checking the scalar residuals (obtained by taking the difference between the scalar data and the field modulus calculated from the vector data), and visual inspection of all components on different time scales.

3. Test of the Proposed Method

We tested the proposed method using the 2008 preliminary and definitive data from nine INTERMAGNET observatories belonging to the Bureau Central de Magnétisme Terrestre (BCMT, http://www.bcmt.fr) network: Addis Ababa (AAE), Borok (BOX), Chambon la Forêt (CLF), Kourou (KOU), Lanzhou (LZH), MBour (MBO), PhuThuy (PHU), Pamataï (PPT) and Tamanrasset (TAM). The observatory geographical locations are shown in Fig. 1.

As the smoothing spline algorithm used for calculating baselines of BCMT observatories prior to 2009 was slightly different than the one proposed in Section 2, we could not select the parameters P based upon the values used for the previous year (step (2) of the proposed method). Instead, we selected P in order to have the best trade-off between the roughness of the baseline and its distance to H_o, D_o and Z_o. The same P was then used to calculate successive temporary baselines and the final baseline for 2008. (Note: it follows that the final baselines obtained are slightly different than the definitive baselines published on the 2008 INTERMAGNET DVD. However, we checked that the difference between the published definitive data and the definitive data used in this paper was always less than 1 nT.) Figure 2 shows the definitive and successive temporary baselines obtained at two observatories, BOX and TAM. At BOX, there is an irregular variability superposed on the annual variation of the differences H_o, D_o, and Z_o between absolute measurements and variometer measurements, due to temperature variations in the variometer room. In order to follow this irregular variability we chose large values for the smoothing parameter: P = 0.3, 0.1 and 0.3 for the H_o, D_o, and Z_o baselines, respectively. In contrast, TAM is an observatory where H_o, D_o, and Z_o show only a regular annual variation (due to temperature variation in the variometer room), and where the number of absolute measurements is very large. For that observatory it is better to choose a very smoothed spline, and we chose P = 0.0001 for all three components.

Fig. 2

Evolution of the temporary baselines (H_o, D_o and Z_o) calculated each month for the BOX and TAM observatories. The red line is the definitive baseline. Crosses are the absolute measurements. For a better readability the vertical scale is distinct for each component and each observatory.

The temporary baselines shown in Fig. 2 closely match the final baseline, except during the last few days of the time interval of each temporary baseline. The smoother is the final baseline, the smaller are the differences between temporary and final baselines. Similar results were obtained for the other seven observatories. Provided the smoothing parameter is carefully chosen, there is no visible difference in the quality of baselines calculated over only two months of data (from 1 December of year Y-1 to 31 January of year Y) and baselines calculated over several months of data. This result suggests that producing quasi-definitive data every month is indeed feasible.

In order to assess the overall quality of quasi-definitive data, we performed a statistical analysis of the differences between successive sets of 1-minute quasi-definitive data and the final 1-minute definitive data in 2008 for the nine magnetic observatories listed above. Our results are reported in Fig. 3 for the X component; results for the Y and Z components are similar (not shown). For all observatories, the mean differences between the first quasi-definitive datasets and the final dataset are larger than the mean differences for the last quasi-definitive datasets. This reflects the larger relative number of days where the temporary baseline diverges from the final one in these first, shorter datasets. For all observatories, both the means and the standard deviations of the differences remain very small (below 0.05 nT at CLF, LZH, MBO, up to 0.3 nT at BOX), well below the 5 nT INTERMAGNET standard of data accuracy. The largest standard deviations, as observed for example in February at BOX (Fig. 3) but also in June at KOU or December at PPT, are obtained for months where the final baseline has a high curvature. In these cases, the smoothing spline cannot closely approximate the final baseline as the curvature is controlled by the data points for the following month, which are not available.

Fig. 3

Evolution through 2008 of the mean and the standard deviation (represented as an error bar) of the differences (in nT) between quasi-definitive and definitive 1-minute data for the X component, at the nine selected observatories. The mean and the standard deviation at month n are calculated using all data from January to month n. For a better readability the vertical scale is distinct for each observatory.

4. Discussion

We have shown that it is possible to produce baseline-corrected data very close to definitive data at nine INTERMAGNET observatories distributed worldwide and operated in various environments. The method described above has been implemented in routine operations for these nine observatories, and quasi-definitive data have been made available at the BCMT website (http://www.bcmt.fr) every month from July 2009 onwards. As the successive quasidefinitive datasets are all very close to the final dataset, a simpler version of the proposed method could also be implemented where quasi-definitive data for a given month would not be updated later during the year. This simpler version would avoid the risk of confusion between successive datasets, although such a risk might also be reduced by properly commenting the quasi-definitive datasets.

Our results show that quasi-definitive data can be produced even if baselines have irregular variations such as those observed at BOX. It follows that the baseline shape is not a limitation for producing quasi-definitive data and that the main limitations are operational and concern the steps (1) and (3) of the proposed method. At some observatories, it might be difficult to remove spikes, deal with gaps, correct for jumps and validate absolute measurements within only a few days or even weeks, due to lack of staff and/or the complexity of the problems encountered. However, it is worth noting that such tasks have to be done anyway when preparing definitive data. Another potential problem is the lack of absolute measurements for an extended period of time at some observatories. Although this situation remains exceptional at INTERMAGNET observatories, it is advisable not to produce quasi-definitive data at the end of a month in which no absolute measurement has been made.

Based on the results presented above, it seems realistic to extend to quasi-definitive data the 5 nT INTERMAGNET standard of accuracy used for definitive data. In a time when magnetic observatories may be at risk of losing some of their perceived relevance, it is important that they provide data with similar quality and timeliness as satellites. This paper shows that producing quasi-definitive data in a continuous manner is feasible and even relatively easy. We hope this will encourage observatories to produce quasi-definitive data before the launch of the upcoming Swarm mission.