On the feasibility of promptly producing quasi-definitive magnetic observatory data
© The Society of Geomagnetism and Earth, Planetary and Space Sciences (SGEPSS); The Seismological Society of Japan; The Volcanological Society of Japan; The Geodetic Society of Japan; The Japanese Society for Planetary Sciences; TERRAPUB 2010
Received: 12 January 2009
Accepted: 3 February 2010
Published: 22 February 2010
Magnetic observatories currently distribute two types of data: preliminary data, available in less than 72 hrs in the case of INTERMAGNET observatories, and definitive baseline-corrected data, produced only once a year. Several users and groups of users have expressed the need for baseline-corrected observatory data produced in a continuous manner. The main applications for such quasi-definitive data include geomagnetic field modeling and the calculation of geomagnetic activity indices. We present an original method for producing quasi-definitive data at the end of each calendar month using temporary baselines. Preliminary and definitive data at nine INTERMAGNET observatories are used to test this method, simulating the production of quasi-definitive data throughout the year 2008. The temporary baselines obtained are very close to the definitive ones, except during the last few days of each time interval. The means and standard deviations of the differences between quasi-definitive and definitive data do not exceed 0.3 nT, well below the current INTERMAGNET standard of accuracy. This result demonstrates the feasibility of promptly producing quasi-definitive data at most magnetic observatories of INTERMAGNET type.
Magnetic observatories continuously measure the Earth’s magnetic field at fixed locations at the Earth’s surface. They are designed to operate for several decades to record the geomagnetic secular variation. Their data are widely used for calculating time-varying core field models, often in combination with satellite magnetic data or historical data (e.g., Hulot et al., 2007; Jackson and Finlay, 2007), and for studying rapid processes in the core, including core flows (Holme, 2007) and so-called geomagnetic jerks (Courtillot et al., 1978). Magnetic observatory data are also used for studying the various electrical current systems in the Earth’s ionosphere and magnetosphere, both during short events such as magnetic storms and substorms, and on longer time scales (e.g., McPherron, 2009).
Magnetic observatories belonging to the global INTERMAGNET (International Real-time Magnetic Observatory Network; http://www.intermagnet.org) network provide one-minute values of the three components of the geomagnetic field. However, vector magnetometers used in magnetic observatories, often of fluxgate-type, usually drift in time due to several factors such as temperature variations and ageing of the electronics. In order to periodically recalibrate the vector magnetometer and, if needed, correct for the tilt of the pillar on which the magnetometer is mounted, absolute measurements are performed on a regular basis (typically once a week). Such measurements are made by a trained observer using a single-axis fluxgate magnetometer mounted onto a non-magnetic theodolite, and a scalar magnetometer (see, e.g., Jankowski and Sucksdorff, 1996). Absolute measurements are used to calculate a calibration curve, called a baseline. INTERMAGNET currently distributes two types of data products: preliminary data, which are made available in quasi-real time (less than 72 hours), and definitive data, which are produced only once a year, typically a few months after the end of the civil year. Definitive data are corrected for baseline variations, whereas preliminary data may not have any baseline corrections applied and can thus only be used for studying rapid variations of external origin.
The need for baseline-corrected observatory data released on a shorter time scale has recently emerged within the community of global modeling. Global spherical harmonic models of the recent geomagnetic field usually rely on a combination of observatory and satellite data (e.g., Lesur et al., 2008; Olsen et al., 2009), except for their most recent part as definitive observatory data are only available after a one year delay. This is to be contrasted with fully calibrated data from magnetic satellites such as Oersted and CHAMP, which are available within only a few days after the commissioning phase. For the same reason, the usefulness of standard observatory data products for calculating and validating the International Geomagnetic Reference Field (IGRF) model every five years (Macmillan and Maus, 2005) is limited.
In order to address this need, INTERMAGNET is currently preparing to distribute a new data product called quasi-definitive data, defined as data corrected using temporary baselines shortly after their acquisition and very near to being the final data of the observatory (Baillie et al., 2009; Chulliat et al., 2009). Quasi-definitive data will thus fill the gap between preliminary and definitive data. They should prove particularly useful for calculating the Level 2 products (field models of the various field sources) of the upcoming ESA Swarm mission (Friis-Christensen et al., 2006). Using quasi-definitive data is also expected to improve the quality of quick-look versions of geomagnetic indices such as the Dst.
We propose a method where quasi-definitive data are calculated every month. Shortly after the end of month M of year Y (noted M/Y hereafter), the following data processing steps are performed: (1) preprocessing of the variation data of M/Y; (2) calculation of a temporary baseline from 1st December of year Y-1 to the last day of M/Y; (3) calculation and validation of baseline-corrected data from 1st January of year Y to the last day of M/Y. Using this method, twelve quasi-definitive datasets are successively produced during year Y; each new dataset replacing the previous one. However, as we will see later, at most observatories it is possible to implement a variant of this method where step (3) is only applied to data from M/Y.
Step (1) involves removing spikes and correcting for jumps in the raw one-second or one-minute variation data recorded by the vector magnetometer, following standard observatory practice. Most spikes or disturbances generated by nearby human activities (e.g. vehicles), have typical signatures distinct from those of magnetic pulsations and can thus be quickly identified and removed. Spikes of unknown origin are left unprocessed provided their amplitude is smaller than 5 nT. Jumps caused by planned movements of the magnetometer or changes to the electronics are corrected for by adding a constant offset to the data.
In step (2), we calculate a temporary baseline based on absolute measurements, made at least once a week according to INTERMAGNET recommendations, but sometimes less often due to staff shortage or other organizational constraints in some observatories. We use cubic smoothing splines with a constant smoothing parameter (csaps function of the Matlab™ software, which relies on the algorithm of de Boor, 1978) to approximate the differences Ho, Do, and Zo between absolute measurements and variometer measurements. The choice of the smoothing parameter P determines the trade-off between the roughness of the baseline and its distance to Ho, Do or Zo: if P = 0, the spline is the least squares straight line fit to the data; if P = 1, the spline is the variational cubic spline interpolant. P is selected for each component and each observatory based upon the value used for calculating the definitive data of the previous year (when available), assuming that the main properties of the baseline do not change from one year to the other.
In step (3), quasi-definitive data are generated from the baseline obtained in step (2) and then reviewed by a qualified observer for final validation. The quality control procedure consists of visual inspection of temporary baselines (outlier removals), checking the continuity between the quasi-definitive data and the definitive data of the previous year, checking the scalar residuals (obtained by taking the difference between the scalar data and the field modulus calculated from the vector data), and visual inspection of all components on different time scales.
3. Test of the Proposed Method
We tested the proposed method using the 2008 preliminary and definitive data from nine INTERMAGNET observatories belonging to the Bureau Central de Magnétisme Terrestre (BCMT, http://www.bcmt.fr) network: Addis Ababa (AAE), Borok (BOX), Chambon la Forêt (CLF), Kourou (KOU), Lanzhou (LZH), MBour (MBO), PhuThuy (PHU), Pamataï (PPT) and Tamanrasset (TAM). The observatory geographical locations are shown in Fig. 1.
The temporary baselines shown in Fig. 2 closely match the final baseline, except during the last few days of the time interval of each temporary baseline. The smoother is the final baseline, the smaller are the differences between temporary and final baselines. Similar results were obtained for the other seven observatories. Provided the smoothing parameter is carefully chosen, there is no visible difference in the quality of baselines calculated over only two months of data (from 1 December of year Y-1 to 31 January of year Y) and baselines calculated over several months of data. This result suggests that producing quasi-definitive data every month is indeed feasible.
We have shown that it is possible to produce baseline-corrected data very close to definitive data at nine INTERMAGNET observatories distributed worldwide and operated in various environments. The method described above has been implemented in routine operations for these nine observatories, and quasi-definitive data have been made available at the BCMT website (http://www.bcmt.fr) every month from July 2009 onwards. As the successive quasidefinitive datasets are all very close to the final dataset, a simpler version of the proposed method could also be implemented where quasi-definitive data for a given month would not be updated later during the year. This simpler version would avoid the risk of confusion between successive datasets, although such a risk might also be reduced by properly commenting the quasi-definitive datasets.
Our results show that quasi-definitive data can be produced even if baselines have irregular variations such as those observed at BOX. It follows that the baseline shape is not a limitation for producing quasi-definitive data and that the main limitations are operational and concern the steps (1) and (3) of the proposed method. At some observatories, it might be difficult to remove spikes, deal with gaps, correct for jumps and validate absolute measurements within only a few days or even weeks, due to lack of staff and/or the complexity of the problems encountered. However, it is worth noting that such tasks have to be done anyway when preparing definitive data. Another potential problem is the lack of absolute measurements for an extended period of time at some observatories. Although this situation remains exceptional at INTERMAGNET observatories, it is advisable not to produce quasi-definitive data at the end of a month in which no absolute measurement has been made.
Based on the results presented above, it seems realistic to extend to quasi-definitive data the 5 nT INTERMAGNET standard of accuracy used for definitive data. In a time when magnetic observatories may be at risk of losing some of their perceived relevance, it is important that they provide data with similar quality and timeliness as satellites. This paper shows that producing quasi-definitive data in a continuous manner is feasible and even relatively easy. We hope this will encourage observatories to produce quasi-definitive data before the launch of the upcoming Swarm mission.
- Baillie, O., E. Clarke, S. Flower, S. Reay, and C. Turbitt, Reporting quasidefinitive observatory data in near real-time, presentation at the 11th IAGA Assembly, Sopron, 2009 (unpublished).Google Scholar
- Chulliat, A., A. Peltier, F. Truong, and D. Fouassier, Proposal for a new observatory data product: quasi-definitive data, presentation at the 11th IAGA Assembly, Sopron, 2009 (unpublished).Google Scholar
- Courtillot, V., J. Ducruix, and J.-L. Le Mouël, Sur une accélération récente de la variation séculaire du champ magnétique terrestre, C. R. Acad. Sci. D}, 287}, 1095–1098, 1Google Scholar
- De Boor, C., A Practical Guide to Splines, Springer-Verlag, New York, 1978.View ArticleGoogle Scholar
- Friis-Christensen, E., H. Lühr, and G. Hulot, Swarm: A constellation to study the Earth’s magnetic field, Earth Planets Space, 58, 351–358, 2006.View ArticleGoogle Scholar
- Holme, R., Large scale flow in the core, in Treatise on Geophysics, vol. 8, edited by P. Olson and G. Schubert, pp. 107–130, Elsevier, 2007.View ArticleGoogle Scholar
- Hulot, G., T. J. Sabaka, and N. Olsen, The present field, in Treatise on Geophysics, vol. 5, edited by M. Kono and G. Schubert, pp. 33–75, Elsevier, 2007.Google Scholar
- Jackson, A. and C. C. Finlay, Geomagnetic secular variation and its applications to the core, in Treatise on Geophysics, vol. 5, edited by M. Kono and G. Schubert, pp. 147–193, Elsevier, 2007.View ArticleGoogle Scholar
- Jankowski, J. and C. Sucksdorff, Guide for Magnetic Measurements and Observatory Practice, International Association of Geomagnetism and Aeronomy, Warsaw, 1996.Google Scholar
- Lesur, V., I. Wardinski, M. Rother, and M. Mandea, GRIMM: the GFZ Reference Internal Magnetic Model based on vector satellite and observatory data, Geophys. J. Int., 173, 382–394, 2008.View ArticleGoogle Scholar
- Macmillan, S. and S. Maus, International Geomagnetic Reference Field—the tenth generation, Earth Planets Space, 57, 1135–1140, 2005.View ArticleGoogle Scholar
- McPherron, R. L., The utilization of ground magnetometer data in magnetospheric physics, in Proceedings of the XIIIth IAGA Workshop on geomagnetic observatory instruments, data acquisition and processing, edited by J. J. Love, U.S. Geological Survey Open-File Report 2009, pp. 171–189, 2009.Google Scholar
- Olsen, N., M. Mandea, T. J. Sabaka, and L. Tøffner-Clausen, CHAOS—2—A geomagnetic field model derived from one decade of continuous satellite data, Geophys. J. Int., 179, 1477–1487, 2009.View ArticleGoogle Scholar