Observatory data and the Swarm mission
© The Society of Geomagnetism and Earth, Planetary and Space Sciences (SGEPSS); The Seismological Society of Japan; The Volcanological Society of Japan; The Geodetic Society of Japan; The Japanese Society for Planetary Sciences; TERRAPUB 2013
Received: 14 January 2013
Accepted: 29 July 2013
Published: 22 November 2013
The ESA Swarm mission to identify and measure very accurately the different magnetic signals that arise in the Earth’s core, mantle, crust, oceans, ionosphere and magnetosphere, which together form the magnetic field around the Earth, has increased interest in magnetic data collected on the surface of the Earth at observatories. The scientific use of Swarm data and Swarm-derived products is greatly enhanced by combination with observatory data and indices. As part of the Swarm Level-2 data activities plans are in place to distribute such ground-based data along with the Swarm data as auxiliary data products. We describe here the preparation of the data set of ground observatory hourly mean values, including procedures to check and select observatory data spanning the modern magnetic survey satellite era. We discuss other possible combined uses of satellite and observatory data, in particular those that may use higher cadence 1-second and 1-minute data from observatories.
Magnetic observatories carry out continuous and accurate monitoring of the strength and direction of the Earth’s magnetic field over many years, making measurements at least every minute. Observatory data reveal how the field is changing on a wide range of time scales from seconds to centuries, and this is important for understanding processes both inside and outside the Earth. There are approximately 160 observatories currently operating around the world. The distribution of observatories is largely determined by the location of habitable land and by the availability of local expertise, funds, data transmission infrastructure and energy supply, and as result, it is uneven and somewhat sparse in the southern hemisphere and oceanic regions.
Geomagnetism is a cross-disciplinary science, and as a result, observatories are run by a wide variety of institutes whose interests range from geology, mapping, geophysics (including seismology and earthquake prediction), meteorology to solar-terrestrial physics and astronomy. Nonetheless, strong networks have been established by staff in these institutes involved in the operation of observatories, for example the biannual IAGA observatory workshops, INTER-MAGNET and the World Data Centres for Geomagnetism. The ESA Swarm mission will take advantage of these networks.
Processing satellite magnetic observations often initially involves their selection on the basis of data obtained by ground observatories, usually in the form of geomagnetic indices like Kp and Dst. Also, many Swarm L2 products are derived using observatory hourly mean data and/or indices and some require observatory data products for their validation. Thus observatory data products are used in satellite data selection, L2 product derivation and validation. We describe briefly how the Kp and Dst index data sets are prepared and then give details of the quality control measures we found were required for the recent historic hourly mean data sets. Efforts to improve the real-time availability of absolute observatory data are then discussed, before a short section with some concluding remarks.
2. Preparation of Observatory Indices
Kp is a 3-hour range index describing global geomagnetic activity (e.g. Siebert and Meyer, 1996). It is routinely derived by GFZ Helmholtz Centre in Potsdam using 1-minute data from 13 observatories. The Dst index monitors the strength of the axisymmetric part of the magnetospheric ring current (e.g. Sugiura and Kamei, 1991) and is determined using hourly mean values from 4 observatories by the World Data Centre (WDC) for Geomagnetism at Kyoto. Both indices are reformatted and delivered by GFZ Helmholtz Centre for distribution through the ESA Payload Data Ground Segment (PDGS). Kp and its linear equivalent ap is provided with time given in Modified Julian Day 2000 (as is the case for most Swarm AUX data products) in the auxiliary product with filename similar to SW_OPER_AUX_KP__2__ YYYYMMDDThhmmss_YYYYMMDDThhmmss_VVVV.DBL where “YYYYMMDDThhmmss” stands for time stamp of first, respectively last, data point in the file and “VVVV” is a four digit version number. Data for January 1st 1998 to January 31st 2011 are for instance available in the file SW_OPER_AUX_KP__2__19980101T000000 20110201T000000 0001.DBL.Dst and its decomposition into magnetospheric (external) and internally induced parts using a 1-D model of electrical conductivity of the Earth’s mantle (e.g. Maus and Weidelt 2004; Olsen et al. 2005) are provided as time series of 1 hour sampling rate in the auxiliary data product with filename similar to SW_OPER_AUX_DST_2__YYYYMMDDThhmmss_ YYYYMMDDThhmmss_VVVV.DBL.
3. Preparation of Observatory Hourly Mean Data
Observatory data are extracted from data holdings at the WDC for Geomagnetism at Edinburgh. This WDC, and the Kyoto WDC, are part of the ICSU (International Council for Science) World Data Centre System and ensures the long-term availability of geomagnetic data for research. Although data quality checks have been made on incoming data over the years, and feedback provided to the data suppliers, the WDC is ultimately a depository. It therefore contains data of varying quality and use of these data must be preceded by a selection procedure. In preparation of the Swarm mission, and in order to support the analysis of the pre-Swarm satellite data (in particular Ørsted and CHAMP) we provide a consistent data set of quality-checked observatory hourly mean values for the years 1997 and onwards. When Swarm is launched these data will be augmented with quasi-definitive data mainly from INTERMAGNET observatories. In later releases of the observatory hourly mean data series these quasi-definitive data will be replaced by definitive data, once available. INTER-MAGNET is an organisation involved in promoting standards in digital observatory operations around the world and has facilitated the improvement in observatory data quality since the early 1990s. More information can be found at www.intermagnet.org
For all new data received for assimilation into the WDC holdings, quality control (QC) procedures are undertaken. A range of computer programs is used for this QC, the main one being the INTERMAGNET data viewer program imcd-view developed by BGS. Values are then compared with a global model to check for gross errors such as incorrect signs.
contain minimal measurement noise
be as complete as possible (no data gaps in time)
be corrected to absolute values over multi-year periods, i.e. drift-free
be without discontinuities
comply with an agreed data format.
in the same coordinate frame as the satellite data
time- and position-stamped like the satellite data.
The following 29 observatories have some data spanning one or more years during the period 1997–2010 which are deemed of insufficient quality and are not used: AAA, ABG, API, ARS, BGY, BMT, BNG, CNH, DLR, ELT, HBK, IQA, KNZ, KSH, LVV, MIZ, MOS, NGP, PHU, PST, SIL, THY, TIR, TND, TSU, VNA, VSK, VSS and WHN.
Five observatories (API, BOX, KIR, MCQ and TEO) have typographical errors in their data during the period 1997–2010 which were corrected during this project. These typographical errors are incorrect flag values and are corrected in the data available from the WDC. A note is added to this effect in the observatory metadata and the original files are retained internally by the WDC.
The following 7 observatories have some data during the period 1997–2010 which are flagged as erroneous (generally spanning a few days, sometimes just one component): ARS, BGY, HLP, LRM, PAG, SFS and TIR.
Data from all observatories for one year are provided in one file with filename similar to SW_OPER_AUX_OBS_2__ YYYYMMDDThhmmss YYYYMMDDThhmmss VVVV.DBL (data for the year 2015 will for instance be made available in the file SW_OPER_AUX_OBS_2__20150101T003000 20151231T233000_0001.DBL). During the operational phase of the Swarm satellite mission the data will be updated every 3rd month and will be distributed by the ESA PDGS similar to all other Swarm data products, and also directly from ftp://ftp.nerc-murchison.ac.uk/geomag/smac/AUX_OBS_2/. The TEST (replacing OPER in the names above) files for 1997–2010 containing the data presented in this paper are available from this ftp address.
4. Hour-by-Hour Spherical Harmonic Analysis
As is often the case with scientific data QC, it is the detailed scientific analysis that uncovers remaining problems in the data. This is particularly the case with geomagnetic data where it is difficult to detect small discontinuities, spikes and drifts over and above the natural magnetic field variations. These variations can be both very small e.g. annual secular variation and oceanic signals, and large, e.g. during magnetic storms. For the observatory hourly means further problems were discovered by fitting hour-by-hour spherical harmonic models to the data with a priori estimates of known signals removed.
The core and large-scale crustal signal was first removed using the latest in the CHAOS series of models (Olsen et al., 2010). The ionospheric (primary and induced) contributions as predicted by CM4 (Sabaka et al., 2004) were then removed. The remaining crustal field was determined and removed by subtracting the mean of the local nighttime values during geomagnetic quiet periods (Kp < 2+, ǀdDst/dtǀ < 2 nT/hr). The data were then rotated into a dipole coordinate system and hour-by-hour robust spherical harmonic models up to degree 9 and order 1 were fitted to the horizontal and vertical data separately. This procedure does not produce realistic magnetospheric field models but it does help reduce any potential and non-potential signals that are coherent in time and space in particular regarding neighbouring sites. The time series of the residuals to these models are then plotted in order of dipole latitude.
Figure 5 shows residuals from observatories near the dipole equator and at low dipole latitudes in the southern hemisphere. The many gaps in Fig. 5 demonstrate that a considerable amount of data has already been rejected, or are missing. Remaining indicators of data problems include a curious drift over several years in AAE1 (Addis Ababa, Ethiopia), a spike in VSK1 (Visakhapatnam, India) in 2006, noise in PPT0 (Pamatai, French Polynesia) during 1997–99, drift/step in PPT0 during 2000 and 2002, drift in API0 (Apia, Western Samoa) during 2007 and a drift in HBK0 (Hartebeesthoek, South Africa) during 2005.
5. Quasi-Definitive Data
One difficulty in using observatory data for producing models of the Earth’s magnetic field is that there is often a long delay till the final definitive data are released. The reasons for this vary from one institute to another and from one observatory location to another and may be related to the presence of the afore-mentioned data quality issues, lack of man-power and the tradition of annual processing for the production of yearbooks. Often, by the time the definitive data are released, they have passed their period of potential peak usefulness. In contrast, satellite data are often available after a few days or weeks after acquisition. A time delay in the availability of observatory data therefore hampers the optimal use of these data in combination with most recent satellite observations.
6. Concluding Remarks
In this paper we have concentrated on hourly mean data from observatories because the amount of data that have to be processed and analysed simultaneously with satellite data is easier to handle compared e.g. to using one-minute data. We have described the additional quality control measures we found necessary for the observatory hourly means spanning the current magnetic survey satellite era, the format of the files and where online they will be made available with a 3-month update cycle.
It should be noted that the standard product from observatories is one-minute mean data, with work underway at many observatories to move to one-second data. One-minute data are available directly from the World Data Centres in Edinburgh (www.wdc.bgs.ac.uk), Kyoto (wdc.kugi.kyoto-u.ac.jp), Moscow (www.wdcb.ru) and Boulder (spidr.ngdc.noaa.gov) and from INTERMAGNET (www.intermagnet.org). The problems we have found in the hourly mean values (which are based on minute values) are very likely also present in data of higher temporal resolution, and once hourly mean values have been flagged as erroneous or data jumps and spikes have been identified it should be possible to correct the one-minute data.
One-second data are becoming more widely available now with much development in sensors, digitisers and in data distribution. With the Swarm satellite instruments sampling the field at 1 Hz and higher, and with many interesting high frequency signals to be analysed and understood, it seems likely that these higher cadence observatory products will be very useful.
The following organisations are thanked for supporting the observatories considered in this study: Centre de Recherche en Astronomie Astrophysique et Geophysique, ALGERIA; Servicio Meteorologico Nacional, ARGENTINA; Universidad Nacional de la Plata, ARGENTINA; Geoscience Australia, AUSTRALIA; Zentralanstalt fur Meteorologie und Geodynamik, AUSTRIA; Institut Royal Meteorologique de Belgique, BELGIUM; CNPq-Observatorio Nacional, BRAZIL; Academy of Sciences, BULGARIA; Geological Survey of Canada, CANADA; Academy of Sciences, CHINA; Seismological Bureau, CHINA; Directorate General of Telecommunications, CHINA (Taiwan); Instituto Geografico Agustín Codazzi, COLOMBIA; Academy of Sciences, CZECH REPUBLIC; Technical University of Denmark, DENMARK; Addis Ababa University, ETHIOPIA; Finnish Meteorological Institute, FINLAND; Sodankyla Geophysical Observatory, FINLAND; Institut de Physique du Globe de Paris, FRANCE; Ecole et Observatoire des Sciences de la Terre, FRANCE; Institut Francais de Recherche Scientifique pour le Developpement, FRANCE; Academy of Sciences, GEORGIA; Ludwig Maximilians University Munich, GERMANY; Alfred-Wegener-Institute for Polar and Marine Research, GERMANY; GeoForschungsZentrum Helmholtz Zentrum in Potsdam, GERMANY; Universities of Karlsruhe and Stuttgart, GERMANY; Institute of Geology and Mineral Exploration, GREECE; Academy of Sciences, HUNGARY; Eotvos Lorand Geophysical Institute of Hungary, HUNGARY; University of Iceland, ICELAND; Indian Institute of Geomagnetism, INDIA; Meteorological and Geophysical Agency, INDONESIA; The Irish Meteorological Service, IRELAND; Survey of Israel, ISRAEL; Instituto Nazionale di Geofisica e Vulcanologia, ITALY; Japan Coast Guard, JAPAN; Japan Meteorological Agency, JAPAN; Geographical Survey Institute, JAPAN; Institute of the Ionosphere, KAZAKHSTAN; National Centre for Geophysical Research, LEBANON; Universite d’Antananarivo, MADAGASCAR; Universidad Nacional Autonoma de México, MEXICO; Institute of Geological and Nuclear Sciences, NEW ZEALAND; University of Tromsø, NORWAY; Instituto Geofísico del Peru, PERU; Academy of Sciences, POLAND; Instituto Nacional de Geologia, REPUBLICA DE MOCAMBIQUE; Geological Survey of Romania, ROMANIA; Academy of Sciences, RUSSIA; Institute of Solar-Terrestrial Physics, RUSSIA; Dept. of Agriculture, Forestry, Fisheries and Meteorology, SAMOA; Geomagnetic College Grocka, SERBIA and MONTENEGRO; Slovenska Akademia Vied, SLOVAKIA; National Research Foundation, SOUTH AFRICA; Observatori de l’Ebre, SPAIN; Real Instituto y Observatorio de la Armada, SPAIN; Instituto Geografico Nacional, SPAIN; Sveriges Geol-ogiska Undersokning, SWEDEN; Swedish Institute of Space Physics, SWEDEN; Earthquake Research Institute, TURKEY; US Geological Survey, UNITED STATES OF AMERICA; British Geological Survey, UNITED KINGDOM; Academy of Sciences, UKRAINE; Ukrainian Antarctic Center, UKRAINE and National Centre for Science and Technology, VIETNAM. The following companies are also thanked for their support of JCO and SBL observatories: Halliburton, BP and Sable Offshore Energy. This paper is published with the permission of the Executive Director of the British Geological Survey (Natural Environment Research Council).
- Maus, S. and P. Weidelt, Separating the magnetospheric disturbance magnetic field into external and transient internal contributions using a 1D conductivity model of the Earth, Geophys. Res. Lett., 31(12), L12614, doi:10.1029/2004GL020232, 2004.View ArticleGoogle Scholar
- Olsen, N., T. J. Sabaka, and F. Lowes, New parameterization of external and induced fields in geomagnetic field modeling, and a candidate model for IGRF 2005, Earth Planets Space, 57, 1141–1149, 2005.View ArticleGoogle Scholar
- Olsen, N., M. Mandea, T. J. Sabaka, and L. Tøffner-Clausen, The CHAOS-3 geomagnetic field model and candidates for the 11th Generation IGRF, Earth Planets Space, 62, 719–727, 2010.View ArticleGoogle Scholar
- Peltier, A. and A. Chulliat, On the feasibility of promptly producing quasi-definitive magnetic observatory data, Earth Planets Space, 62(2), e5–e8, doi:10.5047/eps.2010.02.002, 2010.View ArticleGoogle Scholar
- Sabaka, T. J., N. Olsen, and M. E. Purucker, Extending Comprehensive Models of the Earth’s magnetic field with Ørsted and CHAMP, Geophys. J. Int., 159(2), 521–547, 2004.View ArticleGoogle Scholar
- Siebert, M. and J. Meyer, Geomagnetic Activity Indices, in The Upper Atmosphere, edited by W. Dieminger, G. K. Hartmann, and R. Leitinger, pp. 887–911, Springer, 1996.Google Scholar
- St-Louis, B. J. (ed), INTERMAGNET Technical Reference Manual, version 4.5, INTERMAGNET, http://www.intermagnet.org/publications/immanual.pdf (accessed 30-04-13), 2011.
- Sugiura, M. and T. Kamei, Equatorial Dst index, 1957–1986, in IAGA Bulletin 40, edited by Berthelier, A. and M. Menvielle, ISGI Publ. Off. Saint Maur, 1991.Google Scholar