- Technical report
- Open Access
Progress of the IUGONET system - metadata database for upper atmosphere ground-based observation data
© Abe et al.; licensee Springer. 2014
- Received: 31 March 2014
- Accepted: 24 September 2014
- Published: 14 October 2014
The Interuniversity Upper atmosphere Global Observation NETwork (IUGONET) project is a 6-year research project which started in 2009. The objective of this project is to establish a metadata database of various ground-based observation data covering a wide region from the Sun to the Earth; this will encourage more studies on the mechanisms of long-term variations in the upper atmosphere.
For archiving purposes, the metadata database system for cross-searching various data distributed across many universities and institute was developed based on the existing repository software called DSpace as the core component and the Space Physics Archive Search and Extract (SPASE) data model as the metadata format. The IUGONET metadata database is still in operation since it was released in March 2012. The system is continuously examined, tested, and updated to improve its quality. The OpenSearch interface in the IUGONET metadata database allows the user to use external applications easily for exchanging metadata and/or for analyzing data.
We conducted self-examination of our product, which was added for planning future directions of the IUGONET project.
- Metadata database
- Analysis software
- Upper atmospheric physics
- Earth and planetary sciences
- Ground-based observation
In order to understand long-term changes in the Earth's atmosphere, it is essential to discuss the various atmospheric layers as a coupled system and not to regard them as separate layers. The upper atmosphere, the focus of this paper, is defined as the region above about 50 km altitude and consists of six layers, namely the mesosphere, thermosphere, ionosphere, plasmasphere, magnetosphere, and heliosphere. This region is affected by the input of materials, momenta, and energies from the upper region (e.g., ultraviolet radiation from the sun and the electromagnetic energy from the solar wind) and from the lower region (e.g., atmospheric waves from the stratosphere and troposphere). In addition to the vertical coupling processes, it is also important to consider the meridional coupling in the region that covers the equatorial, low, middle, and high latitudes.
The upper atmosphere is characterized by the coexistence of both ionized plasma and neutral gas and also by the drastic changes in the physical quantities across the layers (i.e., density, pressure, temperature, etc.). To clarify the physical mechanisms of the phenomena in the upper atmosphere, therefore, it is necessary to comprehensively analyze various types of physical quantities observed in the multiple layers. However, it is often difficult for researchers of different fields to get from a single source the information of the observed data, for example, physical quantities, instruments, observatories, contact persons, location, and format of data files.
This paper particularly focuses on the metadata database system developed by IUGONET. Background of the IUGONET metadata database system describes the fundamental policy. In the ‘Operation and improvement of the IUGONET metadata database system’ section, the daily operation, maintenance of the system, some evaluations of our product, and the conjunction with data analysis software are described. In the ‘Discussion and future efforts’ and ‘Conclusions’ sections are the discussion and summary, respectively.
Background of the IUGONET metadata database system
To avoid problems like ownership of data, authentication, and authorization, the observational data is managed without a central server. The metadata database is built as a virtual integrated database environment to share the metadata of ground-based observational data including the uniform resource locator (URL) of data file. However, problems were faced before each organization released the data to the public, such as lack of human resources for the implementation of the database and the development members' composition mostly inclined toward the specialists in upper atmospheric physics only.
Another problem was the project timeline. The project timeline was not based on the detailed plan of the system designer. Therefore, there was not enough time to design and implement a metadata database system suitable for natural science.
Under such restrictions, we adopted DSpace (DuraSpace, DSpace, http://www.dspace.org/, Accessed 5 Oct 2014), a free software, which wrapped Apache httpd, PostgreSQL, Tomcat, etc., as a metadata database system. We also paid attention to the following minimum requirements to build the metadata database system. Easy technical information sharing on system installation, customization, and management is one of the reasons for DSpace adaptation. The total number of DSpace worked as an institutional repository around the world is about 2,500 (DuraSpace, DSpace User Registry, http://registry.duraspace.org/registry/dspace/, Accessed 5 Oct 2014) and in Japan it is about 300 (National Institute of Informatics, NII Institutional Repositories Program, http://www.nii.ac.jp/irp/list/, Accessed 5 Oct 2014). Most institutional repositories are managed by university libraries. Therefore, the institutional repository is also managed in the library of the institution which has participated in the IUGONET project except for NIPR. High interoperability is also one of the reasons for DSpace adaptation. DSpace supports the common interoperability standards used in Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) (Open Archives Initiative, The Open Archives Initiative Protocol for Metadata Harvesting, http://www.openarchives.org/OAI/2.0/openarchivesprotocol.htm, Accessed 5 Oct 2014), Search/Retrieve via URL/Search/Retrieve Web Service (SRU/SRW) (Library of Congress, Search/Retreval via URL, http://www.loc.gov/standards/sru/, Accessed 5 Oct 2014), OpenSearch (A9.com, Inc., OpenSearch, http://www.opensearch.org/Home, Accessed 5 Oct 2014), etc. These web application programming interfaces (APIs) are compatible with external systems like databases and data analysis software. Scalability is another reason for DSpace adaptation. If the metadata records become large, it is difficult to deal with them by a single server. Therefore, we are investigating the system structure again (e.g., distributed model). The cross-searching using external interface, such as OpenSearch, makes it possible to form multi-database connections like the relationship between National Diet Library (NDL) Search (National Diet Library, NDL Search, http://iss.ndl.go.jp/, Accessed 5 Oct 2014) and Institutional Repositories in Japan (DuraSpace, DSpace User Registry, http://www.nii.ac.jp/irp/list/, Accessed 5 Oct 2014). The IUGONET metadata database is running under DSpace 1.7.0.
Concerning the metadata format, there was not enough time to define an original metadata format for the project. Several existing metadata formats were investigated, and as a result, the Space Physics Archive Search and Extract (SPASE) was chosen for the base metadata format (Thieman, J. R., Welcome to the SPASE Group, http://www.spase-group.org/, Accessed 5 Oct 2014). SPASE is suitable for the IUGONET project because it is closely related to Solar Terrestrial Physics (STP) and upper atmosphere researches. In addition, SPASE has a scalability format. We can append new metadata elements and terms for our data. In fact, we appended some modifications to the SPASE format, for example, additional terms to represent non-digital archives, additional terms to represent heliospheric coordinates, and new metadata elements to describe observation location and range. SPASE is written in XML format, but DSpace 1.7.0 cannot handle XML format directly. To solve this problem, a SPASE Dublin-Core converter was developed for the IUGONET metadata database (see ‘Operation and improvement of the IUGONET metadata database system’ section).
Operation and improvement of the IUGONET metadata database system
Routine operation and maintenance of the system
Since February, 2011, the main system and stand-by system of the IUGONET metadata database have been running at Kyushu University and Nagoya University, respectively. The metadata XML files provided by the IUGONET members are also stored in these universities.
Our metadata database has functions of browsing with an internet browser, and of XML interfacing with external programs (see ‘Performance evaluation and its improvement’ section). The browsing is done as follows: (1) open the metadata search page using the internet browser; (2) search the data by specifying the metadata type, keywords, observation date and time, observation location (latitude and longitude), and so on; and (3) select one item from the search results to obtain details of the metadata. The details of the metadata include description of data, instrument, observation location (latitude and longitude), location of the data files (URL), contact person, data usage policy, etc., so it is possible for users to not only get the information but also download data files from the remote data server.
Listed below are the examples of search terms used in the metadata database by the users. The search words include some terms related to satellites, planets, materials, etc., as well as many terms related to the Earth's upper atmosphere, which implies that the IUGONET has a potential to extend to cover various fields of science other than the upper atmosphere.
Examples of search terms used in the metadata database by the users:
Earth's upper atmosphere
MF radar, Super DARN, MAGDAS, EISCAT, smart, magnetogram, dst, aurora, ionosphere, geomagnetic field, etc.
ceilometers, electron, ozone, X-ray, Jupiter, climate, CO2, O3, GOES, cloud, carbon, etc.
Performance evaluation and its improvement
Cooperation with data analysis software
As mentioned in the ‘Background of the IUGONET metadata database system’ section, IUGONET metadata database is available not only by internet browser (http://search.iugonet.org/iugonet/) but also by using external applications. The metadata database accommodates queries with OpenSearch and act as a back-end of applications. The external programs can utilize the metadata database as follows: (1) make a URL that has a query including the search parameters, (2) search data by GET method in the HTTP protocol, (3) get the search result in the XML format, (4) parse the XML file to obtain the necessary information, and (5) use the information for visualizing and analyzing data. The following is an example of the format for OpenSearch query: http://search.iugonet.org/iugonet/open-search/request?(parameter=value)&(parameter=value)&…&(). The query terms that could be used are described in http://www.iugonet.org/en/opensearch.html. When the database receives a query, the database returns the XML file which includes all elements of appropriate metadata for ATOM1.0 format.
Parts of the routine in iUgonet Data Analysis Software (UDAS) refer the metadata database to get some information by OpenSearch. For example, the routine for loading the Solar Magnetic Active Research Telescope (SMART) data gets the URL of the data file from the metadata database. For the other load routines, the URL of the data files is hard-coded in the routines, so we need to modify the load routines whenever the file location changes In the case of the load procedure for SMART telescope, the procedure to change the URL is not necessary. Only an update of the metadata for the URL of the database is needed, and the change will be reflected immediately.
We also provide some procedures (also included in UDAS) to get information of observatories, or plot the location of observatories on the map using ‘latitude’ and ‘longitude’ elements of ‘Observatory’ metadata on the metadata database. The number of registered observatories is still increasing, and thus, it is good to refer the metadata database instead of including it in UDAS.
Discussion and future efforts
One of the purposes of IUGONET is the promotion of cross-cutting research. Therefore, it is important that IUGONET metadata database is used in various research fields. In order to achieve the above purpose, it is necessary to provide users a method to operate our metadata database in their own server. In addition, the products developed on our project may be no longer used and/or maintained since the IUGONET project is scheduled to end. To avoid such a situation, it is necessary to open our software to the general public. Therefore, we developed some support software which can assist to construct and manage our IUGONET metadata database. In addition, we put our working products in the shared web service of the Internet. We use a hosting service called GitHub for the software development project. By using GitHub, our operational costs can be drastically reduced. In addition, any user can try to install and operate our product via GitHub. The IUGONET metadata database is already used outside the IUGONET institute, for example, as a metadata management system of the imager and medium frequency (MF) radar of National Institute of Information and Communications Technology. Furthermore, our metadata database is also considered as a base model for managing radiation data of Fukushima Prefecture. These result shows our product can be expected to be accepted as a of the cross-cutting research system.
In recent days, discussions about open data are increasing in many research fields. The STP community which almost all IUGONET institutes belong to is no exception. One of the topics of open data is an approach towards data citation, for example, appending the digital object identifier (DOI) to data. It is natural that the IUGONET rides this worldwide flow which promotes to utilize the data. In order to deal with this framework on the IUGONET metadata database, we are trying to renew a metadata schema for IUGONET common metadata format. Moreover, in this update, we are considering reexamination of a namespace in the metadata schema. This renewal gives the IUGONET metadata format XML schema the interoperability and compatibility and contributes to the advancement of IUGONET metadata database.
In order to confirm what kind of contribution the IUGONET metadata database has made to the community until now, we interviewed five institutes (seven organizations) inside the IUGONET. As a result, we found that our activities are widely respected for the quality of metadata archive system, for example, as a starting point for data search. On the other hand, many users request us to support new datasets, such as satellite data, and to improve data analysis functions. We will fulfill these requests in the next phase of the IUGONET project. In addition, we understand the need to improve the function of data visualization and associative searching for beginners, which a part of the functions can be ready in our system, in the near future.
In this paper, we have discussed the progress and the future vision of IUGONET metadata database. To develop a system for the upper atmosphere data from ground-based observation accumulated over 50 years since the first IGY by Japanese universities/institutes, and accelerate cross-cutting researches by using the system, we released the metadata database. It was a big challenge in our communities. The system is based on DSpase software and SPASE metadata format. We examined some evaluations of our product and made numerous improvements in it. One of the applications in our system is a linkage of data analysis software. For scientists in natural science, data visualization is an important basic tool for their researches. Our product can support their requests by several methods. Our reliable self-assessment helps to improve our product and define actions for future efforts.
Project name: Inter-university Upper atmosphere Global Observation NETwork (IUGONET) project
Project home page: http://www.iugonet.org/
Operating system(s): Linux
Programming language: Java, Ruby
Other requirements: DSpace, Apache, Tomcat, PostgreSQL
License: BSD licence
Any restrictions to use by non-academics: none
We thank all IUGONET institutes and members. We wish to express our gratitude especially to the representatives of each institute, Professor Natsuo Sato and Takuji Nakamura of NIPR; Takayuki Ono and Takahiro Obara of Tohoku University; Ryoichi Fujii, Tatsuki Ogino, and Kazuo Shiokawa of Nagoya University; Toshitaka Tsuda, Toshihiko Iyemori, and Kazunari Shibata of Kyoto University; and Kiyohumi Yumoto, Tohru Hada, and Akimasa Yoshikawa of Kyushu University.
- Hayashi H, Koyama Y, Hori T, Tanaka Y, Abe S, Shinbori A, Kagitani M, Kouno T, Yoshida D, UeNo S, Kaneda N, Yoneda M, Umemura N, Tadokoro H, Motoba T, IUGONET project team: Inter-university Upper Atmosphere Global Observation Network (IUGONET). Data Sci J 2013, 12: WDS179-WDS184. doi:10.2481/dsj.WDS-030 doi:10.2481/dsj.WDS-030View ArticleGoogle Scholar
- Tanaka Y-M, Shinbori A, Hori T, Koyama Y, Abe S, Umemura N, Sato Y, Yagi M, UeNo S, Yatagai A, Ogawa Y, Miyoshi Y: Analysis software for upper atmospheric data developed by the IUGONET project and its application to polar science. Adv Polar Sci 2013, 24: 231–240. doi:10.3724/SP.J.1085.2013.00231 doi:10.3724/SP.J.1085.2013.00231View ArticleGoogle Scholar
- Olsen N, Friis-Christensen E, Floberghagen R, Alken P, Beggan CD, Chulliat A, Doornbos E, da Encarnação JT, Hamilton B, Hulot G, van den IJssel J, Kuvshinov A, Lesur V, Lühr H, Macmillan S, Maus S, Noja M, Olsen PEH, Park J, Plank G, Püthe C, Rauberg J, Ritter P, Rother M, Sabaka TJ, Schachtschneider R, Sirol O, Stolle C, Thébault E, Thomson AWP, et al.: The Swarm Satellite Constellation Application and Research Facility (SCARF) and Swarm data products. Earth Planets Space 2013, 65(11):1189–1200. 10.5047/eps.2013.07.001View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.