Performance assessment of radio occultation data from GeoOptics by comparing with COSMIC data

Abstract

Responding to the ever-growing demand for environmental information, the National Oceanic and Atmospheric Administration (NOAA) seeks to enter into contracts to purchase Global Navigation Satellite System (GNSS) radio occultation (RO) observations produced by commercial vendors at a low-cost. GeoOptics is one commercial vendor awarded a contract with NOAA. GeoOptics operates the Community Initiative for Cellular Earth Remote Observation (CICERO) constellation of low-earth-orbiting (LEO) 6U CubeSats. The 6U-sized CICERO will enable the deployment of GNSS array consisting of RO satellites in the Earth’s atmosphere to obtain many atmospheric observations which can improve weather forecasting. Applying GeoOptics RO data to reliable weather forecasting requires an assessment of its performance. This study analyzes the performance of GeoOptics CubeSats measurements by comparing it with the Constellation Observing System for Meteorology, Ionosphere, and Climate (COSMIC) missions (COSMIC-1 and COSMIC-2). The performance analysis was carried on data coverage capabilities and measurement quality. The analysis of data coverage confirmed that GeoOptics can acquire global observational coverage with adequate low-altitude penetration capability, while there should be updated in local time coverage. The analysis of RO measurement quality showed that GeoOptics RO measurements are comparable to those of COSMIC-2, even though GeoOptics exhibited a lower signal-to-noise ratio (SNR). The potential of GeoOptics allows for the development of a GNSS array in the Earth’s atmosphere and a large amount of effective RO measurements to be obtained for reliable weather forecasting.

Graphical Abstract

Introduction

As Global Navigation Satellite System (GNSS) radio occultation (RO) has been extensively developed, numerical weather prediction (NWP) models for operational forecasts have correspondingly improved (Kuo et al. 2000; Kursinski et al. 1997; Rocken et al. 1997; Wickert et al. 2001). GNSS RO is a remote sensing technique which allows the Earth’s weather parameters to be obtained through GNSS signals received by low-earth-orbit (LEO) satellites. As the GNSS signal passes through the atmosphere, the signal is delayed and its ray is bent. By measuring the signal delay and bend, GNSS RO provides fundamental measurements of key parameters including bending angle, refractivity, temperature, pressure, and water vapor; all are required as input for climate modeling and numerical weather forecasting (Jasper et al. 2013). GNSS RO measurements provide the global coverage, high vertical resolution and all-weather capabilities. GNSS RO can potentially augment and improve data availability, thereby advancing the weather forecasting and climate research. The high vertical resolution of GNSS RO ionospheric data is also useful to study space weather and ionospheric research; not only long-term global ionospheric climatology (Yue et al. 2012), but also regional ionospheric disturbances, such as traveling ionospheric disturbances (TIDs) and equatorial plasma bubbles (EPBs) (Yoon and Lee 2014; Yoon et al. 2017; Kepkar et al. 2020).

The first Constellation Observing System for Meteorology, Ionosphere, and Climate (COSMIC-1) was the radio occultation mission launched in April 2006 which provided a large increase in the number of GNSS RO observations. Each satellite of the COSMIC-1 constellation carries an Integrated GPS Occultation Receiver (IGOR) (Anthes et al. 2008). The mission provides high-quality RO profiles that are having a significant positive impact on global NWP forecasts (Ho et al. 2020a). Although the COSMIC-1 constellation has lasted much longer than expected, over time the number of observations received from COSMIC-1 have slowly decreased as the satellites have encountered operational problems. COSMIC-2 is a follow-on mission to COSMIC-1 which was launched on June 2019. The COSMIC-2 constellation consists of six LEO satellites with next-generation GNSS RO payloads, including a Tri-GNSS RO system (TGRS) (Schreiner et al. 2020; Tien et al. 2012). COSMIC-2 will support improved weather forecasts by providing high-quality RO profiles and more soundings penetrating into the lower troposphere (Schreiner et al. 2020; Ho et al. 2020b).

Abundant GNSS RO observations can lay the foundation toward more accurate and reliable NWP models. As the demand for more accurate NWPs has intensified, some studies have attempted to increase the number of GNSS RO observations (Bauer et al. 2014; Harnisch et al. 2013; Tseng et al. 2018; Wang and Liang 2017). Moreover, additional RO observations can provide risk mitigation for any unexpected RO sounding shortage issue. A larger number of GNSS RO observations can potentially be obtained by using a larger number of LEO satellites equipped with GNSS RO receivers (Xu et al. 2019). However, the corresponding development cycles and costs strongly limit the number of LEO satellites for RO applications.

Recently, several private companies have started providing the RO data from CubeSats. The National Oceanic and Atmospheric Administration (NOAA) is interested in purchasing commercial RO sounding data from private companies through the Commercial Weather Data Pilot (CWDP) program and assessing their potential. GeoOptics is one commercial weather data vendor that provides RO data for NOAA’s operational NWP models (Chen et al. 2021; National Oceanic and Atmospheric Administration 2020). GeoOptics operates a new RO constellation of nanosatellites, which is called the Community Initiative for Cellular Earth Remote Observation (CICERO). The CICERO was deployed to demonstrate the feasibility of obtaining quality RO data using low-cost CubeSats in 2017.

Unlike early high-end GNSS receivers, such as IGOR in COSMIC-1 and TGRS in COSMIC-2 (Turk et al. 2019), each CICERO CubeSat carries relatively low-cost, low-power, and low-mass receivers called CIONs (Franklin et al. 2018). The CION was developed by Jet Propulsion Laboratory (JPL) and represents a miniature GNSS receiver hosting a 2 × 3 patch antenna assembly. CION, with a mass of 1.2 kg, fits within a 30 × 10 × 6-cm envelope. The miniaturization of the GNSS RO receiver allows a CICERO payload to fit into a 6U CubeSat. By taking advantage of CubeSat, a dense GNSS array of RO satellites can be launched to acquire massive quantities of occultation data at low costs.

This study aims at evaluating the performance of RO data from GeoOptics by comparing it with that from COSMIC-1 and COSMIC-2 to elucidate its applicability for accurate numerical weather forecasting. Previously, there have been several studies on performance assessment of GeoOptics RO data (Chen et al. 2021; National Oceanic and Atmospheric Administration 2020). Chen et al. 2021 presented the comparison results between GeoOptics and collocated COSMIC-2 RO observations in terms of bending angle retrieval uncertainty, penetration depth, and signal-to-noise ratio (SNR). The CWDP summary report from NOAA briefly showed the evaluation results of geographic/temporal coverages and bending angle quality of GeoOptics RO data. However, the influence of SNR and the impact of spatial mismatches, which are related to RO data quality and uncertainty in the comparison results, were not discussed in detail in the previous studies. In order to assess the GeoOptics performance in multifaceted aspects and increase reliability of our study, we also analyze the influence of SNR on RO data quality and the spatial mismatch impact on the comparison results.

The remainder of the article is organized as follows—the data used in this study are explained in Section “Data”. Section “Measurement coverage” shows the analysis results of the geographic, altitude, and local time coverage of the GeoOptics RO measurements. Section “Influence of SNR” analyzes the influence of the SNR on the RO measurements. Section “Quality assessment of RO measurements by comparing with other data sets” assesses the RO measurement quality by comparing the atmospheric profiles from collocated climate reanalysis/radiosonde observations. The final section summarizes the findings of our study.

Data

Radio occultation data

The GeoOptics CICERO constellation consists of several 6U CubeSats in low-Earth orbit. Early operational satellites (e.g., the OP1 constellation) occupy circular orbits at a ~ 500 km altitude and high inclination (98$^\circ$). GeoOptics launched the first of its operational CICERO satellites on January 11, 2018. Future launches will introduce satellites at lower inclinations to improve regional coverage of the mid-latitudes and tropics (Yunk et al. 2016). This study addresses two study periods: (1) June–July 2019 and (2) June 2021, referred to as “Period 1” and “Period 2”, respectively. During Period 1, the GNSS RO data were available from only two satellites. GeoOptics data from Period 1 were provided by GeoOptics, Inc. under an agreement with the University of Colorado. During Period 2, there were three operational GeoOptics LEO satellites. The GeoOptics data from Period 2 were provided by the COSMIC Data Archive and Analysis Center (CDAAC).

The COSMIC-1 RO data during June–July 2019 were used to assess the GeoOptics RO data performance during Period 1. COSMIC-1 includes six LEO satellites and signifies the first satellite constellation dedicated to the remote sensing of the Earth’s atmosphere and ionosphere using GNSS RO (Liou et al. 2007; Yue et al. 2014). The COSMIC-1 RO constellation carried the IGOR; an improved version of the BlackJack receiver, developed by JPL. Note that only a single LEO satellite in the COSMIC-1 system was operational during June–July 2019, as COSMIC-1 was near the end of its lifespan.

COSMIC-2 data from June 2021 were used to assess the GeoOptics data performance during Period 2. COSMIC-2 is an operational follow-on to the COSMIC-1 research mission. The COSMIC-2 constellation consists of 6 LEO satellites in 24$^\circ$ inclination orbits at an altitude of 520 km. Each LEO satellite carries an advanced TGRS developed at JPL. Table 1 summarizes the data period and characteristics of each RO system used in this study.

Table 1 Data period and characteristics of each RO mission

Reference data for analysis of RO measurement quality—ECMWF, radiosonde

To evaluate the quality of the RO observations, we used the European Centre for Medium-Range Weather Forecasts (ECMWF) analysis field data and radiosonde observations as the reference estimates.

The ECMWF data were obtained from ‘echPrf’ provided by CDAAC and contains atmospheric profiles (temperature, pressure, refractivity) generated from the ECMWF resolution gridded analysis and collocated with occultation profiles. The bending angle profiles from ECMWF were used as reference profiles for evaluating the quality of the bending angle profiles retrieved from the RO measurements.

The radiosonde observations were obtained from ‘sonPrf’ for the analysis of Period 1 and from Vaisala RS41 of the Global Climate Observing System Reference Upper Air Network (GRUAN) for the analysis of Period 2. The ‘sonPrf’ data are the atmospheric profile generated from radiosonde data from the National Center of Atmospheric Research mass store and collocated with occultation profiles. For the analysis of Period 2, we used the measurements from RS41 because during the period of interest, COSMIC-2 did not provide ‘sonPrf’. Fundamentally, a Vaisala RS41 radiosonde measures vertical profiles of atmospheric information such as pressure, temperature, and humidity. The provided atmospheric information can be used to calculate the refractivity, observed by the radiosonde measurements (Smith and Weintraub 1953) by Eq. (1):

$$\mathrm{Refractivity }\left(N\right)=77.6\frac{P}{T}+3.73\times {10}^{5}\frac{{P}_{W}}{{T}^{2}},$$

(1)

where $T$ is the temperature in K, and $P$ and ${P}_{W}$ are the total air pressure and partial pressure of water vapor in hPa, respectively. To avoid additional uncertainty due to assumptions applied in the process of converting refractivity to the bending angle, the refractivity is used for comparison between RO soundings and radiosonde observations. Table 2 summarizes the data sets used in the analyses.

Table 2 Data sets used in each analysis

Measurement coverage

Large amounts of RO measurements can be used for composition of atmospheric parameter-based climatologies (Borsche et al. 2006; Foelsche et al. 2008). The validity of the climatologies not only depends on the reliability of the measurements and its retrieval (Hajj et al. 2004; Wickert et al. 2004), but also on the sampling time and location (Foelsche et al. 2008). Therefore, assessing the RO performance in terms of measurement coverage is essential.

In this section, the geographic, altitude, and local time coverages of the GeoOptics RO measurements are assessed. To this end, we compare these measurements with those of COSMIC-1 (Period 1) and COSMIC-2 (Period 2).

Geographic coverage

The geographic coverage (e.g., spatial coverage) of RO measurements is an important characteristic that affects the RO performance. In this context, numerous well-distributed occultation events can be useful for operational climatologies.

Table 3 shows the maximum number of daily occultation events from one LEO for each RO system during the analysis period. As shown, COSMIC-2 had the largest number of daily occultation events, followed by GeoOptics and then COSMIC-1. The higher number of GeoOptics occultation events than those of COSMIC-1 potentially emerged because GeoOptics received GNSS signals from both GPS and GLONASS, while COSMIC-1 only received GPS signals.

Table 3 Maximum number of daily occultation events from one LEO for each RO system

Figure 1a shows the occultation events from COSMIC-1 (red squares), COSMIC-2 (green circles), and GeoOptics (blue triangles) from one LEO during 1 day. Figure 1b illustrates the latitudinal distributions of COSMIC-1 (dotted curve with red squares), COSMIC-2 (dash-dotted curve with green circles), and GeoOptics (solid curve with blue triangles). COSMIC-1 and GeoOptics latitudinal distributions demonstrated that all the latitudes were covered by observations, being somewhat clustered in the mid-latitude regions. However, the COSMIC-2 RO events only covered the low-latitude regions within approximately $\pm$46.5$^\circ$ due to the low orbit inclination of the COSMIC-2 satellites (24$^\circ$). Meanwhile, GeoOptics satellites, characterized by high-inclination orbits (98$^\circ$), resulted in the global coverage of GeoOptics RO events, which were mainly clustered at mid-latitude regions. The spatial coverage of GeoOptics can complement the six operational COSMIC-2 satellites.

Fig. 1

a Global coverage of occultation events for COSMIC-1 (red squares), COSMIC-2 (green circles) and GeoOptics (blue triangles). b latitudinal distributions for COSMIC-1 (dotted curve with red squares), COSMIC-2 (dash-dotted curve with green circles) and GeoOptics (solid curve with blue triangles)

Altitude coverage

The efficiency of RO-based retrieval of key weather parameters in the lower troposphere depends on its ability to penetrate low altitudes. To understand this aspect, an altitudinal analysis was performed by using the “penetration depth” of an occultation event. In this analysis, we defined the penetration depth (i.e., penetration altitude) as the minimum altitude above ground level with a valid bending angle for a given occultation event. Figure 2 shows the global map of the penetration depth for COSMIC-1, COSMIC-2 and GeoOptics. The color scale bar reflects the penetration altitudes of RO measurements indicating a higher penetration ability as the altitude decreases. The gray-colored areas are the regions with no data. According to Fig. 2, within the geographic coverage of COSMIC-2, the penetration depth map of COSMIC-2 exhibited a wider region of low penetration depth (dark blue-colored points) than the two other systems. This indicates that the occultation events from COSMIC-2 can penetrate deeper in a wider area than other systems. The penetration depth maps of COSMIC-1 and GeoOptics (Fig. 2a, c, respectively) show that a penetration depth within $\pm 45^\circ$ latitudes have a higher penetration depth than other latitude regions. The higher penetration depth in lower latitude regions is likely due to the presence of large vertical moisture variation in the lower troposphere (Ao et al. 2012).

Fig. 2

Penetration depth maps. a COSMIC-1, b COSMIC-2, c GeoOptics

To statistically compare the penetration ability, we also investigated the percentage of signal penetration at a given altitude to analyze the altitude distribution. To this end, the GeoOptics and COSMIC-1 RO data from the latitudes within ± 46.5$^\circ$ (upper/lower latitude bounds of COSMIC-2 coverage) were used for comparing the penetration percentage under the same conditions. Figure 3 shows the percentage of RO events penetrating different altitudes (the vertical axis) for COSMIC-1 (red dotted curve), COSMIC-2 (green dash-dotted curve) and GeoOptics (blue solid curve). The penetration percentage of GeoOptics was similar to that of COSMIC-1 and COSMIC-2 at altitudes between 4 and 8 km. However, the difference between the COSMIC-2 and GeoOptics graphs is more prominent at lower altitudes, while COSMIC-1 and GeoOptics exhibit nearly identical trends. Below 1 km, COSMIC-2 exhibit the largest penetration percentage value (85%), followed by GeoOptics (76%) and COSMIC-1 (75%). Notably, GeoOptics stands out with a high penetration capability, compared with other RO systems, such as CHAMP and GPS/MET, with penetration percentages of 45% and 35%, respectively (below 1 km) (Schreiner et al. 2003). The percentages of RO events that penetrated below the altitude of 200 m were estimated to be 40%, 51% and 45% for COSMIC-1, COSMIC-2 and GeoOptics, respectively, showing a slight advantage for the COSMIC-2 satellites.

Fig. 3

Penetration percentage versus penetration depth for COSMIC-1 (red dotted curve), COSMIC-2 (green dash-dotted curve) and GeoOptics (blue solid curve)

Local time coverage

Furthermore, a local time distribution analysis is necessary to assess the RO performance, given its influence on the sampling errors of RO climatologies (Foelsche et al. 2008; Pirscher et al. 2007; Shen et al. 2021). For instance, if diurnal temperature variability is strong, local time intervals not accounting for the entire day may not reflect the actual temperature trend, thereby triggering new errors in climatologies (Kirk-Davidoff et al. 2005). Figure 4 shows the local time–latitude distributions of COSMIC-1 (red dots), COSMIC-2 (green dots) and GeoOptics (blue dots). COSMIC-1 and COSMIC-2 occultation events had 24-h local time coverage at each geographic coverage, while GeoOptics exhibited a somewhat concentrated local time distribution. The GeoOptics data at the middle and low-latitude regions had no occultation events within 2–8 LT and 12–21 LT. Figure 5 shows the local time distributions of each RO system. As indicated, COSMIC-1 exhibited nearly uniform distribution with some concentrations around 5 LT, 11 LT and 19 LT (dotted curve with red squares). If all COSMIC-1 LEO satellites are operational, the distribution would be more uniformed. As the COSMIC-2 satellites are equally distributed in local time around the Earth, they provide RO data with uniform local time coverage (dash-dotted curve with green circles). Due to the satellites in sun-synchronous orbit, GeoOptics provided the RO data with limited local time coverage, especially around 11 and 23 LT (solid curve with blue triangles). It is reasonable to suggest that future GeoOptics constellations must be designed with more evenly distributed local time coverage to be more useful for climate applications.

Fig. 4

Local time–latitude distributions of radio occultation measurements for COSMIC-1 (red dots), COSMIC-2 (green dots) and GeoOptics (blue dots)

Fig. 5

Local time distributions for COSMIC-1 (dotted curve with red squares), COSMIC-2 (dash-dotted curve with green circles) and GeoOptics (solid curve with blue triangles)

Hereafter, we used the COSMIC-1 and GeoOptics data within the COSMIC-2 geographic coverage to evaluate their performance within the same geographic condition.

Influence of SNR

The SNR is a key parameter for assessing the RO performance. The SNR is the RO signal amplitude divided by the noise level and is provided in voltage-to-voltage unit (V/V). Because the SNR includes noise in the original RO signal as well as noise contributions from the GNSS satellites, ray path, and receiver in LEO satellites, it is more suitable for characterizing the overall RO signal quality than carrier-to-noise ratio (CNR) which captures the noise in the RF transport path only. Previous studies indicate that SNR is important in several aspects: (a) quality of retrieval measurements, (b) detection of sharp atmospheric boundary layer (ABL) tops, and (c) penetration of soundings lower into the troposphere (Gorbunov et al. 2022; Schreiner et al. 2020). On this basis, we analyzed the influence of SNR on the following three parameters: standard deviation of the difference between the bending angles and a model climatology bending angles between 60–80 km (STDV), bending angle lapse (BAL), and penetration depth. STDV is conventionally determined by receiver noise and other residual errors (Schreiner et al. 2011, 2020), thereby being a good indicator of the retrieval quality. BAL can be applied by utilizing its maximum value to detect the top of the ABL. As discussed in Section “Altitude coverage”, the penetration depth indicates the low-altitude penetration capability of RO systems.

Figure 6 shows the average GeoOptics STDV (curve with blue triangles), the maximum BAL (curve with purple circles), and penetration depth (curve with pink squares) as a function of L1 SNR. The analysis of SNR against STDV shows that STDV decreased as SNR increased, thereby indicating that a high SNR fundamentally predetermines low uncertainty in the retrieved bending angle measurements. The analysis of SNR against BAL demonstrates that a higher SNR allows retrievals of bending angle profiles with a larger maximum value of BAL, thereby strengthening the reliability of the detection of sharp ABL tops. Furthermore, the comparison of SNR with the penetration depth plot shows that a higher SNR allows lower altitude penetration. These results are in line with previous findings (Schreiner et al. 2020; National Oceanic and Atmospheric Administration 2020).

Fig. 6

Average GeoOptics STDV (curve with blue triangles), BAL (curve with purple circles), and penetration depth (curve with pink squares) as a function of L1 SNR

Figure 7 shows the histograms of L1 SNR, which was computed from the average SNR values in the 60–80 km geometric height range of the L1 signal. COSMIC-1 ranges from 220 V/V to 1090 V/V with a mean SNR of 660 V/V. COSMIC-2 varied from 200 V/V to 2610 V/V with a mean SNR of 1340 V/V. GeoOptics ranged from 70 V/V to 1300 V/V with a mean SNR of 520 V/V. Of these data estimates, GeoOptics exhibited the lowest value of L1 SNR. We further evaluated whether the low SNR caused lower performance of GeoOptics compared to that of current COSMIC-2. To compare the SNR effects of each COSMIC-2 and GeoOptics system within the same conditions, we established collocation pairs with the criteria of 200 km and 2 h during June 1st, 2021, and their locations are shown in Fig. 8. We utilized the collocation pairs above the ocean, which are shown with blue markers, to refute any effects of elevation from the land (e.g., mountain regions). Moreover, given the relatively low sea surface temperature variation, the uncertainty, induced by spatial/temporal mismatches, is smaller above the ocean. The bounds of the selected region were −25$^\circ$ to 25$^\circ$ for latitude, and -180$^\circ$ to -130$^\circ$ for longitude (see the box with light-blue dashed lines in Fig. 8).

Fig. 7

Histograms of L1 SNR for COSMIC-1 (red bar), COSMIC-2 (green bar) and GeoOptics (blue bar). The mean values of SNR for each RO system are indicated with the dotted lines

Fig. 8

COSMIC-2 (circles) and GeoOptics (triangles) collocation points during June 1st, 2021. Selected 20 pairs are indicated with blue markers within the box with light-blue dashed lines

Figure 9 shows the STDV, BAL, and penetration depth of COSMIC-2 (circles) and GeoOptics (triangles) for each of the selected 20 collocation pairs. The SNR values are indicated with marker color (Red: low SNR, Blue: high SNR). Except for Fig. 9a, we found, from Fig. 9b, c, that as SNR increases, the performance of COSMIC-2 RO improves in terms of BAL and penetration depth. Markers having a darker blue color (higher SNR) have higher BAL values and lower penetration depth values. Note that even for the occultation points above the sea surface, impacts of spatial/temporal mismatches could be retained between each point of the COSMIC-2 occultation event. This could be one reason why the SNR vs STDV trend of COSMIC-2 in Fig. 9a shows weak correlation between high SNR and low STDV. Figure 9a further demonstrates that, except for pair numbers 3, 6, 12, and 20, the GeoOptics STDV values were similar to that of COSMIC-2, despite GeoOptics exhibiting lower SNR values. According to Fig. 9b, there were also some cases (1, 2, 4, 5, 7, 8, 10), where GeoOptics exceeded the COSMIC-2 BAL, despite having lower SNR values than COSMIC-2. Figure 9c illustrates that, most COSMIC-2 cases exhibited lower penetration depth, compared with GeoOptics. However, despite GeoOptics yielding lower SNR values, some cases (2, 4, 11, 17, 18, 20) indicated similar penetration depth compared to COSMIC-2.

Fig. 9

a STDV (sorted by COSMIC-2 STDV), b BAL (sorted by COSMIC-2 BAL), c penetration depth (sorted by COSMIC-2 penetration depth) for each collocation pair

Overall these results demonstrate that, despite GeoOptics having a lower SNR than that of COSMIC-2, it exhibited comparable (but not outperforming) performance to COSMIC-2.

Quality assessment of RO measurements by comparing with other data sets

In this section, we assess the quality of GeoOptics RO measurements by comparing them with other data sets. As mentioned, we compared the bending angle and refractivity retrieved from the RO measurements with the ECMWF global analysis and operational radiosonde observations.

Comparison with ECMWF

Figure 10 shows the mean and standard deviation of the fractional differences between the bending angle profiles from RO data and the collocated ECMWF global analysis. As shown, COSMIC-1, COSMIC-2, and GeoOptics exhibited similar trends in both mean and standard deviation plots. The mean plots (Fig. 10a) indicate that COSMIC-1, COSMIC-2 and GeoOptics exhibited almost zero error means at an altitude above approximately 3 km. At an altitude below 3 km, the mean errors of all three RO systems were negatively biased. The negative biases in RO refractivity measurements were fundamentally driven by the receiver tracking error, a half-cycle ambiguity in determining the phase delay and atmospheric ducting (Ao et al. 2003; Beyerle et al. 2003; Hajj et al. 2004; Sokolovksiy 2001). The standard deviation of the fractional refractivity differences of all three RO systems were found to be < 2.5% between 10 and 30 km, but increased to nearly 10–15% outside these altitudes. Both mean and standard deviation plots (Fig. 10a, b) indicate that COSMIC-1 statistics exhibited the lowest values. Notably, because the COSMIC-1 data were taken from a different period (Period 1), additional effects could emerge. In particular, the comparison of COSMIC-2 with GeoOptics shows very similar values (the averaged differences between COSMIC-2 and GeoOptics are within 0.06% for the mean profiles, and 0.3% for the standard deviation profiles), while the standard deviation of GeoOptics above 40 km was higher (more than 1%) than that of COSMIC-2, with the biggest difference (4%) at 50 km.

Fig. 10

a Mean, b standard deviation of the fractional bending angle differences $\left( {\frac{{{\text{BA}}_{{{\text{RO}}}} - {\text{BA}}_{{{\text{ECMWF}}}} }}{{{\text{BA}}_{{{\text{ECMWF}}}} }} \times 100} \right)$ and c counts of observations

Comparison with radiosonde

For further analysis, the RO measurements were compared with the radiosonde data. The collocated profiles obtained from the RO data were selected to calculate the refractivity error of the RO measurements with respect to radiosonde measurements in the nearby areas. In this study, we defined the time window as 6 h and the maximum spatial distance for the collocation condition as 400 km (the same criteria were also applied to sonPrf).

Figure 11 shows the fractional refractivity differences, compared with the radiosonde data. Figure 11a–c shows the results of the refractivity errors of COSMIC-1 (red) and GeoOptics (blue) during analysis Period 1. Figure 11d–f illustrates the results of COSMIC-2 (green) and GeoOptics (blue) during analysis Period 2. Note that high fluctuations of the results during analysis Period 1 (Fig. 11a–c), were driven by the low and variable vertical resolution of sonPrf profiles. While the vertical resolution of RS41 (Fig. 11d–f) is generally constant with the value of ~ 6 m, the sonPrf profiles (Fig. 11a–c) have a strong variation of vertical resolutions (from ~ 10 m to ~ 2 km).

Fig. 11

a Mean, b standard deviation of the fractional refractivity differences $\left( {\frac{{{\text{N}}_{{{\text{RO}}}} - {\text{N}}_{{{\text{radiosonde}}}} }}{{{\text{N}}_{{{\text{radiosonde}}}} }} \times 100} \right)$ and c counts of observations from COSMIC-1 (red) and GeoOptics (blue) during Period 1. d–f Same information as a–c with the data from COSMIC-2 (green) and GeoOptics (blue) during Period 2

The comparisons between the GeoOptics RO soundings and radiosonde observations (Fig. 11a, b, blue dash-dotted curves), and between the COSMIC-1 RO soundings and radiosonde observations (Fig. 11a, b, red dotted curves) are very similar. The mean fractional refractivity differences of COSMIC-1 and GeoOptics RO data from radiosonde observations were found to be negative at altitudes below approximately 1 km, with magnitudes approaching 3% and 2% for COSMIC-1 and GeoOptics, respectively. This was driven by the negative refractivity bias associated with the RO soundings in the lower troposphere (Rocken et al. 1997; Sokolovskiy 2003). The standard deviations of fractional refractivity differences of both COSMIC-1 and GeoOptics are within 2% between 7 and 30 km, having a minimum of 1% at ~ 10 km. Below 7 km, the standard deviations increase with the decrease of altitude and have a maximum of 5% at ~ 2 km. The decreasing standard deviations below ~ 2 km could be caused by the significant decrease in the number of data (Fig. 11c). Like the results in Fig. 11a, b, the comparisons between the GeoOptics RO soundings and radiosonde observations (Fig. 11d, e, blue dash-dotted curves), and between the COSMIC-2 RO soundings and radiosonde observations (Fig. 11d, e, green dotted curves) are also very similar. The mean fractional refractivity differences of both COSMIC-2 and GeoOptics are negatively biased near the surface with the value of −2%. The standard deviations of both GeoOptics and COSMIC-2 have a minimum of 1% at ~ 10 km and a maximum of 6% at ~ 2 km.

Spatial mismatches impact

The collocation issue induced by spatial mismatches will remain a source of uncertainty in the comparisons shown in Fig. 11. Figure 12 further illustrates an example of the spatial mismatches impact on refractivity error. We compared the refractivity error profiles from a COSMIC-2–radiosonde pair and two GeoOptics–radiosonde pairs. Figure 12a illustrates the locations of radiosonde (black circle), one COSMIC-1 (green diamond), and two GeoOptics occultation events (blue and purple triangles). The distance between RO events and radiosonde locations is summarized in the table in Fig. 12a. It was found that the first GeoOptics–radiosonde pair (blue triangle and black circle connected with the light-blue dashed line) exhibited the largest distance. Figure 12b shows the refractivity profiles, while Fig. 12c displays the refractivity error differences compared to the radiosonde refractivity profiles. The refractivity error profiles (Fig. 12c) indicate that the first pair of GeoOptics–radiosonde, which exhibited the largest distance, yielded more biased errors compared to those from the other pairs (blue dash-dotted curve in Fig. 12c). Due to this, we also assessed the impacts of spatial mismatches on the refractivity error analysis.

Fig. 12

a Locations, b refractivity profiles of radiosonde (black), COSMIC-2 (green), and GeoOptics (blue, purple) occultation events. c Refractivity differences of COSMIC-2 (green) and GeoOptics (blue, purple) occultation events compared to radiosonde observations

Figure 13 illustrates the spatial mismatch-related impacts on the statistics of the refractivity error compared to radiosonde observations. The statistics of the refractivity error are calculated by adding the absolute mean and standard deviation of fractional refractivity differences ($\left|{\mu }_{\Delta N}\right|+{\sigma }_{\Delta N}$) within the altitude range of 5–20 km. The collocation pairs are divided into bins of 50 km spatial mismatch intervals centered at 125 km, 175 km, and 375 km. The statistics and the number of collocations of COSMIC-2 errors are shown by the green curve with diamond markers and green dotted curve, respectively. The statistics and number of collocations of GeoOptics are expressed by the blue curve with triangle markers and the blue dotted curve, respectively. Both COSMIC-2 and GeoOptics results demonstrate that, as the distance of spatial mismatches increased, the statistics of the refractivity error also increased. Although COSMIC-2 always had better performance (lower error statistics) compared with GeoOptics for every bin of spatial mismatch distance, GeoOptics retained a comparable performance.

Fig. 13

Error statistics of refractivity (compared to radiosonde) with respect to spatial mismatch distances. The error statistics and counts of COSMIC-2 RO are shown as the green curve with diamonds and green dotted curve, respectively. The error statistics and counts of GeoOptics RO are shown as the blue curve with triangles and blue dotted curve, respectively

Analysis of random error uncertainty by using the 3CH method

We used the three-cornered hat (3CH) method (Anthes and Rieckh 2018; Schreiner et al. 2020) to estimate the random error uncertainty (standard deviation) of the GeoOptics refractivity observations by using the collocated reference data including ECMWF, radiosonde, and COSMIC-2. Figure 13 shows that the profiles from the two data sets become more alike as the distances become smaller. However, opting for stricter collocation criteria (smaller distance of spatial mismatch) would lead to insufficient data, and subsequently cause errors in 3CH estimates. We used radiosonde and COSMIC-2 profiles collocated within 3 h and 300 km from the GeoOptics RO events; these criteria were also used in Schreiner et al. 2019. The collocation events of COSMIC-2 which have the same ray path with GeoOptics are too few to obtain reliable statistics. Thus, we considered the location of occultation points only when we search for the collocation events. With the four data sets, there are three estimates of error variance (i.e., square of the standard deviation). The error variances of the GeoOptics (${\sigma }_{\text{GeoOptics}}^{2}$, and error is defined as ‘GeoOptics minus truth’, where the truth is unknown) could be estimated with the following three equations:

$${\mathrm{Estimate }1: \sigma }_{\text{GeoOptics}}^{2}=\frac{1}{2}\left(E\left({\left({\text{GeoOptics}}-{\text{radiosonde}} \right)}^{2}\right)+E\left({\left({\text{GeoOptics}}- {\text{ECMWF}}\right)}^{2}\right)-E\left({\left({\text{ECMWF}}-{\text{radiosonde}}\right)}^{2}\right)\right),$$

(2)

$${\mathrm{Estimate }2: \sigma }_{\text{GeoOptics}}^{2}=\frac{1}{2}\left(E\left({\left({\text{GeoOptics}}-{\text{radiosonde}}\right)}^{2}\right)+E\left({\left({\text{GeoOptics}}- {\text{COSMIC2}} \right)}^{2}\right)-E\left({\left({\text{COSMIC2}}-{\text{radiosonde}}\right)}^{2}\right)\right),$$

(3)

$$\mathrm{Estimate }3: {\sigma }_{{\text{GeoOptics}}}^{2}=\frac{1}{2}\left(E\left({\left({\text{GeoOptics}}-{\text{ECMWF}}\right)}^{2}\right)+E\left({\left({\text{GeoOptics}}- {\text{COSMIC2}}\right)}^{2}\right)- E\left({\left({\text{COSMIC2}}-{\text{ECMWF}}\right)}^{2}\right)\right),$$

(4)

where ${\text{GeoOptics}}$ (or ${\text{radiosonde}}$, ${\text{ECMWF}}$, ${\text{COSMIC2}}$) corresponds to the value of refractivity as estimated by GeoOptics (or radiosonde, ECMWF, COSMIC-2) and $E$ denotes the mean. The differences between the two data sets (e.g., ${\text{GeoOptics}}-{\text{radiosonde}}$, ${\text{ECMWF}}-{\text{radiosonde}}$, …) are the normalized differences. The “${\text{GeoOptics}}-{\text{others}}$” (i.e., ${\text{GeoOptics}}-{\text{ECMWF}}$, ${\text{GeoOptics}}-{\text{COSMIC2}}$, ${\text{GeoOptics}}-{\text{radiosonde}}$) in the equations are the apparent errors of the GeoOptics RO observation. The variance of the apparent error is given by

$${\sigma }_{\text{GeoOptics}-{\text{others}}}^{2}={\sigma }_{\text{GeoOptics}}^{2}+{\sigma }_{\text{others}}^{2}$$

(5)

under the assumption in the 3CH method that the errors of the data sets are uncorrelated. The radiosonde, ECMWF and COSMIC-2 also have three estimates of error variance. The solid curves in Fig. 14 are the mean of the three standard deviation estimates for radiosonde (pink), ECMWF (yellow), COSMIC-2 (green) and GeoOptics (blue).

Fig. 14.

3CH estimates of refractivity error standard deviation of radiosonde (pink), ECMWF analysis (yellow), COSMIC-2 (green), GeoOptics (blue). The mean standard deviation of differences between GeoOptics and each of the other data sets is shown with black dash-dotted curve. The counts are shown by the black dotted curve

The mean standard deviation of “${\text{GeoOptics}}-{\text{others}}$” are also shown as the black dash-dotted curve in Fig. 14 and are estimated by averaging the standard deviations of the apparent errors between GeoOptics data and other data sets (i.e., averaging ${\sigma }_{{\text{GeoOptics}}-{\text{ECMWF}}}$, ${\sigma }_{{\text{GeoOptics}}-{\text{COSMIC2}}}$ and ${\sigma }_{{\text{GeoOptics}}-{\text{radiosonde}}}$). The 3CH estimates of the GeoOptics error standard deviation (blue curve) were always lower than the standard deviations of the difference between GeoOptics and the other data sets (black dash-dotted curve). This indicates that apparent errors are always larger than true errors; therefore, the estimates of true errors were reasonable by satisfying Eq. (5). For these data sets, the ECMWF generally have the lowest standard deviation, which is reasonable considering that ECMWF uses a data assimilation system that assimilates numerous independent observations. The radiosonde has the highest standard deviation at all altitudes, mainly due to the representativeness errors of the radiosonde, which provide point measurements while the other data sets are larger-scale (~ 100 km) horizontal averages (Anthes and Rieckh 2018). The standard deviations of COSMIC-2 and GeoOptics errors are comparable and are in the middle (between ECMWF and radiosonde). In the stratosphere (above ~ 20 km), COSMIC-2 is smaller than GeoOptics. However, GeoOptics errors were lower than the COSMIC-2 profile in the troposphere (between ~ 5 and ~ 20 km). This result could be affected by collocation criteria. According to the analysis results in Section “Spatial mismatches impact”, as the distances between the locations of two different data sets increase, the differences between the measurements from the two data sets also increase. Moreover, since the collocated pairs are selected by applying the collocation criteria (300 km) based on the locations of GeoOptics RO soundings, the actual distance between COSMIC-2 and other data sets (ECMWF and radiosonde) could be larger than the criteria. This may induce the additional uncertainty in the apparent errors of COSMIC-2 (i.e., “${\text{COSMIC2}}-{\text{ECMWF}}$”, “${\text{COSMIC2}}-{\text{radiosonde}}$”, “${\text{COSMIC2}}-{\text{GeoOptics}}$”), thereby increasing the standard deviation of COSMIC-2. If the locations of COSMIC-2 profiles are closer to GeoOptics occultation points, the standard deviation of COSMIC-2 would decrease, thereby becoming lower than the standard deviation of GeoOptics (not shown). At the deeper altitudes, the COSMIC-2 error reached a maximum at approximately 3 km, then rebounded to values lower than GeoOptics. Below ~ 5 km, GeoOptics errors were smaller: down to nearly the same value as the ECMWF profile. These results demonstrate the high-quality observations from GeoOptics RO which have smaller errors than the radiosonde errors and have comparable errors to those of COSMIC-2 at all altitudes. In addition, GeoOptics RO measurements have similar errors to accurate ECMWF global analysis in the lower troposphere.

Summary and conclusions

This study evaluates the performance of GeoOptics RO by comparing its data with that from COSMIC-1 and COSMIC-2 to elucidate its applicability for accurate numerical weather forecasting. With global coverage, GeoOptics RO data can complement the COSMIC-2 coverage, which is restricted to low-latitude regions. The GeoOptics altitude coverage demonstrated a satisfactory low-altitude penetration capability. The local time coverage evaluation results suggest that future GeoOptics satellite constellations should be designed to achieve a more evenly distributed local time coverage. The analysis of the influence of SNR on RO performance shows that, despite GeoOptics having lower SNR values compared to those of COSMIC-1 and COSMIC-2, it clearly has potential for retrieving high-quality bending angle, detecting ABL, and penetrating a lower troposphere.

We assessed the quality of the RO measurements by comparing them with other data sets as well. This assessment confirmed the high quality of GeoOptics RO retrieval measurements. The comparisons of bending angles and refractivity estimates derived by GeoOptics with ECMWF global analysis and radiosondes show very small errors. In the upper-troposphere and lower-stratosphere, the mean fractional bending angle differences compared to ECMWF are near zero and within 2.5%, respectively. In the same altitude range, the mean and standard deviation of fractional refractivity differences compared to radiosonde observations are also near zero and within 2%, respectively. These errors are similar to those from the COSMIC-1 and COSMIC-2 data. The GeoOptics random errors of refractivity were found to be less than that of radiosondes and comparable to that of COSMIC-2 at all altitudes. In the lower troposphere, the GeoOptics random errors are similar to that of ECMWF.

Overall, our results indicate that GeoOptics can yield high-quality RO data comparable to those of the COSMIC-1 and COSMIC-2 missions. Although further analysis is required to assess the advantages of GeoOptics RO data on an operational NWP forecast, this study confirms that the GeoOptics RO CubeSats produce high-quality RO measurements. Future studies can be motivated to develop a dense GNSS array composed of RO CubeSats in low-Earth orbit. The dense GNSS array of RO CubeSats will provide many valuable atmospheric measurements which have the potential to make a strong contribution to weather forecasting and climate research.