Some reasoning on the RELM-CSEP likelihood-based tests

Abstract

The null hypothesis is the essence of any statistical test: this is basically a comparison of what we observe with what we would expect to see if the null hypothesis was true. In this work, I explore the suitability of the null hypothesis of likelihood-based tests (LBTs), which are often adopted by the laboratories of the Collaboratory for the Study of Earthquake Predictability (CSEP), to check earthquake forecast models. First, I discuss the LBT in the wider context of classical statistical hypothesis testing. Then, I present some cases in which the null hypothesis of LBT is not appropriate for determining the merits of earthquake forecast models. I justify these results from a theoretical point of view, within the framework of point process theory. Finally, I propose a possible upgrade of LBT to enable the correct assessment of the forecasting capability of earthquake models. This study may provide new insights to the CSEP LBT.

Background

The increasing interest of the seismological community in earthquake forecasting has highlighted the need for a proper evaluation of forecast models. This has motivated the birth of the working group on Regional Earthquake Likelihood Models (RELM, Schorlemmer and Gerstenberger 2007) and of the Collaboratory for the Study of Earthquake Predictability (CSEP, Jordan 2006), both designed to evaluate the quality of forecast models. The protocol adopted by RELM/CSEP is based on classical statistical hypothesis testing (Schorlemmer et al. 2007). This is then finalized to reject or accept the null hypothesis (hereinafter H₀) on the basis of a numerical summary of the data. RELM/CSEP working groups adopt two main types of testing methods: likelihood-based tests (LBTs) (Schorlemmer et al. 2007; Zechar et al. 2010) and alarm-based tests (ABTs) (Zechar and Jordan 2008). In this study, I focus on LBTs and specifically on N and L tests (Schorlemmer et al. 2007).

The RELM/CSEP working groups formalized the LBT to test hypotheses that ‘should follow directly the model, so that if the model is valid, the hypothesis should be consistent with data used in a test. Otherwise, the hypothesis, and the model on which it was constructed, can be rejected’ (Schorlemmer et al. 2007). Actually, as I discuss below, this intent was not attained (Lombardi and Marzocchi 2010a; Schorlemmer et al. 2007, 2010a; Werner et al. 2010).

The CSEP testing centers use the N and L tests to check the consistency of expected (Λ={λ_(i,j)}) and observed (Ω={ω_(i,j)}) values of variables X_(i,j), representing the number of earthquakes with magnitude above a threshold M _F , in nonoverlapping bins $\left\{\right(T_i,R_j);T_i\in \mathcal{T},R_j\in \mathcal{R}\}$ of a predetermined spatio-temporal space $\mathcal{S}=\mathcal{R}$ ×$\mathcal{T}$ (Jordan 2006; Zechar et al. 2010). A model is represented by forecasts Λ, which are the only values provided by the modelers. The correct calculation of the p values of the LBT requires the probability distribution of X_(i,j) given by the model and specifically the probabilities

$$p_n^{\mathit{\text{ij}}}=P\{X_{(i,j)}=n\}\text{for}n=0,1,2,\dots$$

(1)

As this information is not available to modelers, the LBT assumes, as the null hypothesis H₀, that the variables X_(i,j) are independent and follow a Poisson distribution with mean λ_(i,j). Therefore, the set of probabilities $p_n^{\mathit{\text{ij}}}$ are substituted for the probabilities

$$q_n^{\mathit{\text{ij}}}=\frac{{\left[{\lambda }_{(i,j)}\right]}^n}{n!}{\text{exp}}^{-{\lambda }_{(i,j)}}\text{for}n=0,1,2,\dots$$

(2)

and the p values of the LBT are computed accordingly (Schorlemmer et al. 2007).

Specifically, the N test measures the probability of observing $N_i^O=\sum _j{\omega }_{(i,j)}$ events, for each forecast time period T _i . The p values of the N test are given by the probabilities (Zechar et al. 2010):

$${\delta }_1=P(X_i\ge N_i^O){\delta }_2=P(X_i\le N_i^O),$$

(3)

where $X_i=\sum _j{X}_{(i,j)}$. The RELM/CSEP protocol rejects a model if δ₁ or δ₂ is too small, meaning that the model overpredicts or underpredicts the observed seismicity. Under H₀, X _i is a Poisson variable with expectation $N_i^F=\sum _j{\lambda }_{(i,j)}$ (and PDF $q_n^i={\left[N_i^F\right]}^n{e}^{-N_i^F}/n!$), and the percentiles δ₁/ δ₂ are computed by this distribution (see Schorlemmer et al. 2007).

The L-test measures the probability of the joint log-likelihood L(Ω _i |Λ) of observing Ω, given the forecast Λ. Under H₀, L(Ω _i |Λ) is given by:

$$L\left({\Omega }_i\right|\Lambda )=\sum _j\left[ln\frac{{\left({\lambda }_{(i,j)}\right)}^{{\omega }_{(i,j)}}}{{\omega }_{(i,j)}!}-{\lambda }_{(i,j)}\right].$$

(4)

The p value of the L test is estimated by comparing L(Ω _i |Λ) with a predetermined number N of synthetic likelihood values $L\left({\Omega }_i^S\right|\Lambda )=\{L\left({\Omega }_i^{S_l}\right|\Lambda ),l=1,\dots ,N\}$, computed by Equation 4, of simulated catalogs ‘consistent with the forecast’ (Schorlemmer et al. 2007). This means that the forecast grids ${\Omega }_i^{S_l}$ are simulated according to the Poisson hypothesis supposed by H₀, and the p value of the L test is given by the proportion of simulated log-likelihoods below the value L(Ω _i |Λ):

$$\gamma =\frac{\left|\left\{L\right({\Omega }_i^{S_l}\left|\Lambda \right)\left|\right.L\left({\Omega }_i^{S_l}\right|\Lambda )\le L({\Omega }_i\left|\Lambda \right);l=1,\dots ,N\}\right|}{N}.$$

(5)

This shows that the LBT does not check the hypothesis that a forecast model has merit with the given data (marked hereinafter by Hyp₁). Actually, the LBT tests whether {ω_(i,j)} are independent random variables, from a Poisson population with mean {λ_(i,j)} (marked hereinafter by Hyp₂). When a model is not consistent with Hyp₂, i.e., when the set of probabilities $\left\{p_n^{\mathit{\text{ij}}}\right\}$ is significantly different from $\left\{q_n^{\mathit{\text{ij}}}\right\}$, the specific computation of the p values of the LBT is misleading, causing a potentially unjustified rejection of the model itself (Lombardi and Marzocchi 2010a).

The CSEP laboratories still systematically use the LBT, but a process of revision has begun. This study is intended to provide a contribution to this process.

Methods

A suitable revision of the LBT requires the full recognition and quantification of the causes and effects of the present inefficiencies. For this purpose, I apply the N and L tests to two classes of 1,000 simulated forecast grids, generated by different spatio-temporal magnitude models. In this way, the data are perfectly known, and the rejection of H₀ cannot mean the failure of the model being tested.

First, I generate two sets of synthetic catalogs. Each catalog covers a time period of 1 month (January 1 to 31, 2012), the Italian collecting region, and a magnitude range of [ 2.5,9.0], as chosen by CSEP (Schorlemmer et al. 2010b).

The first class of simulations is consistent with a version of the epidemic-type aftershocks sequence (ETAS) model (Ogata 1998), submitted to the CSEP-Italy testing region (Lombardi and Marzocchi 2010b). The rate of the model at time t, with location (x,y) and magnitude m, is given by:

$$\begin{array}{c}{\lambda }_1(t,x,y,m/{\mathcal{\mathscr{H}}}_t)\hfill \\ =\left\{\mu ·u(x,y)+\sum _{T_i<t}\frac{K{e}^{\alpha (M_i-M_0)}}{{(t-T_i+c)}^p}\frac{c_{\mathrm{dq\gamma }}^i}{{[r_i^2+{\left(d{e}^{\gamma (M_i-M_0)}\right)}^2]}^q}\right\}\\ \times \frac{b10^{b(m-M_0)}}{1-10^{b(M_{\text{max}}-M_0)}}\hfill \end{array}$$

(6)

where {μ,K,c,p,α,d,q,γ,b} are the model parameters, M₀ and M_max are the minimum and maximum magnitudes, ${\mathcal{\mathscr{H}}}_t=\left\{\right(T_i,X_i,Y_i,M_i);T_i<t\}$ is the history (i.e., the information relative to past events) up to time t, and r _i is the distance between location (x,y) and the epicenter of the i th event (X _i ,Y _i ) (see Lombardi and Marzocchi 2010b, for details). To compute the rate ${\lambda }_1(t,x,y,m/{\mathcal{\mathscr{H}}}_t)$, I include in the history the seismic bulletin of the Istituto Nazionale di Geofisica e Vulcanologia (INGV) from April 16, 2005 to December 31, 2011. Moreover, I add a synthetic event (T_ms,X_ms,Y_ms,M_ms) at time 00:00:00 on January 1, 2012 (T_ms), with magnitude M_ms = 6.0 and coordinates (X_ms,Y_ms)=(13.384°E,42.346°N). The parameter values used in this study are μ = 0.7, K = 0.026, p = 1.15, c = 0.01, α = 1.4, d = 0.7, q = 1.5, γ = 0.3, b = 1.0, M₀ = 2.5, and M_max = 9.0.

To generate the ETAS forecasts for day T _i and catalog C _k , I mimic the CSEP real-time experiment: specifically, I include the triggering rate for events with history ${\mathcal{\mathscr{H}}}_{T_i}$ of C _k and average the triggering rates of 1,000 simulated realizations of the process inside T _i (see Lombardi and Marzocchi 2010b, for details).

The second class of simulations follows a nonstationary poisson (NP) process. Specifically, the rate λ₂(t,x,y,m) is given by a stationary background and the triggering effect of event (T_ms,X_ms,Y_ms,M_ms). The rate of the NP model is as follows:

$$\begin{array}{c}{\lambda }_2(t,x,y,m)\hfill \\ =\left\{\mathrm{\mu u}(x,y)+\frac{K{e}^{\alpha (M_{\text{ms}}-M_0)}}{{(t-T_{\text{ms}}+c)}^p}\frac{c_{\mathrm{dq\gamma }}}{{\left[r^2+{\left(d{e}^{\gamma (M_{\text{ms}}-M_0)}\right)}^2\right]}^q}\right\}\\ \times \frac{b10^{b(m-M_0)}}{1-10^{b(M_{\text{max}}-M_0)}}\hfill \end{array}$$

(7)

where r is the distance between (x,y) and (X_ms,Y_ms). The parameters used here are μ = 0.7, K = 0.1, p = 0.9, c = 0.02, α = 1.4, d = 0.7, q = 1.5, γ = 0.3, b = 1.0, M₀ = 2.5, M_max = 9.0.

The simulations represent the average seismicity of the first month of a sequence (following a shock with magnitude 6.0), as predicted by the ETAS and NP models. The basic difference between the models is that the rate of the ETAS model depends on the whole history ${\mathcal{\mathscr{H}}}_t$ (i.e., information relative to past events), whereas the rate of the NP model depends on the coordinates of only one event (T_ms,X_ms,Y_ms,M_ms). Thus, the rate of the NP model is deterministic and decreasing in time from T_ms, whereas the rate of the ETAS model has a random nonmonotonic time evolution, depending on history ${\mathcal{\mathscr{H}}}_t$.

For each synthetic catalog, I compute the 1-day binned forecast grids Λ (M _F = 2.5) by integrating (in time, space, and magnitude) the rate of the model used to generate the catalog. The forecast grids Λ cover a period of 1 month (starting from January 1, 2012) and the test spatial grid adopted for the CSEP Italian laboratory (Schorlemmer et al. 2010b). Finally, I apply the CSEP/RELM N and L tests (with significance level α = 0.05 and M _F = 2.5) on all simulated catalogs, using the forecast grids previously computed.

In this paper, I propose an obvious upgrade of LBT, which does without the Poisson distribution. First, the discrete log-likelihood function L(Ω _i |Λ) of variables X _i (Equation 4) is substituted for the continuous-time log-likelihood function (hereinafter, CLF). This is a proper measure of the agreement between model and data, taking into account the features of a model. For a spatio-temporal magnitude earthquake model, this is given by

$$\begin{array}{ll}\text{CLF}& =\sum _{i=1}^{N_{\mathcal{Rx}\mathcal{Tx}\left[M_0{M}_{\text{max}}\right]}}ln\lambda (t_i,x_i,y_i,m_i)\\ -\underset{\mathcal{T}}{\int }\underset{\mathcal{R}}{\int }{\int }_{M0}^{M_{\text{max}}}\lambda (t,x,y,m)\mathit{\text{dtdxdydm}}\end{array}$$

(8)

where λ(t,x,y,m) is the rate of the model (Daley and Vere-Jones 2003) and $N_{\mathcal{Rx}\mathcal{Tx}\left[M_0{M}_{\text{max}}\right]}$ is the number of events inside the spatio-temporal magnitude space $\mathcal{Rx}\mathcal{Tx}\left[M_0{M}_{\text{max}}\right]$.

Second, the percentiles of the distributions of both the variables X _i and the CLF are derived directly by the model. This information allows the computation of more reliable p values for the tests (Werner and Sornette 2008; Schorlemmer et al. 2010a).

In brief, the new testing procedure presented here consists of the following steps:

• 1. For each forecast period T _i , the number of events (Ω _i ) and the CLF (CLF_M,i) of model M being tested are computed.

• 2. For each T _i , N catalogs given by model M are simulated; the occurrences ${\Omega }_{M,i}^S=\left\{{\Omega }_{M,i}^{S_l},l=1,\dots ,N\right\}$ and the likelihood ${\text{CLF}}_{M,i}^S=\left\{{\text{CLF}}_{M,i}^{S_l},l=1,\dots ,N\right\}$ are computed for all catalogs.

• 3. The percentiles of the empirical distributions generated in the previous step, used to perform a test at the 95% confidence level, are estimated. Specifically, the 2.5th and 97.5th percentiles $\left(P_{M,i}^{\Omega }\left[2.5\%\right]\text{and}{P}_{M,i}^{\Omega }\left[97.5\%\right]\right)$ of values ${\Omega }_i^S$ and the 5th percentile $\left(P_{M,i}^{\text{CLF}}\left[5\%\right]\right)$ of quantities ${\text{CLF}}_{M,i}^S$ are identified.

• 4) The observed values Ω _i and CLF_M,i are compared with the percentiles computed in the previous step. In this way, model M is rejected or retained for T _i . Specifically, model M is rejected if ${\Omega }_i<P_{M,i}^{\Omega }\left[2.5\%\right]$ or ${\Omega }_i>P_{M,i}^{\Omega }\left[97.5\%\right]$ or if ${\text{CLF}}_{M,i}\le P_{M,i}^{\text{CLF}}\left[5\%\right]$.

In this procedure, the percentiles of model M are estimated by simulations because it is often not possible to derive them analytically. However, the use of simulations is not mandatory for modelers, of course.

Results

First, I apply the CSEP LBT to two classes of ETAS and NP simulations. Figure 1a shows the fraction of rejections F _R (i.e., the proportion of catalogs for which H₀ is rejected) of the N and L tests as a function of time. As shown in Lombardi and Marzocchi (2010a), F _R for the ETAS simulations is well above 5%, which is the threshold justifiable by chance. On the other hand, F _R for the NP simulations is close to or below 5%, suggesting that Hyp₂ is consistent with the NP model.

Figure 1

Fraction of rejections. Application of CSEP/RELM LBT and the proposed testing procedure on simulated catalogs. (a) F _R of daily CSEP N and L tests, for ETAS and NP simulations of Italian seismicity and M _F = 2.5. (b) Comparison of F _R values of testing procedure proposed here with those obtained by CSEP/RELM LBT, for ETAS simulations of Japanese seismicity (M _F = 4.0), with a forecast time span of 3 months.

To investigate whether previous results depend on M _F or on the average seismic rate of the region, I apply the procedure described above to 1,000 new catalogs, reproducing the average seismicity of Japan (which has a seismic rate two orders of magnitude higher than that of Italy). These datasets are simulated by using an ad hoc ETAS model of this region. In this experiment, I consider a forecast time span T _i of 3 months, an overall time period of 10 years, and M _F =4.0. This last value is the threshold magnitude adopted by the Japanese CSEP laboratory for short-term forecasting experiments (Nanjo et al. 2011; Tsuruoka et al. 2012). I find that F _R is equal to 40% t o 50% and 60% t o 75% for the N and L tests, respectively (see Figure 1b).

I apply the new testing procedure described previously to the simulated Japanese catalogs. This gives the values of F _R in Figure 1b. The improvement, with respect to the CSEP version of the N and L tests, is clear: F _R is close to or below 0.05 for both tests. To clearly compare the CSEP methodology and the new testing procedures, Figure 2 shows the PDF of occurrences and log-likelihoods computed by the CSEP LBT and the proposed procedure for the first ETAS simulated Japanese catalog. The observed occurrences (solid black line, Figures 2a,b) are well above or below the confidence bounds (dashed black lines, Figure 2a) of the Poisson PDF (Equation 1) supposed by Hyp₂. This is because the distribution expected by the ETAS model (contour plot, Figure 2b), estimated by the empirical PDF of ${\Omega }_{\text{ETAS},i}^S$, has a long/heavy tail, which is clearly not consistent with Hyp₂. Similar results are found for the log-likelihood. The log-likelihoods L(Ω _i |Λ) computed by Equation 4 are well below the values of $L\left({\Omega }_i^S\right|\Lambda )$ expected by Hyp₂ (contour plot, Figure 2c). However, the log-likelihoods CLF_ETAS,i (Equation 8) are fully consistent with the log-likelihoods ${\text{CLF}}_{\text{ETAS},i}^S$ expected by the ETAS model (contour plot, Figure 2d).

Figure 2

Distribution of the number of events and of likelihoods for ETAS simulations. Contour plot of probability density as a function of time interval T _i of the number of events and likelihood for the first ETAS Japanese simulated catalog. (a) Contour plot of probabilities $q_n^i$ predicted by Poisson hypothesis Hyp₂. Solid black line marks the observed number of events. Dashed black lines mark the 2.5th and 97.5th percentiles of distribution. (b) The same as (a) but for the distribution expected by ETAS model. Specifically, dotted lines mark the values $P_{\text{ETAS},i}^{\Omega }\left[2.5\%\right]$ and $P_{\text{ETAS},i}^{\Omega }\left[97.5\%\right]$. (c) Contour plot of PDF of log-likelihoods L(Ω _i |Λ) predicted by Poisson hypothesis Hyp₂ (Equation 4). Solid black line marks the observed log-likelihoods. Dashed black lines mark the 5.0th percentile expected by Poisson distribution. (d) The same as (c) but for the CLF (see Equation 8). Solid black line marks the observed values CLF_ETAS,i. Dotted black line marks the percentile $P_{\text{ETAS},i}^{\text{CLF}}\left[5.0\%\right]$.

Discussion

The rejection of the null hypothesis of a statistical test can be due to chance because it is really false or because it is probabilistically inadequate (Stark 1997; Luen and Stark 2008). The null hypothesis H₀ of the RELM/CSEP LBT supposes that X_(i,j) are independent (in time and space) and Poisson random variables, with mean λ_(i,j), given by the model. The CSEP protocol interprets the rejection of H₀ as the failure of the model being tested. However, this procedure is misleading because H₀ is not consistent with any model (Lombardi and Marzocchi 2010a).

The above findings may be explained with the help of stochastic point process theory (Daley and Vere-Jones 2003); this is the natural context in which stochastic earthquake models may be discussed. A point process is fully represented by its ‘conditional intensity function’ (CIF) $\lambda (t,\overrightarrow{x}/{\mathcal{\mathscr{H}}}_t)$, i.e., the probability of observing an event in the instant $t\in \mathcal{T}$ and with additional variables (called marks) $\overrightarrow{x}\in \overrightarrow{\mathcal{X}}$, given the realization ${\mathcal{\mathscr{H}}}_t$ of the process before t (Daley and Vere-Jones 2003). The CIF of the models described in the previous section are given by Equations 6 and 7; the marks are locations and magnitudes. In the case of an NP process, the CIF is a deterministic function of time and marks, but it is independent of the past history (i.e., $\lambda (t,\overrightarrow{x}/{\mathcal{\mathscr{H}}}_t)=\lambda (t,\overrightarrow{x})$). Therefore, the events in nonoverlapping subsets of $\mathcal{T}\times \overrightarrow{\mathcal{X}}$ are independent and Poisson random variables (Daley and Vere-Jones 2003), as supposed by the RELM/CSEP LBT. In the most general case, the CIF is also a function of history ${\mathcal{\mathscr{H}}}_t$, and the variables X_(i,j) are not Poisson, unless the history is fully known (Meyer 1971; Papangelou 1972a; 1972b; Daley and Vere-Jones 2003). In a real-time forecast experiment, the history inside the forecast time window T _i is unknown; therefore, for such history-dependent models, such as ETAS, Hyp₂ is inadequate.

The hypothesis Hyp₂ has been questioned in several studies (Schorlemmer et al. 2010a; Werner et al. 2010) and, in the specific context of ETAS modeling, by Lombardi and Marzocchi (2010a). Here, I examine the causes and effects of the failure of the LBT. Specifically, I show that the failure of the LBT may be significant for high values of M _F and that it has heavy consequences for long forecast time windows. This is because the longer the forecast time window T _i , the greater the randomness of forecasts (due to the effect of the unknown history inside T _i ) and the lower the reliability of Hyp₂. This result contradicts the statement that the Poisson distribution is a good approximation of the forecast variability when M _F is large (Werner et al. 2010).

The process of revising the LBT has begun inside the scientific community. Some people have proposed replacing the Poisson distribution with a negative binomial distribution (Werner et al. 2010) to compute the p values of the tests. However, this solution does not significantly improve the LBT because the negative binomial distribution (as for the Poisson or any other distribution) is not consistent with all models. Inside the CSEP community, some suggest updating the forecasts more regularly, leaving the LBT unchanged (personal communication). I do not think this is the best way to resolve the inefficiencies of the LBT, as these do not derive from the regularity of the forecast calculations.

The procedure described above is an obvious upgrade of the N and L tests. It accounts for the actual variability of the X_(i,j) given by the model being tested. Moreover, it uses the CLF, which is a better tool for checking the agreement between models and data than the discrete log-likelihood (Equation 4) used by the CSEP L test and based on Hyp₂ (Schorlemmer et al. 2007).

This study has focused on short-term forecasts, without analyzing the dependence of results on the size of the forecast window. From a theoretical point of view, LBT might also fail for long-term forecasts because of dissimilarities between the sets of probabilities $\left\{p_n^{\mathit{\text{ij}}}\right\}$ and $\left\{q_n^{\mathit{\text{ij}}}\right\}$ (see Equations 1 and 2) or, in other words, the unsuitability of Hyp₂. This study is not relevant to models that are explicitly supposed to be time-invariant, such as the models tested in the 5-year mainshock RELM experiment (Schorlemmer et al. 2010a; Zechar et al. 2013). However, the failure of the LBT might be significant for medium long-term forecast models with strong time-dependent components, especially in testing regions with a high seismic rate. In other words, the present study does not invalidate most of the results of the first RELM/CSEP forecast experiments, which focus on long-term time-invariant models. However, the inclusion of different forecast time-spans and time-dependent models in new CSEP experiments requires both an urgent revision of the testing procedure and an effort by modelers to provide full distributions of the variables being tested.

Conclusions

The main goal of this study was to interpret the failures of the CSEP/RELM LBT and to propose a possible upgrade of the N and L tests. The main findings can be summarized as follows:

• 1. All LBTs are based on classical statistical hypothesis testing; therefore, they are intended to reject or not reject a null hypothesis H ₀. The null hypothesis of the LBT is that the variables X _(i,j) are independent and Poisson-distributed, with the rate given by forecasts. Therefore, the LBT is inadequate for checking the merits of a forecast model that is inconsistent with Hyp₂.

• 2. Specifically, Hyp₂ is not adequate for history-dependent models, such as ETAS, because the unknown history inside the forecast period means that X _(i,j) do not follow a Poisson distribution.

• 3. In these cases, the LBT may fail for large values of M _F , especially for large forecast time windows, as the effect of the unknown history is greater.

• 4. I propose a revised version of the LBT that (1) adopts the CLF and (2) requires the percentiles of the distributions of X _i and CLF_M,i.

• 5. The points discussed in this study highlight the need to revise the testing procedure for present and future experiments, which include many time-dependent models. However, they have a relative effect on the first RELM/CSEP experiments, mainly focused on long-term time-independent models.