# Conventional N-, L-, and R-tests of earthquake forecasting models without simulated catalogs

- Masajiro Imoto
^{1}Email author, - David A. Rhoades
^{2}, - Hiroyuki Fujiwara
^{1}and - Naoko Yamamoto
^{1}

**63**:10

https://doi.org/10.5047/eps.2010.08.007

© The Society of Geomagnetism and Earth, Planetary and Space Sciences (SGEPSS); The Seismological Society of Japan; The Volcanological Society of Japan; The Geodetic Society of Japan; The Japanese Society for Planetary Sciences; TERRAPUB. 2011

**Received: **12 January 2010

**Accepted: **24 August 2010

**Published: **4 March 2011

## Abstract

We propose a new procedure for testing the expected number (N-test), log likelihood (L-test), and log likelihood-ratio (R-test) of seismicity models. In these tests, scores obtained from observed earthquakes are compared with distributions of scores estimated from earthquakes expected from the models under test. We introduce a method to estimate the test score distributions analytically where uncertainties in magnitude and hypocentral parameters are involved. The analytical formulas used to estimate expected values and standard deviations of the test scores for earthquakes conforming to the test models were derived in earlier published studies, which allowed calculation of normal approximations by which to test score distributions. Using these two methods simultaneously, we can perform N-, L-, and R-tests for seismicity models without using any simulated catalogs. As a case study, the proposed procedure was applied to two seismicity models for Kanto, central Japan. To compare our procedure with the current one based on the Monte Carlo method, we randomly generated sets of 10,000 earthquake catalogs of two kinds: one set conforming to the model under test, and the other derived from the observed catalog allowing for uncertainties in magnitude and hypocentral parameters. The distributions of L-scores obtained from both sets are in good agreement with those obtained by the proposed procedure. This comparison suggests that the analytical approach presented here could be useful for conducting the N-, L-, and R-tests in a conventional way.

### Key words

Seismicity models probabilistic prediction CSEP project N-, L-, and R-scores uncertainties in earthquake source parameters simulated catalogs## 1. Introduction

With the development of statistical seismology, probabilistic predictions of earthquakes are now more common than has hitherto been possible. The probabilities of earthquake occurrence are usually estimated based on specific seismicity models. To provide a reliable probabilistic prediction, the model should pass well-defined tests. The Collaboratory for the Study of Earthquake Predictability (CSEP) Project (Jordan, 2006) has been organized to solve questions related to earthquake predictability and to develop an adequate experimental infrastructure to conduct scientific prediction experiments under rigorous conditions. The aim of the CSEP Project is to evaluate each proposed model with three statistical tests, i.e., the N-, L-, and R-tests (Kagan and Jackson, 1995). In these tests, observed scores are compared with the distributions of respective scores expected from a proposed model. When the observed scores fall within an acceptance range, the model is not rejected.

In calculating observed scores, effects due to uncertainties in the earthquake source parameters of location, depth, and magnitude are also taken into account in the CSEP procedures (Schorlemmer et al., 2007). The underlying rationale for this step is that parameter uncertainties may cause earthquakes to be associated with bins differing from those originally assigned. A serious problem may arise if these uncertainties are ignored. For example, the involvement of a particular earthquake in the score can become largely a matter of chance if it is close to the boundaries of the test region or to the lower magnitude limit of the tests.

A large number of simulated catalogs are generated in the CSEP tests in order to obtain distributions of N-, L-, and R-scores expected from the proposed models (Schorlemmer et al., 2007). The observed scores are estimated using simulated catalogs derived from an original one by allowing for uncertainties in earthquake source parameters. Generating a large number of catalogs consumes a great deal of computational time that is proportional to the number of proposed models and time and space segments necessary for forecasting.

In this paper, we propose a method for conducting these tests analytically without generating simulated catalogs. This method is based on the assumptions that the rate in each cell is far less than unity and that at most one earthquake occurs in one cell. For each test, we analytically derive two sets of the mean and variance of the respective score: that expected from a proposed model (Imoto, 2009; Imoto and Rhoades, 2010) and that of the probable score if uncertainties in hypocenter parameters are taken into account. The central limit theorem allows us to regard the distribution of the scores as approximately normal, with the analytically obtained means and variances, if the number of earthquakes in the test area is large enough.

The proposed method is applied to two seismicity models for Kanto, central Japan. The first model (Hazmap model) is tentatively introduced as a candidate for testing and is a subset of a seismicity model used in estimating the recent seismic hazard maps for Japan. The second one is the EEPAS model (Rhoades and Evison, 2004, 2005), which was developed into a horizontal multi-layered model for seismicity in the Kanto region, central Japan (Rhoades and Evison, 2006; Imoto and Rhoades, 2010). The Hazmap model is configured to the tectonic setting in Kanto, where three plates (the Eurasian (North American, or Okhotsk) Plate, the Philippine Sea Plate, and the Pacific Plate) converge. Differences in configuration between the Hazmap and EEPAS models could be resolved by re-configuring the EEPAS model to be compatible with the Hazmap under a simple assumption. Thus, the Hazmap model could be compared with the EEPAS model in the R-test.

In order to compare our analytical approach with that involving the use of simulated catalogs, we have compared the distributions of L-scores by both methods. The results show that the means of L-scores for earthquakes conforming to a model under test and for those with parameter uncertainties taken into consideration are similar whether computed by our method or by the method involving simulated catalogs.

## 2. N-, L-, and R-Tests

In the N-, L-, and R-tests, scores obtained from the observed data are compared with a distribution of scores that could be expected assuming a given model to be correct. If the observed scores are not consistent with these distributions, the null hypothesis that the real earthquake sequences conform to the model may be rejected. If an observed score falls outside an acceptance range, the model should be rejected at the respective level of significance. In the following discussion of the three tests, we will introduce methods to estimate distributions of the test scores for a catalog with uncertain parameters. Using methods already presented in previous papers (Imoto, 2009; Imoto and Rhoades, 2010), these tests can be conducted without using simulated catalogs.

### 2.1 N-test

#### 2.1.1 N-score expected from a proposed model

*V*

_{0}cells, the number of earthquakes in each cell follows the Poisson process. The Poisson rate in the

*i*-th cell is noted as λ

_{ i }, i.e., the expected number of events, and its variance . The total expected number, E [

*n*(O

^{1})], is given bywhere the superscript 1 refers to the first model, and O

^{1}denotes data conforming to the first model. Earthquakes are assumed to be independent. The variance of the total number is given as follows.

*g*(

*n*):where

*µ*

_{1}and are given by E[

*n*(O

^{1})] and Var[

*n*(O

^{1})], respectively. In any case, the Poisson distribution itself exactly defines the acceptance range for the test.

#### 2.1.2 N-score from observed events with uncertain hypocentral parameters

*j*-th event could be located in

*m*

_{ j }different cells with probabilities of

*P*

_{ j,k }(

*k*= 1, 2,...

*m*

_{ j }). The observed number of events,

*n*(O

^{0}), expected from catalogs with errors in parameters taken into account is given bywhere O

^{0}denotes the actual earthquake occurrences in the study area, and

*N*

_{0}is the number of events, the possible parameters of which belong to the study area. If it is certain that a specific event is inside the study area, the summation for the event in Eq. (4) becomes 1. However, for events near the border of the study area, the summation is not necessarily 1. The variance of

*n*(O

^{0}) is given as

Under an assumption similar to that in the distribution of *g*(*n*), the observed number of earthquakes may be well represented by a normal distribution, *f*(*n*), with the mean and the variance given in Eqs. (4) and (5).

*n*(O

^{0}) represented by the distribution

*f*(

*n*), the expected value of the probability that the number of earthquakes in the model is less than

*n*(O) is given by

If falls outside the acceptance range (for example, 0.05–0.95), the proposed model should be rejected.

### 2.2 L-test

#### 2.2.1 L-score expected from a proposed model

*i*-th cell is given bywhere

*Y*

_{ i }is 1 when an earthquake occurs and 0 otherwise, and Ln refers to the natural logarithm. It is assumed that λ

_{ i }is far less than unity and that at most one earthquake occurs in one cell. This condition might be satisfied by less than 1 as a first order estimate. The expected value of the log likelihood, E[

*l*

_{ i }(O

^{1})] is expressed by

*l*O

^{1})] and variance Var[

*l*O

^{1})] of the log likelihood for the whole domain are given byand

#### 2.2.2 L-score from observed events with uncertain hypocentral parameters

*j*-th cell (and no event in any other cell) is then given as

*j*-th cell and no event, Δ

*l*

_{ j }(O

^{0}), is given by

If the event occurs in the *j*-th cell, not in the *j*-th cell, the difference in likelihood is given by the same formula with *j* in the place of *j*. If the location and/or magnitude parameters contain significant errors, an earthquake could be located in several different cells with certain probabilities.

*j*-th event is located in one of

*m*

_{ j }cells with probabilities , the expected value E[Δ

*l*

_{ j }(O

^{0})] of the difference in the log likelihood between the cases with and without the

*j*-th earthquake is given bywhere the subscript λ

_{ j,k }indicates the

*k*-th possible cell for the

*j*-th earthquake.

In a similar way as for the N-test, the L-test can be performed based on the parameters given in Eqs. (11), (12), (16), and (17).

### 2.3 R-test

#### 2.3.1 R-score expected from two models

*i*-th cell, , is expressed bywhere superscripts 1 and 2 refer to the first and second models, respectively. Here it is assumed that earthquake sequences conform to the first model (Imoto, 2009; Imoto and Rhoades, 2010). The variance is given by

#### 2.3.2 R-score from observed events with uncertain hypocentral parameters

*R*

^{12}] is estimated from Eq. (16).

In a similar manner as for the above tests, the R-test can be performed based on the parameters given in Eqs. (20), (21), (22), and (23).

## 3. Models and Data

### 3.1 Seismic hazard map model

Probabilistic seismic hazard maps were prepared for Japan based on the long-term probability of earthquakes (Fujiwara, 2004; Fujiwara et al., 2009). Future earthquakes in and around Japan are classified into several categories, such as along major and minor inland active faults, thrust faults along subduction zones, and others. The probabilistic seismic hazard estimates from every category are then merged into a total hazard.

In the present study, one candidate seismicity model tentatively considered is a subset of the long-term probability pertaining to *the category of earthquakes without specified faults* (Hazmap model). In this category, the long-term probability of earthquakes is estimated based on a smoothed seismicity of past earthquakes. The target earthquakes in our tested time-space volume all belong to this category.

### 3.2 EEPAS model

*h*

_{eep}(

*x*), is defined for any time

*t*, magnitude

*m*, and location (

*x*,

*y*) where m exceeds a threshold magnitude

*m*

_{c}, and (

*x*,

*y*) is a point in the region of surveillance

*R*. Each earthquake (

*t*

_{ i },

*m*

_{ i },

*x*

_{ i },

*y*

_{ i }) contributes a transient increment λ

_{ i }(

*t*,

*m*,

*x*,

*y*) to the future rate density in its vicinity, given bywhere

*f*

_{1i}is the density of the probability distributions for time,

*g*

_{1i}is that for magnitude, and

*h*

_{1i}is that for location. These densities take the formsandwhere

*a*

_{M},

*b*

_{M},

*s*

_{M},

*b*

_{T},

*b*

_{T},

*s*

_{T},

*s*

_{A}, and

*b*

_{A}are parameters; otherwise,

*H*(

*s*) = 1 if

*s*> 0 and 0.

The total rate density is obtained by summing over all past occurrences, including earthquakes outside *R* that could affect the rate density within *R*. More detail is given in previous studies (e.g., Evison and Rhoades, 2002, 2004; Rhoades and Evison, 2004). We do not distinguish the magnitudes of target earthquakes. Therefore, the integral form of the rate density is used. A minor modification of the EEPAS model was made to study three-dimensional seis-micity in the Kanto region, Japan (Rhoades and Evison, 2006), where the depth of 0–120 km is divided into six layers, and the EEPAS model is applied to each layer.

Differences in configuration between the Hazmap and EEPAS models (Fig. 1(b)) prevent us from performing the R-test. However, one simple assumption could enable us to make the EEPAS model compatible with the Hazmap model. Assuming that the cell size of the EEPAS model is so small that a uniform Poisson process is maintained in each cell, we could divide a cell into multiple pieces of arbitrary size, the hazard rates of which are estimated to be proportional to their volume. In our case, only the depth range for the EEPAS model differs from that of the Hazmap. Therefore, we performed “divide” and “connect” upon cells of the EEPAS model to make them compatible for the depth range only, since the horizontal grid spacing of 0.1 × 0.1° is common between the two models. The modified EEPAS model thus obtained is adapted in the present study for comparison with the Hazmap model. Hereafter, we refer to this modified EEPAS model as the EEPAS model.

### 3.3 Data

List of target earthquakes. “Prob” in the last column indicates the probability that the earthquake falls in the study volume.

Date | LAT(deg) | LON(deg) | DEP(km) | Mag | Prob | |
---|---|---|---|---|---|---|

1 | 20040311 | 36.322 | 141.007 | 47.5 | 5.3 | 0.99908 |

2 | 20040404 | 36.390 | 141.153 | 49.0 | 5.8 | 1.00000 |

3 | 20040710 | 36.080 | 139.885 | 48.2 | 4.7 | 0.00621 |

4 | 20040821 | 35.037 | 141.480 | 38.4 | 5.4 | 0.97818 |

5 | 20041006 | 35.988 | 140.088 | 66.0 | 5.7 | 0.99992 |

6 | 20041017 | 36.255 | 141.328 | 49.0 | 5.7 | 0.99999 |

7 | 20050101 | 36.783 | 140.983 | 89.4 | 5.0 | 0.69145 |

8 | 20050208 | 36.140 | 140.085 | 67.0 | 4.8 | 0.06681 |

9 | 20050216 | 36.038 | 139.888 | 46.2 | 5.3 | 0.99977 |

10 | 20050411 | 35.727 | 140.620 | 51.5 | 6.1 | 0.99839 |

11 | 20050515 | 36.628 | 139.483 | 8.8 | 4.8 | 0.06681 |

12 | 20050515 | 35.755 | 140.702 | 50.8 | 4.7 | 0.00621 |

13 | 20050519 | 35.558 | 141.082 | 33.3 | 5.4 | 0.99984 |

14 | 20050530 | 35.690 | 140.707 | 51.2 | 4.7 | 0.00621 |

15 | 20050620 | 35.733 | 140.693 | 50.7 | 5.6 | 0.99964 |

16 | 20050711 | 36.413 | 141.040 | 47.8 | 4.7 | 0.00621 |

17 | 20050723 | 35.582 | 140.138 | 73.1 | 6.0 | 1.00000 |

18 | 20050728 | 36.125 | 139.845 | 51.1 | 5.0 | 0.69146 |

19 | 20050807 | 35.558 | 140.113 | 73.3 | 4.7 | 0.00621 |

20 | 20050808 | 36.338 | 141.445 | 46.0 | 5.6 | 0.99997 |

21 | 20050909 | 35.590 | 140.947 | 37.5 | 5.0 | 0.69146 |

22 | 20051016 | 36.038 | 139.937 | 47.1 | 5.1 | 0.93319 |

23 | 20051019 | 36.382 | 141.042 | 48.3 | 6.3 | 1.00000 |

24 | 20051116 | 36.308 | 141.065 | 47.8 | 4.8 | 0.06681 |

25 | 20051228 | 36.182 | 140.027 | 53.0 | 4.8 | 0.06681 |

26 | 20060201 | 35.760 | 140.003 | 101.0 | 5.1 | 0.93305 |

27 | 20060203 | 36.213 | 141.610 | 62.1 | 5.9 | 0.00587 |

28 | 20060520 | 35.202 | 140.112 | 73.8 | 4.7 | 0.00621 |

29 | 20060831 | 35.632 | 140.023 | 75.6 | 4.8 | 0.06681 |

30 | 20060907 | 35.590 | 141.057 | 38.2 | 5.1 | 0.93316 |

31 | 20070314 | 35.705 | 140.710 | 52.4 | 4.7 | 0.00621 |

32 | 20070620 | 35.235 | 141.297 | 24.9 | 5.0 | 0.69146 |

33 | 20070621 | 35.198 | 141.408 | 38.8 | 4.8 | 0.06681 |

34 | 20070816 | 35.443 | 140.530 | 30.8 | 5.3 | 0.99977 |

35 | 20070818 | 35.342 | 140.345 | 20.2 | 5.2 | 0.99379 |

36 | 20070921 | 35.248 | 141.215 | 35.0 | 5.1 | 0.93316 |

37 | 20071001 | 35.225 | 139.118 | 14.2 | 4.9 | 0.30854 |

38 | 20071007 | 35.683 | 140.725 | 44.5 | 4.8 | 0.06680 |

39 | 20071130 | 36.427 | 140.695 | 52.9 | 4.7 | 0.00621 |

40 | 20080116 | 36.423 | 141.180 | 47.0 | 4.7 | 0.00621 |

41 | 20080209 | 36.735 | 141.205 | 46.3 | 4.9 | 0.30813 |

42 | 20080308 | 36.452 | 140.612 | 57.0 | 5.2 | 0.99379 |

43 | 20080401 | 34.945 | 140.585 | 98.1 | 4.7 | 0.00141 |

44 | 20080404 | 36.120 | 139.827 | 53.5 | 5.0 | 0.69103 |

45 | 20080425 | 35.683 | 140.722 | 48.2 | 4.8 | 0.06679 |

46 | 20080508 | 36.227 | 141.607 | 50.6 | 7.0 | 0.01178 |

47 | 20080705 | 36.642 | 140.952 | 49.7 | 5.2 | 0.99379 |

48 | 20080822 | 36.442 | 140.615 | 55.9 | 5.2 | 0.99377 |

49 | 20080906 | 35.355 | 141.245 | 40.3 | 4.7 | 0.00621 |

50 | 20080921 | 35.615 | 140.062 | 70.8 | 4.8 | 0.06680 |

51 | 20081008 | 35.583 | 140.142 | 63.1 | 4.7 | 0.00621 |

52 | 20081231 | 35.532 | 140.940 | 42.7 | 4.7 | 0.00621 |

## 4. Results

*M*= 5.0) for the cases without (with) errors in magnitude. These figures indicate that each model is self-consistent from the viewpoint of earthquake productivity, with the cumulative numbers falling within the uncertainty ranges in four of the five comparisons made.

Figure 4(b) compares the observed R-score with that expected from distributions under the EEPAS model. The intersections of broken lines and solid curves remain in the acceptance range (between 5% and 95%). This would be maintained even if we consider the extreme value of R-score that is estimated as side lobes of the normal density function at the bottom. Based on this result, the null hypothesis that earthquake occurrences conform to the EEPAS model rather than the Hazmap model cannot be rejected.

## 5. Discussion

In this paper, we have derived the means and variances of the distributions for N-, L-, and R-scores in the cases of (1) seismicity compatible with the proposed model and (2) catalogs with errors in the observed parameters. With these means and variances, N-, L-, and R-tests could be performed for the proposed models in a simple way if distributions associated with the tests are well approximated by normal distributions with means and variances. This approximation is basically guaranteed by the central-limit theorem if the number of earthquakes is sufficiently large. For the first case in Section 2.1.1, the Poisson distribution is approximated with a normal distribution. This approximation is likely accomplished with a total expected number of earthquakes exceeding 10.

The next case, presented in Section 2.1.2, is that in which binomial distributions are approximated by a normal distribution, and it should be carefully considered. One of the various rules that may be used to decide whether a sample size is large enough for this approximation is that both the expected value and the value of the sample size minus the expected value must be greater than 5. If we consider a case of only errors in location, only a small proportion of earthquakes contributes to the variance of *n*(O^{0}) in Eq. (5), i.e., only earthquakes near the border of the test area contribute to the variance since only such events are origins of changes in *n*(O^{0}).

Therefore, the above rule for the approximation may not be satisfied. However, if we consider the case of errors both in locations and magnitude, the rule must be satisfied in most cases. For example, provided that we observe ten earthquakes with magnitudes exceeding the cutoff *M*_{c} in the test area and the standard deviation of the magnitude determination is 0.2, we expect about 21 earthquakes in the magnitude range between *M*_{c} - 0.4 and *M*_{c} + 0.4, where the *b*-value of the Gutenberg-Richter relation is assumed to be 1. We could expect seven earthquakes larger than *M*_{c} and 14 less than this, which may satisfy the above rule.

In our case, we presume the standard deviation of the magnitude determination to be 0.1, and earthquakes of magnitude =4.7 are considered in our test. In total, 51 events are used, among which 24 events are registered with magnitude <5.0. The expected number of events for the five years is 25.4 and is given as the summation of the probabilities in the last column of Table 1. Accordingly, the sample size minus the expected value becomes 25.6, which is much greater than 5.0 and satisfies the above rule.

In summary, we present here methods by which to perform N-, L-, and R-tests without generating random catalogs conforming to the test models or catalogs modified from the observed catalog with uncertainties in the parameters. We applied the proposed method to the Hazmap and EEPAS models in Kanto, central Japan. The N- and L-tests for the years 2004–2008 confirm the self-consistency of both models. The values for the R-test for the last 5 years suggest that the EEPAS model is superior to the Hazmap model over this period. This 5-year comparison is in no way a meaningful test of the Hazmap model itself, which is a long-term model designed for a time period of 30 years. Rather, it suggests that the time-varying EEPAS model contains information about earthquake occurrence on a 5-year timescale that is not contained in longer term estimates of seismicity.

Our comparison of L-scores derived analytically with those derived from simulated catalogs indicates no significant difference between the two sets of scores, thus implying that the proposed method is an alternative to the current one. However, caution is warranted because the analytical method is only reliable when the assumptions on which it is based are satisfied.

## Notes

## Declarations

### Acknowledgments

This manuscript was greatly improved by the comments of an anonymous reviewer and K. Nanjo.

## Authors’ Affiliations

## References

- Evison, F. F. and D. A. Rhoades, Precursory scale increase and long-term seismogenesis in California and northern Mexico,
*Ann. Geophys.*,**454**, 479–495, 2002.Google Scholar - Evison, F. F. and D. A. Rhoades, Demarcation and scaling of long-term seismogenesis,
*Pure Appl. Geophys.*,**161**, 21–45, 2004.View ArticleGoogle Scholar - Fujiwara, H., National seismic hazard mapping project of Japan,
*Proceedings of the 5th U.S.-Japan Natural Resources Meeting, U.S. Geological Survey, Open-File Report*2005–1131, 14, 2004.Google Scholar - Fujiwara, H., S. Kawai, S. Aoi, N. Morikawa, S. Senna, N. Kudo, M. Ooi, K. Hao, K. Wakamatsu, Y. Ishikawa, T. Okumura, T. Ishii, S. Matsushima, Y. Hayakawa, N. Toyama, and A. Narita, A study on “National seismic hazard maps for Japan”, Technical Note National Research Institute for Earth Science and Disaster Prevention, No336, 2009 (in Japanese).Google Scholar
- Imoto, M., Comments on the N-, L- and R-tests for seismicity models,
*Zisin 2*,**61**, 207–209, 2009 (in Japanese).View ArticleGoogle Scholar - Imoto, M. and D. A. Rhoades, Seismicity models of moderate earthquakes in Kanto, Japan, utilizing multiple predictive parameters,
*Pure Appl. Geophys.*,**167**, 831–843, 2010.View ArticleGoogle Scholar - JMA, The Seismological and Volcanological Bulletin of Japan, Japan Meteorological Agency, ISSN 1349–8320, 2009.Google Scholar
- Jordan, T. H., Earthquake predictability, brick by brick,
*Seismol. Res. Lett.*,**77**, 3–6, 2006.View ArticleGoogle Scholar - Kagan, Y. Y. and D. D. Jackson, New seismic gap hypothesis: five year after,
*J. Geophys. Res.*,**100**, 3943–3960, 1995.View ArticleGoogle Scholar - NIED, J-SHIS at http://wwwold.j-shis.bosai.go.jp/j-shis/index en.html (as of July 1, 2010).Google Scholar
- Rhoades, D. A. and F. F. Evison, Long-range earthquake forecasting with every earthquake a precursor according to scale,
*Pure Appl. Geophys.*,**161**, 47–72, 2004.View ArticleGoogle Scholar - Rhoades, D. A. and F. F. Evison, Test of the EEPAS forecasting model on the Japan earthquake catalogue,
*Pure Appl. Geophys.*,**162**, 1271–1290, 2005.View ArticleGoogle Scholar - Rhoades, D. A. and F. F. Evison, The EEPAS forecasting model and the probability of moderate-to-large earthquakes in central Japan,
*Tectono-physics*,**417**, 119–130, 2006.View ArticleGoogle Scholar - Schorlemmer, D., M. C. Gerstenberger, S. Wiemer, D. D. Jackson, and D. A. Rhoades, Earthquake likelihood model testing,
*Seismol. Res. Lett.*,**78**, 17–29, 2007.View ArticleGoogle Scholar