A candidate secular variation model for IGRF-13 based on MHD dynamo simulation and 4DEnVar data assimilation

Abstract

We have submitted a secular variation (SV) candidate model for the thirteenth generation of International Geomagnetic Reference Field model (IGRF-13) using a data assimilation scheme and a magnetohydrodynamic (MHD) dynamo simulation code. This is the first contribution to the IGRF community from research groups in Japan. A geomagnetic field model derived from magnetic observatory hourly means, and CHAMP and Swarm-A satellite data, has been used as input data to the assimilation scheme. We adopt an ensemble-based assimilation scheme, called four-dimensional ensemble-based variational method (4DEnVar), which linearizes outputs of MHD dynamo simulation with respect to the deviation from a dynamo state vector at an initial condition. The data vector for the assimilation consists of the poloidal scalar potential of the geomagnetic field at the core surface and flow velocity field slightly below the core surface. Dimensionless time of numerical geodynamo is adjusted to the actual time by comparison of secular variation time scales. For SV prediction, we first generate an ensemble of dynamo simulation results from a free dynamo run. We then assimilate the ensemble to the data with a 10-year assimilation window through iterations, and finally forecast future SV by the weighted sum of the future extension parts of the ensemble members. Hindcast of the method for the assimilation window from 2004.50 to 2014.25 confirms that the linear approximation holds for 10-year assimilation window with our iterative ensemble renewal method. We demonstrate that the forecast performance of our data assimilation and forecast scheme is comparable with that of IGRF-12 by comparing data misfits 4.5 years after the release epoch. For estimation of our IGRF-13SV candidate model, we set assimilation window from 2009.50 to 2019.50. We generate our final SV candidate model by linear fitting for the weighted sum of the ensemble MHD dynamo simulation members from 2019.50 to 2025.00. We derive errors of our SV candidate model by one standard deviation of SV histograms based on all the ensemble members.

Introduction

The secular variation (SV) of the geomagnetic field is caused by a highly nonlinear process in the outer core of the Earth. International Association of Geomagnetism and Aeronomy (IAGA) releases a set of geomagnetic main field model and linear prediction of SV field as International Geomagnetic Reference Field (IGRF) model every 5 years, to keep the model’s deviation from the actual field small. The previous generation, the 12th generation of IGRF, was released in 2015 (Thébault et al. 2015a). Each generation of IGRF has been produced by the IAGA Working Group V-MOD by integrating independent contributions from research groups all over the world, and consists of three types of models: the Definitive IGRF (DGRF) field 5 years before the release epoch, the geomagnetic main field for the release epoch, and the SV model valid for the 5 years on and after the release epoch. The truncation of spherical harmonics for IGRF and DGRF is now $L_{\text{MF}} = 13$, while for SV is $L_{\text{SV}} = 8$. Evaluations of all the submitted candidate models are unveiled altogether with the release of IGRF (e.g., Thébault et al. 2015b).

Research groups in Japan have never contributed to any IGRF models so far, although there have been important contributions of highly quality-controlled observatory data, e.g., Kakioka, operated by Japan Meteorological Agency. Since both established techniques and expertise for the fields of magnetohydrodynamic (MHD) dynamo simulation and data assimilation are present in Japan, we decided to contribute to the IGRF community by an SV candidate model first. This was actually enabled by a bilateral collaboration between Japan and the geomagnetic group in Institut de Physique du Globe de Paris (IPGP), France; this group brought to the project the required expertise to build, from raw data, a geomagnetic field model, which, in turn, has been used as input data to our assimilation scheme.

SV candidate models for IGRF have been commonly presented as linear or polynomial extrapolation of known latest SVs. The GFZ Potsdam team, Germany, for example, generated an SV candidate model for IGRF-12, namely SV from 2015.0 to 2020.0, by averaging the SV of their parent model from 2013.5 to 2014.5 (Lesur et al. 2015). Abrupt change in SV, such as geomagnetic jerks, however, may deteriorate the prediction of short-time SVs. This suggests necessity of prediction reflecting dynamic process in the Earth’s outer core, e.g., through numerical geodynamo models. There have been limited attempts to use geodynamo models for contributions to the SV models of IGRF. Kuang et al. (2010) was the first to contribute to IGRF-SV models using a geodynamo-based data assimilation scheme. They adopted a Kalman filter approach and used over 7000-year data with a prediction–correction algorithm to reduce the systematic error of the geodynamo model. In addition to the full MHD prediction, the number of attempts to use controlled dynamic processes has been increasing. Fournier et al. (2015) provided an SV model for IGRF-12, using a geodynamo model (Aubert et al. 2013) with a steady fluid velocity in the outer core.

Variational data assimilation approaches also have a high possibility for providing good forecasts of the geomagnetic field, although there has been a very limited number of their applications to the geomagnetic field. The variational approaches simultaneously adjust a dynamic model to all the observations available in the assimilation window, while the sequential methods, such as the Kalman filter approach, update (or correct) the model output with given observations at each time step (see Talagrand (1997) or Fournier et al. (2010) for introduction to both). One of big advantages of the variational approaches is that the resulting trajectory of the state vector keeps the dynamic consistency of the system over the assimilation window. Fournier et al. (2007) show a possibility that the variational approach can improve historical geomagnetic data with the recent dense satellite magnetic data, from experiments using a one-dimensional MHD dynamo toy model. Li et al. (2011; 2014) have developed variational assimilation schemes for more realistic three-dimensional MHD systems. In particular, Li et al. (2014) performed experiments of variational data assimilation using an inertia-free MHD dynamo model with synthetic data sets, which shows a potential to handle actual magnetic observations in the variational assimilation scheme. For their application to real magnetic data, however, difficulties arise from the requirement of an adjoint model of a fully nonlinear MHD dynamo model and from computational cost for construction of the derivative of the cost function with respect to an initial state, i.e., for backward propagation of the error information (e.g., Fournier et al. 2010).

In this study, we adopt an iterative ensemble-based variational scheme based on the four-dimensional ensemble-based variational method (4DEnVar; Liu et al. 2008) with a fully nonlinear three-dimensional MHD dynamo model (Takahashi 2012, 2014) to predict SV in the coming 5 years from 2020.0. A noticeable feature of 4DEnVar is that it linearizes the dynamo output variables at a given time comparable to observations with respect to the deviation of the state vector from a mean state at the initial time, assuming weak nonlinearity of the model over the assimilation window. The 4DEnVar approach, thereby, does not need an adjoint model of the forward model and numerical cost for the backward propagation of the error information, even though it is named “variational method”. This strategy seems reasonable when both assimilation window and forecast period are shorter than the timescale of nonlinearity of the system. We illustrate that this approach is feasible with our iterative algorithm based on 4DEnVar (Nakano 2020) for generation of a candidate SV model for IGRF, with numerical experiments using the real geomagnetic data from 2004.50 to 2014.25 and with comparison of the forecast performances with that of IGRF-12.

The remainder of this paper is organized as follows: we first explain our assimilation method (“Method” section). Secondly, we report results of numerical experiments using the past datasets (“Numerical experiments” section), and then describe details of estimation of our candidate model for IGRF-13 SV (“The SV candidate model for IGRF-13” section). After the explanation of our error estimation (“Error estimation of the candidate SV model” section), we summarize this study (“Discussion and conclusions” section).

Method

Our intent is to forecast SV by performing the 4DEnVar data assimilation using a numerical geodynamo code and existing data with respect to the poloidal scalar potential of the geomagnetic field at the core–mantle boundary (CMB) and the toroidal and poloidal components of the core surface flow. From the variational data assimilation experiments with an inertia-free MHD dynamo model, Li et al. (2014) found that the magnetic field below the CMB is hard to reconstruct only from the magnetic data taken outside the core due to the diffusion-dominant Ekman boundary layer. In agreement with their proposal, we use the preliminary estimated core surface flow as part of observation vectors in our data assimilation as well as the geomagnetic data. Inclusion of the core surface flow in data vectors results in indirect inclusion of SV data in the data assimilation.

In this section, we first describe the assimilation theory (“Data assimilation theory” section) and details of our dynamo model and how to adjust dimensionless time to the actual time (“Geodynamo simulation: parameters and scaling of time” section). We then briefly discuss the nonlinearity of our dynamo model (“Nonlinearity of the numerical geodynamo: error growth e-folding time” section). Next, we explain details of preparation of the observational data and how to convert dimensionless simulation outputs to variables comparable to real data (“Data 1: poloidal scalar potential at the CMB obtained from the MCM model” section and “Data 2: core surface flow” section) and finally describe the way of practical implementation of the data assimilation (“Implementation of assimilation” section).

Data assimilation theory

We consider the minimization of the following cost function:

$$V\left( {\varvec{x}_{0} } \right) = \frac{1}{2}\mathop \sum \limits_{k = 1}^{K} \left[ {\varvec{y}_{k} - \varvec{h}_{k} \left( {\varvec{x}_{k} } \right)} \right]^{\text{T}} {\mathbf{R}}_{k}^{ - 1} \left[ {\varvec{y}_{k} - \varvec{h}_{k} \left( {\varvec{x}_{k} } \right)} \right],$$

(1)

where $\varvec{x}_{k}$ is the state vector of a dynamo model at time $t_{k}$, $\varvec{y}_{k}$ denotes the observation vector, ${\mathbf{R}}_{k}$ is the covariance matrix of observation noise, and $\varvec{h}_{k}$ is an observation operator which converts a state vector $\varvec{x}_{k}$ to observable variables for the comparison with $\varvec{y}_{k}$. Given the dynamo model, $\varvec{x}_{k}$ is uniquely determined from the initial state $\varvec{x}_{0}$. This allows us to represent $\varvec{x}_{k}$ as a function of $\varvec{x}_{0}$, that is, $\varvec{x}_{k} = \varvec{f}_{k} \left( {\varvec{x}_{0} } \right)$. Defining a function $\varvec{g}_{k}$ as $\varvec{g}_{k} \left( {\varvec{x}_{0} } \right) = \varvec{h}_{k} \left( {\varvec{f}_{k} \left( {\varvec{x}_{0} } \right)} \right)$, the cost function in Eq. (1) can be rewritten as follows:

$$V\left( {\varvec{x}_{0} } \right) = \frac{1}{2}\mathop \sum \limits_{k = 1}^{K} \left[ {\varvec{y}_{k} - \varvec{g}_{k} \left( {\varvec{x}_{0} } \right)} \right]^{\text{T}} {\mathbf{R}}_{k}^{ - 1} \left[ {\varvec{y}_{k} - \varvec{g}_{k} \left( {\varvec{x}_{0} } \right)} \right].$$

(2)

The minimization of this cost function is achieved by an iterative algorithm based on the 4DEnVar method (Liu et al. 2008). At the $m$th iteration, we approximate the cost function by using an ensemble of the simulation outputs $\left\{ {\varvec{x}_{0:K,m}^{\left( 1 \right)} \ldots ,\varvec{x}_{0:K,m}^{\left( N \right)} } \right\}$, where $N$ is the size of ensemble and $\varvec{x}_{0:K,m}^{\left( n \right)} , n \in \left\{ {1, \ldots ,N} \right\}$ is the sequence of vectors $\varvec{x}_{0,m}^{\left( n \right)} , \varvec{x}_{1,m}^{\left( n \right)} , \ldots , \varvec{x}_{K,m}^{\left( n \right)}$. This ensemble is calculated by MHD dynamo simulations from the initial conditions $\left\{ {\varvec{x}_{0,m}^{\left( 1 \right)} , \ldots ,\varvec{x}_{0,m}^{\left( N \right)} } \right\}$, which are prepared so that the ensemble mean, $\left( {{{\varSigma }}_{n = 1}^{N} \varvec{x}_{0,m}^{\left( n \right)} } \right)/N$, is equal to the $m$th estimate $\bar{\varvec{x}}_{0,m}$. At the $m$th iteration, we seek $\varvec{x}_{0}$ that minimizes Eq. (2), which turns out to be $\bar{\varvec{x}}_{0,m + 1}$, with given $\bar{\varvec{x}}_{0,m}$ and $\left\{ {\varvec{x}_{0:K,m}^{\left( 1 \right)} , \ldots ,\varvec{x}_{0:K,m}^{\left( N \right)} } \right\}$. As an important first step of the 4DEnVar, we express $\varvec{g}_{k} \left( {\varvec{x}_{0} } \right)$ in terms of the first-order Taylor expansion,

$$\varvec{g}_{k} \left( {\varvec{x}_{0} } \right) \approx \varvec{g}_{k} \left( {\bar{\varvec{x}}_{0,m} } \right) + \varvec{G}_{k} \left( {\varvec{x}_{0} - \bar{\varvec{x}}_{0,m} } \right),$$

(3)

where $\varvec{G}_{k}$ is the Jacobian of $\varvec{g}_{k}$ at $\bar{\varvec{x}}_{0,m}$. We then approximate $\varvec{x}_{0}$ as a weighted sum of the ensemble members. Now we define the following matrices ${\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{X} }}_{0,m}$ and ${\hat{\mathbf{\varGamma }}}_{k,m}$ for convenience:

$${\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{X} }}_{0,m} = \frac{1}{{\sqrt {N - 1} }}\left( {\varvec{x}_{0,m}^{\left( 1 \right)} - \bar{\varvec{x}}_{0,m} \cdots \varvec{x}_{0,m}^{\left( N \right)} - \bar{\varvec{x}}_{0,m} } \right),$$

(4)

$$\begin{aligned}{\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\Gamma } }}_{k,m} = \frac{1}{{\sqrt {N - 1} }}\left( {\varvec{g}_{k} \left( {\varvec{x}_{0,m}^{\left( 1 \right)} } \right) - \varvec{g}_{k} \left( {\bar{\varvec{x}}_{0,m} } \right) \cdots \varvec{g}_{k} \left( {\varvec{x}_{0,m}^{\left( N \right)} } \right) - \varvec{g}_{k} \left( {\bar{\varvec{x}}_{0,m} } \right)} \right).\end{aligned}$$

(5)

This allows us to write $\varvec{x}_{0} = \bar{\varvec{x}}_{0,m} + {\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{X} }}_{0,m} \varvec{w}$, where $\varvec{w}$ consists of weight for each ensemble member. $\bar{\varvec{x}}_{0,m}$ is the mean of $\varvec{x}_{0,m}^{\left( n \right)} \left( {n = 1, \ldots ,N} \right)$. Using Eqs. (4) and (5), the function $\varvec{g}_{k} \left( {\varvec{x}_{0} } \right)$ in Eq. (3) can then be expressed:

$$\varvec{g}_{k} \left( {\varvec{x}_{0} } \right) \approx \varvec{g}_{k} \left( {\bar{\varvec{x}}_{0,m} } \right) + \varvec{G}_{k} {\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{X} }}_{0,m} \varvec{w} \approx \varvec{g}_{k} \left( {\bar{\varvec{x}}_{0,m} } \right) + {\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\Gamma } }}_{k,m} \varvec{w}.$$

(6)

Note here that the Jacobian $\varvec{G}_{k}$ disappears in the expression of $\varvec{g}_{k} \left( {\varvec{x}_{0} } \right)$ with the aid of the relationship derived from Eq. (3), $\varvec{g}_{k} \left( {\varvec{x}_{0,m}^{\left( n \right)} } \right) - \varvec{g}_{k} \left( {\bar{\varvec{x}}_{0,m} } \right) \approx \varvec{G}_{k} \left( {\varvec{x}_{0,m}^{\left( n \right)} - \bar{\varvec{x}}_{0,m} } \right), \left( {n = 1, \ldots ,N} \right)$. Equation (6) allows us to circumvent direct calculation of the Jacobian $\varvec{G}_{k}$. On the other hand, the linear approximation Eq. (3) (and resulting Eq. (6)) imposes us on two requirements:

• (I) the assimilation window indexed by $k = 1, \ldots ,K$ is so short that nonlinearity of $\varvec{g}_{k} \left( {\varvec{x}_{0} } \right)$ is negligible (or weak);

• (II) the deviations, $\varvec{x}_{0,m}^{\left( n \right)} - \bar{\varvec{x}}_{0,m}$ ($n = 1, \ldots ,N)$ in Eq. (4), are small enough.

For the first requirement, we discuss the nonlinearity of our dynamo model using the error growth rate (Hulot et al. 2010) in “Nonlinearity of the numerical geodynamo” section later, while we see that ensembles shrinking through iterations meet the second requirement in “Numerical experiments” section. From Eqs. (2) and (6), we introduce the following objective function;

$$\hat{J}_{m} \left( \varvec{w} \right) = \frac{{\sigma_{m}^{2} }}{2}\varvec{w}^{\text{T}} \varvec{w} + \frac{1}{2}\mathop \sum \limits_{k = 1}^{K} \left[ {\varvec{y}_{k} - \varvec{g}_{k} \left( {\bar{\varvec{x}}_{0,m} } \right) - {\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\Gamma } }}_{k,m} \varvec{w}} \right]^{\text{T}} {\mathbf{R}}_{k}^{ - 1} \left[ {\varvec{y}_{k} - \varvec{g}_{k} \left( {\bar{\varvec{x}}_{0,m} } \right) - {\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\Gamma } }}_{k,m} \varvec{w}} \right],$$

(7)

where ${{\sigma }}_{m}$ is a parameter, which is fixed to $\sigma_{m} = 1$ in this study, while we decrease elements of ${\mathbf{R}}_{k}$ at each step. This cost function is minimized provided that:

$$\hat{\varvec{w}}_{m} = \left( {\mathop \sum \limits_{k} \left[ {\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}} {\Gamma}}_{k,m}^{\text{T}} {\mathbf{R}}_{k}^{ - 1} {\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\Gamma } }}_{k,m} } \right] + \sigma_{m}^{2} \varvec{I }} \right)^{ - 1} \mathop \sum \limits_{k} \left( {\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\Gamma }}_{k,m}^{\text{T}} {\mathbf{R}}_{k}^{ - 1} \left[ {\varvec{y}_{k} - \varvec{g}_{k} \left( {\bar{\varvec{x}}_{0,m} } \right)} \right]} \right).\varvec{ }$$

(8)

The $\left( {m + 1} \right)$th estimate $\bar{\varvec{x}}_{0,m + 1}$ is then obtained as

$$\bar{\varvec{x}}_{0,m + 1} = \bar{\varvec{x}}_{0,m} + {\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{X} }}_{0,m} \hat{\varvec{w}}_{m} ,$$

(9)

and we proceed to the next iteration. The first term of the right-hand side in Eq. (7) is added to ensure robustness. This iterative application of Eq. (8), which is similar to the iterative ensemble Kalman smoother algorithm (Gu and Oliver 2007; Bocquet and Sakov 2013), minimizes Eq. (2) in the subspace spanned by the ensemble members (Nakano 2020). After obtaining $\bar{\varvec{x}}_{0,m + 1}$ it is necessary to perform MHD dynamo simulations with a set of initial conditions $\left\{ {\varvec{x}_{0,m + 1}^{\left( 1 \right)} , \ldots ,\varvec{x}_{0,m + 1}^{\left( N \right)} } \right\}$ to renew the ensemble members for the ($m + 1$)th iteration. See Appendix A for how to prepare the set of initial conditions from $\bar{\varvec{x}}_{0,m + 1}$, ${\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{X} }}_{0,m}$ and $\hat{\varvec{w}}_{m}$.

At the final (5th is chosen in this study) iteration, we also estimate the bias and trend components which correspond to model error in the dynamo model, by minimizing the following function:

$$\begin{aligned} \hat{J}_{m} \left( {\varvec{w},\varvec{b},\varvec{a}} \right) & = \frac{{\sigma_{m}^{2} }}{2}\varvec{w}^{\text{T}} \varvec{w} + \frac{1}{2}\varvec{b}^{\text{T}} {\mathbf{P}}_{b}^{ - 1} \varvec{b} + \frac{1}{2}\varvec{a}^{T} {\mathbf{P}}_{a}^{ - 1} \varvec{a} \\ & \quad + \frac{1}{2}\mathop \sum \limits_{k = 1}^{K} \left[ {\varvec{y}_{k} - \varvec{g}_{k} \left( {\bar{\varvec{x}}_{0,m} } \right) - {\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\Gamma } }}_{k,m} \varvec{w} - \varvec{b} - k\varvec{a}} \right]^{\text{T}} \\ & \quad \times {\mathbf{R}}_{k}^{ - 1} \left[ {\varvec{y}_{k} - \varvec{g}_{k} \left( {\bar{\varvec{x}}_{0,m} } \right) - {\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\Gamma } }}_{k,m} \varvec{w} - \varvec{b} - k\varvec{a}} \right], \\ \end{aligned}$$

(10)

where $\varvec{b}$ denotes the bias component, while $\varvec{a}$ is the coefficient for the trend component. The bias and trend terms correspond to the offset and the linear departure in time between the observation and the model, respectively. Here, we assume that the observation is mostly explained by the dynamo model output and that the bias and trend components are minor. We thus select ${\mathbf{P}}_{a}$ and ${\mathbf{P}}_{b}$ as:

$${\mathbf{P}}_{a} = {\mathbf{P}}_{b} = 10^{ - 4} {\mathbf{R}}_{k} .$$

(11)

Large norms of ${\mathbf{P}}_{b}^{ - 1}$ and ${\mathbf{P}}_{a}^{ - 1}$ suppress intensities of $\varvec{b}$ and $\varvec{a}$ while minimizing Eq. (10). $\varvec{w},\varvec{b},\varvec{a}$ that minimize $\hat{J}_{m} \left( {\varvec{w},\varvec{b},\varvec{a}} \right)$ in Eq. (10) can be obtained in a similar manner to Eq. (8) (see Appendix B for details). The minimization of Eq. (10) gives the approximate minimum of the following cost function:

$$V\left( {\varvec{x}_{0} } \right) = \frac{1}{2}\mathop \sum \limits_{k = 1}^{K} \left[ {\varvec{y}_{k} - \varvec{g}_{k} \left( {\varvec{x}_{0} } \right) - \varvec{b} - k\varvec{a}} \right]^{\text{T}} {\mathbf{R}}_{k}^{ - 1} \left[ {\varvec{y}_{k} - \varvec{g}_{k} \left( {\varvec{x}_{0} } \right) - \varvec{b} - k\varvec{a}} \right].$$

(12)

The final estimate and prediction are obtained by the following equation:

$$\bar{\varvec{g}}_{k,M} = \varvec{g}_{k} \left( {\bar{\varvec{x}}_{0,M} } \right) + {\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\varGamma } }}_{k,M} \hat{\varvec{w}}_{M} + \hat{\varvec{b}} + k\hat{\varvec{a}},\varvec{ }$$

(13)

where $M$ indicates the final step, i.e., $M = 5$, and $\hat{\varvec{w}}_{{_{M} }}$, $\hat{\varvec{b}}$ and $\hat{\varvec{a}}$ are solutions to Eq. (10). Then we can obtain the final estimate of Eq. (13) by the sum of a single MHD simulation starting from $\bar{\varvec{x}}_{0,M}$, the weighted sum of the Mth ensemble members, and the trend and bias terms. Note that we can use Eq. (13) not only for the final estimate within the assimilation window, but also for the estimate in the forecast period outside the assimilation window when the future extensions of $\varvec{g}_{k} \left( {\bar{\varvec{x}}_{0,M} } \right)$ and $\varvec{g}_{k} \left( {\varvec{x}_{0,M}^{\left( n \right)} } \right)$, $(K < k)$, are available, which requires only additional dynamo runs for $\varvec{g}_{k} \left( {\bar{\varvec{x}}_{0,M} } \right)$ and all the ensemble members. Figure 1 shows how to prepare an SV model for IGRF-13 by our assimilation scheme and Eq. (13), where the future extensions of the ensemble members (the gray area after “Release of IGRF”) generate a future prediction (the blue line after “Release of IGRF”) by the weighted sum of the ensemble members via Eq. (13).

Fig. 1

A schematic figure of the 4DEnVar and prediction of the future geomagnetic field. The weighted sum of an ensemble of dynamo run (the thick blue line) is the final estimate after the assimilation corresponding to Eq. 13. The SV candidate model for IGRF-13SV (the red dashed line) is determined by line-fitting to the MHD expectation of the next 5 years. One can use geomagnetic data up to ~ 6 months before the release of IGRF for generation of candidate models because the deadline of their submissions is ~ 3 months before the release (October 1st for IGRF-13)

Note that the first-order approximation of the function $\varvec{f}_{k}$ allows us to approximate the state at an arbitrary time $t_{{k^{\prime}}}$ ($0 \le k^{\prime} \le K$) in a similar manner to Eq. (6):

$$\varvec{x}_{{k^{\prime}}} \approx \varvec{f}_{{k^{\prime}}} \left( {\bar{\varvec{x}}_{0,m} } \right) + \varvec{F}_{{k^{\prime}}} \left( {\varvec{x}_{0} - \bar{\varvec{x}}_{0,m} } \right) \approx \bar{\varvec{x}}_{{k^{\prime},m}} + {\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{X} }}_{{k^{\prime},m}} \varvec{w} ,$$

(14)

where $\varvec{F}_{k}$ is the Jacobian of $\varvec{f}_{k}$ at $\bar{\varvec{x}}_{0,m}$ and ${\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{X} }}_{{k^{\prime},m}}$ is a matrix obtained by replacing 0 by $k'$ in Eq. (4) as follows:

$${\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{X} }}_{{k^{\prime},m}} = \frac{1}{{\sqrt {N - 1} }}\left( {\varvec{x}_{{k^{\prime},m}}^{\left( 1 \right)} - \bar{\varvec{x}}_{{k^{\prime},m}} \cdots \varvec{x}_{{k^{\prime},m}}^{\left( N \right)} - \bar{\varvec{x}}_{{k^{\prime},m}} } \right).$$

(15)

The state at $t_{{k^{\prime}}}$ can be estimated from the following equation:

$$\bar{\varvec{x}}_{{k^{\prime},m + 1}} = \bar{\varvec{x}}_{{k^{\prime},m}} + {\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{X} }}_{{k^{\prime},m}} \hat{\varvec{w}}_{m} ,$$

(16)

without any modification to $\hat{\varvec{w}}_{m}$ in Eq. (8), which is the same as used in Eq. (9). To obtain $\bar{\varvec{x}}_{{k^{\prime},m + 1}}$ by Eq. (16) means that we minimize the cost function Eq. (2) (or (12)) using the first-order Taylor expansion of $\varvec{g}_{{{{\Delta }}k}} \left( {\varvec{x}_{{k^{\prime}}} } \right)$ at $\bar{\varvec{x}}_{{k^{\prime},m}}$,

$$\varvec{g}_{k} \left( {\varvec{x}_{0} } \right) = \varvec{g}_{{{{\Delta }}k}} \left( {\varvec{x}_{{k^{\prime}}} } \right) \approx \varvec{g}_{{{{\Delta }}k}} \left( {\bar{\varvec{x}}_{{k^{\prime},m}} } \right) + \varvec{G}_{{{{\Delta }}k}} \left( {\varvec{x}_{k'} - \bar{\varvec{x}}_{k',m} } \right),$$

(17)

instead of Eq. (3), where ${{\Delta }}k = k - k'$, and $\varvec{g}_{{{{\Delta }}k}}$ and $\varvec{G}_{{{{\Delta }}k}}$ are associated with the inverse function of $\varvec{f}_{{ - {{\Delta }}k}}$ when ${{\Delta }}k < 0$. This enables us to choose arbitrary time, $t_{{k^{\prime}}}$, at which we restart MHD dynamo simulation after the data assimilation, for the purpose of better accuracy of the linear approximation at times close to $t_{{k^{\prime}}}$. For the future forecast, single MHD or kinematic dynamo (KD; meaning steady flow) simulation can also estimate future magnetic fields, running them from the initial condition optimized at arbitrary time $t_{{k^{\prime}}}$ (Eq. 16) by data assimilation. We discuss differences in forecasts by MHD and kinematic dynamo simulation from different $t_{k}^{'}$ in “Numerical experiments” section.

Geodynamo simulation: parameters and scaling of time

We adopt the geodynamo simulation code established by Takahashi (2012, 2014) for our data assimilation and forecast processes. In this code, the velocity and magnetic fields are decomposed into the poloidal and toroidal parts and spatially expanded in terms of spherical harmonics in the tangential directions, while combined compact finite differencing and the Crank–Nicolson scheme are adopted for discretization in the radial and time coordinates, respectively. The temperature and composition are treated separately (Takahashi 2014). We select the following dimensionless parameters for the geodynamo simulation: the Ekman number is $E = 3 \times 10^{ - 5}$, the magnetic Prandtl number is $Pm = 2.0$, the Prandtl number is 0.1, the compositional Prandtl (or Schmidt) number is 1.0, the modified thermal Rayleigh number is 500, and the modified compositional Rayleigh number is 2000. The magnetic Reynolds number ($R_{m}$) of this dynamo is 181 with the standard deviation of 15, which is evaluated from the initial ensemble members for our data assimilation (see “Numerical experiments” section for details of the ensemble).

Since all the parameters and variables are dimensionless in the numerical geodynamo model, the simulation outputs should be scaled for comparison with the actual data of the geomagnetic main field and the core surface flow. A time scale of the geomagnetic secular variation, $\tau_{\text{SV}}$, can be characterized by simple dependence of spherical harmonic degree, $l$ (Lhuillier et al. 2011a). Christensen and Tilgner (2004) obtained a value of $\tau_{\text{SV}} = 535 {\text{years}}$ by fitting $\tau_{\text{SV}} /l$ ($l \ge 2$) to the time scale of secular variation for each spherical harmonic degree, $\tau_{l}$ (see Christensen and Tilgner 2004, for the definition). In the same manner, we estimate the secular variation time scale of our numerical geodynamo model to be $\tau_{\text{SV}}^{*} = 0.0411$, scaled by the viscous dissipation time; we first estimate the mean $\tau_{l}$ of our numerical geodynamo from a long-time dynamo run (about one magnetic diffusion time, i.e., 26,000 years in actual time) and then evaluate $\tau_{\text{SV}}^{*}$ by fitting $\tau_{\text{SV}}^{*} /l$ to the estimated $\tau_{l}$ for $l \ge 2$. Figure 2 shows the estimated $\tau_{l}$ and the fitting result. Hence, we convert the dimensionless time of the numerical geodynamo into the actual time by using $\tau_{\nu } \equiv \tau_{\text{SV}} /\tau_{\text{SV}}^{*} = 13,049 \;{\text{years }}$ throughout this study.

Fig. 2

Secular variation time scale of our dynamo model. The time scale of secular variation for each spherical harmonic degree $\tau_{l}$ is fitted by $\tau_{SV}^{*} /l$ for $l \ge 2$. The resulting $\tau_{SV}^{*}$ is 0.0411, which was utilized to scale the dimensionless numerical dynamo time

Nonlinearity of the numerical geodynamo: error growth e-folding time

Since the 4DEnVAR approach relies heavily on the linear assumption Eq. (3) and resulting Eq. (6), we should be careful about its validity in the problem in concern, i.e., the data assimilation and forecast of the geomagnetic field for IGRF-13SV. We here investigate the timescale of nonlinearity of our numerical geodynamo by introducing the error growth rate (e.g., Hulot et al. 2010), which measures the rate at which small error in a dynamo state vector grows with time. It is known that the error growth rate is stable once a dynamo model and parameters are provided, independent of the magnitude of inserted error and types of variables into which error is contaminated (Lhuillier et al. 2011b). Its inverse, i.e., the e-folding time of error growth $\tau_{e}$, therefore, can be interpreted as a limit of the period over which data assimilation is feasible with the nonlinear dynamo model; when the length of assimilation window is comparable with $\tau_{e}$, small error in the initial state is critical to the state at the end of the window, and one cannot expect to obtain reasonable initial condition. The linear assumption for 4DEnVar (Eq. 3) clearly requires that the assimilation window should be shorter than $\tau_{e}$ of our numerical geodynamo model.

We estimate the e-folding time $\tau_{e}$ of our numerical geodynamo by following the method in Lhuillier et al. (2011b). We first choose ten different dynamo state vectors from a dynamo run for about one magnetic diffusion time. We then add a small perturbation to the poloidal components of the magnetic field over the first eight harmonic degree by the expression:

$$\tilde{S}_{l}^{m} \left( {t_{p} ,r} \right) = \left( {1 + \alpha_{l}^{m} \epsilon } \right)S_{l}^{m} \left( {t_{p} ,r} \right),$$

(18)

where $\tilde{S}_{l}^{m}$ and $S_{l}^{m}$ are the poloidal scalar functions of the magnetic field with degree $l$ and order $m$ of spherical harmonics for the perturbed and original initial conditions at a chosen time $t_{p}$, respectively. $\alpha_{l}^{m}$ is a coefficient obeying the standard normal law and ϵ is the relative magnitude of perturbation. We choose ϵ = 10⁻¹⁰ in this study. MHD dynamo simulations are performed from both original and perturbed initial conditions and the discrepancy between them are measured by the magnetic error at the CMB,

$${{\Delta }}y_{l} \left( t \right) = \sqrt {\frac{1}{2l + 1}\mathop \sum \limits_{m} \frac{{\left( {\tilde{S}_{l}^{m} \left( {t,r_{\text{CMB}} } \right) - S_{l}^{m} \left( {t,r_{\text{CMB}} } \right)} \right)^{2} }}{\langle {S_{l}^{m} \left( {t,r_{\text{CMB}} } \right)^{2} \rangle }}} ,$$

(19)

where $t$ is time scaled by the magnetic diffusion time $\tau_{\eta }$, $r_{\text{CMB}}$ is the radial coordinate of the CMB, and $\langle \cdot \rangle$ denotes the time average. For each pair of dynamo runs, the error growth rate $\lambda$ and its inverse, the e-folding time $\tau_{e} = \lambda^{ - 1}$, are estimated by fitting the model of $\lambda t + a$, where $a$ is the offset, jointly to $\log {{\Delta }}y_{l} \left( t \right)$ ($1 \le l \le 8$). The time window for the fitting is manually determined, by simultaneously considering the first eight $\Delta y_{l} \left( t \right)$. After the same procedure for ten chosen times of $t_{p}$, we average ten $\lambda$ s and $\tau_{e}$ s for our final estimates.

Figure 3 shows results of estimation of $\lambda$ and $\tau_{e}$ for our geodynamo model. As shown in Fig. 3a, the error growth rate (slope of the blue line) was obtained by the linear regression over the first eight spherical harmonic degrees (the gray lines). As in Fig. 3b, we finally evaluate $\lambda = 187 \pm 36$ [$1/\tau_{\eta }$] and $\tau_{e} = \left( {5.37 \pm 1.02} \right) \times 10^{ - 3} \left[ {\tau_{\eta } } \right]$, where the uncertainties are given by two standard deviations. Using the scaling law, $\tau_{\eta } = P_{m} \tau_{\nu } = 2 \cdot 13,049 {\text{years}}$, $\tau_{e}$ of our numerical dynamo is found to be $\tau_{e} = 140 \pm 27$ years. Then we should choose the assimilation window shorter than ~ 110 years to avoid the effect of nonlinearity of our geodynamo model (Takahashi 2012, 2014). We later select a maximum assimilation window of 10 years and a prediction period of 5 years, which seem short enough to circumvent the nonlinearity effect and to hold Eq. (3).

Fig. 3

Error growth rates of the numerical geodynamo model. The top panel a shows an example of estimation of error growth rate using least-squares linear regression, where the gray lines denote magnetic errors, $\Delta y_{l}$ (Eq. 19), for the spherical harmonic degree from 1 to 8, and the blue line is the result of the linear regression. The bottom panel b shows superimposed ten results of estimation of error growth rates, where the average error growth rate turns out to be 187 with 95 percent uncertainty (2σ) of 36. See text for details of the estimation

Hulot et al. (2010) claimed that $\tau_{e}$ of the Earth’s dynamo system appears to be about 30 years from the linear relationship between $\tau_{e} /\tau_{\text{SV}}$ and $R_{m}$, provided that $R_{m} > 100$, $E < 10^{ - 9}$, and $\tau_{\text{SV}} = 535 {\text{years}}$ (Christensen and Tilgner 2004) for the Earth. Although our $\tau_{e} (\sim 140$ years) is much longer than $\tau_{e} (\sim$ 30 years) by Hulot et al. (2010), it is found that $\tau_{e}$ highly depends on numerical dynamo models even under the similar dynamo parameter settings (e.g., see Fig. 3 in Hulot et al. (2010) for the comparison between $\tau_{e}$ by Hulot et al. (2010) and that by Olson et al. (2009) under $E = 3 \times 10^{ - 4}$). We, therefore, think our long $\tau_{e}$ is a characteristic of our geodynamo model. We here conclude that our assimilation window of maximum 10 years is short enough to adopt linear approximation of Eq. (3) and resulting Eq. (6) from the perspective of $\tau_{e}$ of our numerical geodynamo.

Data 1: poloidal scalar potential at the CMB obtained from the MCM model

We use a version of the MCM model (Ropp et al. 2020) which spans from 2001.5 to 2019.25, referred to as the MCM3 model in this paper hereafter, to construct data vectors for our data assimilation. The MCM3 model has been built using hourly mean observatory data together with CHAMP and Swarm-A satellite magnetic data. These data have been selected depending on their local time and for magnetically quiet periods (c.f., Lesur et al. 2008). The model is made of a series of snapshot models, 3 months apart, that have been built through a Kalman filter approach combined with a correlation-based modeling technique for the analysis step (Holschneider et al. 2016). Each snapshot model is parameterized in terms of spherical harmonics and includes static core field and secular variation contributions. Lithospheric, external and induced fields are also co-estimated as well as crustal offsets at each observatory. Details of the modeling technique are described in Ropp et al. (2020). The Gauss coefficients of the core field and its secular variation can be downward continued to the CMB for use, e.g., for information on the flow in the liquid outer core.

Assuming the mantle an electrical insulator, we convert the Gauss coefficients, $g_{l}^{m} \left( t \right)$ and $h_{l}^{m} \left( t \right)$, given by the MCM3 model to the poloidal scalar potential, $S_{l}^{\text{mc}} \left( t \right)$ and $S_{l}^{\text{ms}} \left( t \right)$, at the CMB, respectively, as

$$\begin{aligned}S_{l}^{\text{mc}} \left( t \right)& = \frac{1}{l}\frac{1}{{r_{\text{oc}}^{2} }}\left( {\frac{{r_{e} }}{{r_{\text{oc}} }}} \right)^{l + 2} g_{l}^{m} \left( t \right), \\ S_{l}^{\text{ms}} \left( t \right)& = \frac{1}{l}\frac{1}{{r_{\text{oc}}^{2} }}\left( {\frac{{r_{e} }}{{r_{\text{oc}} }}} \right)^{l + 2} h_{l}^{m} \left( t \right), \end{aligned}$$

(20)

where $t$ is the dimensional time, $r_{e} = 6371.2 {\text{km}}$ and $r_{oc} = 3485 {\text{km}}$ are the radii of the Earth and outer core, respectively, and $S_{l}^{\text{mc}} \left( t \right)$ and $S_{l}^{\text{ms}} \left( t \right)$ are poloidal scalar functions for cosine and sine terms, respectively. One of data to be assimilated is $S_{l}^{m} \left( t \right)$ which denotes $S_{l}^{\text{mc}} \left( t \right)$ and/or $S_{l}^{\text{ms}} \left( t \right)$, and the truncation of spherical harmonics is $L_{\text{MCM}} = 14$. In the assimilation procedure, however, we use only coefficients with degree up to 13 assigning very small weight to coefficients for $l = 14$. The magnetic data are constructed in the form of

$$d_{{S_{l}^{m} }} \left( t \right) = \frac{{S_{l}^{m} \left( t \right)}}{{{S_{1}^{0}}_{\text{ref}} }},$$

(21)

where ${S_{1}^{0}}_{\text{ref}}$ is a certain reference value for $S_{1}^{0}$. For the observation, we use $S_{1}^{0}$ at the start time of assimilation window as ${S_{1}^{0}}_{\text{ref}}$ throughout this study. For the numerical geodynamo, we substitute a typical $S_{1}^{0}$ value of 0.3 into ${S_{1}^{0}}_{\text{ref}}$ in Eq. (21) to generate vectors comparable with the magnetic data vectors. We prepare the data vector by a 0.25-year interval. From the outputs of the numerical geodynamo, the poloidal scalar potentials are normalized according to Eq. (21) after the conversion of the dimensionless time to the actual time.

Data 2: core surface flow

We also include the core fluid velocity field slightly below the CMB in our data assimilation. In the present numerical simulation of geodynamo, the no-slip condition for the velocity field is imposed at the CMB (Takahashi 2012). It, therefore, is pertinent to adopt core surface flows estimated with the effect of viscous boundary layer at the CMB (Fig. 4a). Hence, we employ a method of Matsushima (2015). Before recalling the method, we define the velocity field, $\varvec{V}$, and the magnetic field, $\varvec{B}$, in the outer core. $\varvec{V}$ can be expressed in terms of the poloidal and toroidal scalar functions as

$$\varvec{V}\left( {r,\theta ,\phi ,t} \right) = \nabla \times \nabla \times \left( {\hat{\varvec{r}}U} \right) + \nabla \times \left( {\hat{\varvec{r}}W} \right),$$

(22)

$$U\left( {r, \theta ,\phi ,t} \right) = \mathop \sum \limits_{l = 1}^{L} \mathop \sum \limits_{m = 0}^{l} \left\{ {U_{l}^{\text{mc}} \left( {r,t} \right)\cos m\phi + U_{l}^{\text{ms}} \left( {r,t} \right)\sin m\phi } \right\}P_{l}^{m} (\cos \theta ),$$

(23)

$$W\left( {r, \theta ,\phi ,t} \right) = \mathop \sum \limits_{l = 1}^{L} \mathop \sum \limits_{m = 0}^{l} \left\{ {W_{l}^{mc} \left( {r,t} \right)\cos m\phi + W_{l}^{ms} \left( {r,t} \right)\sin m\phi } \right\}P_{l}^{m} (\cos \theta ),$$

(24)

where $\left( {r,\theta ,\phi } \right)$ is the spherical coordinates, $\hat{\varvec{r}}$ is a radial unit vector, and $P_{l}^{m}$ is Schmidt quasi-normalized associated Legendre function with degree $l$ and order $m$. Near the core surface, the radial component of the velocity field, $V_{r}$, is likely to be much smaller than the horizontal component, $\varvec{V}_{h}$. Therefore, in core surface flow modeling, only the horizontal component, $\varvec{V}_{h} = \hat{\varvec{\theta }}V_{\theta } + \hat{\phi }V_{\phi }$, is computed:

$$V_{\theta } \left( {r,\theta ,\phi ,t} \right) = \mathop \sum \limits_{l = 1}^{L} \mathop \sum \limits_{m = 0}^{l} \left[ {\left\{ {\frac{1}{r}\frac{{\partial U_{l}^{mc} }}{\partial r}\cos m\phi + \frac{1}{r}\frac{{\partial U_{l}^{ms} }}{\partial r}\sin m\phi } \right\}\frac{{dP_{l}^{m} }}{d\theta } + m\left\{ { - \frac{{W_{l}^{mc} }}{r}\sin m\phi + \frac{{W_{l}^{ms} }}{r}\cos m\phi } \right\}\frac{{P_{l}^{m} }}{\sin \theta }} \right],$$

(25)

$$V_{\phi } \left( {r,\theta ,\phi ,t} \right) = \mathop \sum \limits_{l = 1}^{L} \mathop \sum \limits_{m = 0}^{l} \left[ {m\left\{ { - \frac{1}{r}\frac{{\partial U_{l}^{mc} }}{\partial r}\sin m\phi + \frac{1}{r}\frac{{\partial U_{l}^{ms} }}{\partial r}\cos m\phi } \right\}\frac{{P_{l}^{m} }}{\sin \theta } - \left\{ {\frac{{W_{l}^{mc} }}{r}\cos m\phi + \frac{{W_{l}^{ms} }}{r}\sin m\phi } \right\}\frac{{dP_{l}^{m} }}{d\theta }} \right],$$

(26)

where $\hat{\varvec{\theta }}$ and $\hat{\phi }$ are unit vectors in the $\theta$- and $\phi$-directions, respectively. Hence, Matsushima (2015) computed $\bar{U}_{l}^{m} = \partial U_{l}^{m} /r\partial r$ and $\bar{W}_{l}^{m} = W_{l}^{m} /r$. A boundary condition, $U_{l}^{m} \left( {r_{o} } \right) = 0$, leads to

$$\frac{1}{{r_{o} - \xi_{2}^{*} }}\frac{{\partial U_{l}^{m} }}{\partial r} \approx \frac{1}{{r_{o} - \xi_{2}^{*} }}\frac{{U_{l}^{m} \left( {r_{o} } \right) - U_{l}^{m} \left( {r_{o} - \xi_{2}^{*} } \right)}}{{\xi_{2}^{*} }} = - \frac{1}{{r_{o} - \xi_{2}^{*} }}\frac{{U_{l}^{m} \left( {r_{o} - \xi_{2}^{*} } \right)}}{{\xi_{2}^{*} }},$$

(27)

where $r_{o}$ is the outer radius of the rotating spherical shell in numerical MHD dynamo model, and $\xi_{2}^{*}$ is a depth from $r = r_{o}$. Therefore, taking into account the length scale $r_{oc} - r_{ic}$ and the velocity scale $\left( {r_{oc} - r_{ic} } \right)/\tau_{\nu }$ (see Appendix C), we have

$$U_{l}^{m} \left( {r_{o} - \xi_{2}^{*} } \right) = - \frac{{r_{o} - \xi_{2}^{*} }}{{r_{oc} - r_{ic} }}\frac{{\xi_{2}^{*} }}{{r_{oc} - r_{ic} }}\frac{{\bar{U}_{l}^{m} \left( {r_{oc} - \xi_{2} } \right)}}{{\left( {r_{oc} - r_{ic} } \right)/\tau_{\nu } }},$$

(28)

and

$$W_{l}^{m} \left( {r_{o} - \xi_{2}^{*} } \right) = \frac{{r_{o} - \xi_{2}^{*} }}{{r_{oc} - r_{ic} }}\frac{{\bar{W}_{l}^{m} \left( {r_{oc} - \xi_{2} } \right)}}{{\left( {r_{oc} - r_{ic} } \right)/\tau_{\nu } }}.$$

(29)

Fig. 4

a The schematic view of the viscous boundary layer at the core–mantle boundary (after Matsushima 2015), b The RMS misfit for velocity at $r_{b2} = 3428.5$ km (Eq. 42), where the gray horizontal dashed line denotes the assimilation window from 2004.50 to 2014.25, and c comparison between the real core surface flows at $r_{b2}$ from the MCM3 model and the recovered velocity field for Case A1 (see Table 1 for details). Arrows denote the horizontal flow velocity field, while color contours show upwellings and downwellings given by horizontal divergence of the velocity field. The poloidal and toroidal scalar potentials of velocity are compared at $r_{b2}$ in our data assimilation

In the method of Matsushima (2015),

• (0) At the CMB $\left( {r = r_{oc} } \right)$, geomagnetic secular variations are caused only by magnetic diffusion due to the no-slip boundary condition;

  $$\frac{{\partial B_{r} }}{\partial t} = \frac{\eta }{r}\nabla^{2} \left( {rB_{r} } \right),$$

  (30)

  where $\eta$ is the magnetic diffusivity.

• (1) Inside the viscous boundary layer $\left( {r = r_{b1} = r_{oc} - \xi_{1} } \right)$, the viscous force plays an important role in force balance as

  $$\hat{\varvec{r}} \cdot \nabla \times \left( { - 2{\varvec{\Omega}} \times \varvec{V} + \rho^{ - 1} \varvec{J} \times \varvec{B} + \nu \nabla^{2} \varvec{V}} \right) = 0,$$

  (31)

  where ${\varvec{\Omega}}$ is the rotation rate of the mantle, $\rho$ is the density of the core fluid, $\varvec{J}$ is the electric current density, and $\nu$ is the kinematic viscosity. The magnetic diffusion, motional induction, and advection are assumed to contribute to temporal variations of the magnetic field;

  $$\frac{{\partial B_{r} }}{\partial t} = \frac{\eta }{r}\nabla^{2} \left( {rB_{r} } \right) - \left( {\varvec{V} \cdot \nabla } \right)B_{r} + \left( {\varvec{B} \cdot \nabla } \right)V_{r} ,$$

  (32)

  where core fluid is assumed to be incompressible, $\nabla \cdot \varvec{V} = 0$.

• (2) Below the boundary layer $\left( {r = r_{b2} = r_{oc} - \xi_{2} } \right)$, core flow is assumed to be tangentially magnetostrophic (Matsushima 2020) as

  $$\hat{\varvec{r}} \cdot \nabla \times \left( { - 2{\varvec{\Omega}} \times \varvec{V} + \rho^{ - 1} \varvec{J} \times \varvec{B}} \right) = 0.$$

  (33)

Temporal variation of the magnetic field is assumed to be caused by the motional induction and advection, and the magnetic diffusion is neglected, as in the frozen-flux approximation (e.g., Holme 2015) as

$$\frac{{\partial B_{r} }}{\partial t} = - \left( {\varvec{V} \cdot \nabla } \right)B_{r} + \left( {\varvec{B} \cdot \nabla } \right)V_{r} .$$

(34)

The radial component of the electric current density, $J_{r}$, is likely to be much smaller than the horizontal component, $\varvec{J}_{h}$, near the core surface due to the electrically insulating mantle. Following Shimizu (2006), we assume that the electric field does not contribute to the electric current density, and we have

$$\varvec{J}_{h} = \sigma \left( {\varvec{V} \times \varvec{B}} \right)_{h} \approx \sigma B_{r} \varvec{V}_{h} \times \hat{\varvec{r}},$$

(35)

where $\sigma$ is the core electrical conductivity. Hence, the tangentially magnetostrophic constraint is given as

$$\nabla_{h} \cdot \left( {2\varOmega \cos \theta \bar{V}_{h} + \rho^{ - 1} \sigma B_{r}^{2} \bar{V}_{h} \times \hat{r}} \right) = 0.$$

(36)

The ratio of the Lorentz to the Coriolis forces is the Elsasser number given by

$$\varLambda = \frac{{\sigma B_{r}^{2} }}{{2\rho \varOmega \left| {\cos \theta } \right|}}.$$

(37)

We can have a local value of $\varLambda \approx 0.62$, using actual physical parameters for the outer core, $\sigma = 1 \times 10^{6} {\text{S m}}^{ - 1}$, $\varOmega = 7.29 \times 10^{ - 5} {\text{rad s}}^{ - 1}$, $\rho = 1.1 \times 10^{4} {\text{kg m}}^{ - 3}$, and $B_{r} = 1 {\text{mT}}$. In the numerical MHD dynamo model, we obtain ${{\varLambda }}^{ *} = 1.778$ from the initial ensemble members for our data assimilation (see “Numerical experiments” section for details of the ensemble), which is approximately three times larger than $\varLambda$ for the actual outer core. Such an Elsasser number larger than 1 means relative importance of the Lorentz force in our numerical geodynamo. Hence, we consider that the tangentially magnetostrophic constraint for core surface flow is appropriate. The Elsasser number $\varLambda^{*} = 1.778$ leads to $B_{r} \sim 0.1\; {\text{mT}}$ in our numerical geodynamo model, where we adopt $\sigma = 1.277 \times 10^{5} {\text{S m}}^{ - 1}$ and ${{\varOmega }} = 4.05 \times 10^{ - 8} {\text{rad s}}^{ - 1}$ derived from $\tau_{\nu } = 13049$ years (see Appendix C). This shows that the magnetic field in our numerical geodynamo model is rather weak compared to that of the actual Earth, possibly due to the small $\varOmega$. This is the reason why we use a relative magnetic field $d_{{S_{l}^{m} }} \left( t \right)$ defined in (21). To overcome this problem, we should perform more realistic numerical simulation of geodynamo for a much smaller Ekman number corresponding to much larger $\varOmega$, although this remains a problem to be solved in the future.

We compare the poloidal and toroidal scalar potentials, $U_{l}^{m}$ and $W_{l}^{m}$, for the core flow field with those for the numerical flow field, up to $l = 14$ (the same as the truncation degree of the MCM3 model, and also practically used up to degree 13 in the assimilation by controlling the weight). The core flow data, $U_{l}^{m}$ and $W_{l}^{m}$, are constructed from the main field and SV at 0.25-year intervals in the MCM3 model. To make the dimensionless numerical velocity output comparable with the core flow data, we scale the simulated velocity using the relation of $\tau_{\text{SV}} /\tau_{\text{SV}}^{*} = 13,049 {\text{years}}$ and the kinematic viscosity, which is derived from Ekman number and magnetic Prandtl number adopted in the numerical dynamo. See “Appendix C” for more details of scaling of the dimensionless velocity field. The left column of Fig. 4c shows examples of core flows at $r_{b2} = 3428.5 {\text{km}}$, which is $\xi_{2} = 56.5 {\text{km}}$ below the Earth’s core radius of $r_{oc} = 3485.0 {\text{km}}$, corresponding to a numerical grid point in the radial direction, $r_{89}^{*}$, for the epochs of 2007.0, 2010.0 and 2013.0 calculated from the MCM3 model.

Implementation of assimilation

We implement the data assimilation for the following data vectors:

$$\varvec{y}_{k} = \left[ {\begin{array}{*{20}c} {\varvec{d}_{S} \left( {t_{k} } \right)} \\ {\varvec{d}_{U} \left( {t_{k} } \right)} \\ {\varvec{d}_{W} \left( {t_{k} } \right)} \\ \end{array} } \right],$$

(38)

where $\varvec{d}_{S}$ denotes the magnetic data vector composed of the scaled poloidal component specified in Eq. (21), while $\varvec{d}_{U}$ and $\varvec{d}_{W}$ represent the velocity data vectors. Subscripts U and $W$ denote the poloidal and toroidal scalars of the velocity field, respectively. In this study, $\varvec{d}_{U}$ and $\varvec{d}_{W}$ consist of $U_{l}^{m}$ and $W_{l}^{m}$ at $r_{b2} = 3428.5$ km, respectively. We consider the observation error covariance matrix, ${\mathbf{R}}_{k}$ in Eqs. (2) and (12), as having a simple time-independent form:

$${\mathbf{R}}_{k} = {\mathbf{R}} = \left[ {\begin{array}{*{20}c} {\alpha_{S}^{2} {\mathbf{R}}_{S} } & 0 & 0 \\ 0 & {\alpha_{UW}^{2} {\mathbf{R}}_{U} } & 0 \\ 0 & 0 & {\alpha_{UW}^{2} {\mathbf{R}}_{W} } \\ \end{array} } \right],$$

(39)

where ${\mathbf{R}}_{S}$, ${\mathbf{R}}_{U}$, and ${\mathbf{R}}_{W}$ are the diagonal covariance matrices with each degree dependence. Weights for data sets are controlled by the two scalar coefficients of $\alpha_{S}$ and $\alpha_{UW}$. We simply adopt a time-independent form of ${\mathbf{R}}_{k} = {\mathbf{R}}$, since the MCM3 model is a quality-controlled smooth model. We prepare ${\mathbf{R}}_{S}$ by the expression ${\left\{ {\varvec{R}_{S} } \right\}}_{l} = {{\text{C}}_{\text{S}}}_{{l}^{m}} \left[ {{\sigma_{G}^{2}}_{l = 1} \left( {l + 1} \right)^{ - 1} } \right]$, where $\left\{ {{\mathbf{R}}_{S} } \right\}_{l}$ is the diagonal element of ${\mathbf{R}}_{S}$ for degree $l$, $\left( {l + 1} \right)^{ - 1}$ is the degree dependence of the variance of Gauss coefficients based on the theory by Lowes (1975), ${\sigma_{G}^{2}}_{l = 1}$ is the averaged actual variance of Gauss coefficient at degree $l = 1$ from the MCM3 model, and ${{\text{C}}_{\text{S}}}_{l}^{m} \left[ \cdot \right]$ is a function for conversion of the covariance of Gauss coefficients to that of the poloidal scalar potentials at the CMB for degree $l$ and order $m$. Similarly, we prescribe ${\mathbf{R}}_{U}$ and ${\mathbf{R}}_{W}$ by the expressions $\left\{ {{\mathbf{R}}_{U} } \right\}_{l} = {C_{U}}_{l}^{m} \left[ {{\sigma_{U}^{2}}_{l = 1} \left( {2l + 1} \right)/\left( {l^{2} + l} \right)} \right]$ and $\left\{ {{\mathbf{R}}_{W} } \right\}_{l} = {C_{W}}_{l}^{m} \left[ {{\sigma_{W}^{2}}_{l = 1} \left( {2l + 1} \right)/\left( {l^{2} + l} \right)} \right]$, respectively, where $\left\{ {{\mathbf{R}}_{U} } \right\}_{l}$ and $\left\{ {{\mathbf{R}}_{W} } \right\}_{l}$ are expressions similar to $\left\{ {{\mathbf{R}}_{S} } \right\}_{l}$, $\left( {2l + 1} \right)/\left( {l^{2} + l} \right)$ is the degree dependence of the variance of Schmidt quasi-normalized poloidal and toroidal scalar potentials of the core surface flow based on discussion in Holme (2015). ${\sigma_{U}^{2}}_{l = 1}$ and ${\sigma_{W}^{2}}_{l = 1}$ are the averaged actual variances of the Schmidt quasi-normalized poloidal and toroidal scalar potentials of the core surface flow at degree $l = 1$, respectively. They are calculated from the observation error of the Gauss coefficients in the MCM3 model. ${{\text{C}}_{\text{U}}}_{l}^{m} \left[ \cdot \right]$ and ${{\text{C}}_{\text{W}}}_{l}^{m} \left[ \cdot \right]$ are functions converting the covariances of the poloidal and toroidal scalar potentials for the velocity field at the CMB to covariances of those dimensionless at the radial grid $r_{89}^{*}$ in our numerical dynamo model with the velocity scaling by $\left( {r_{\text{oc}} - r_{\text{ic}} } \right)/\tau_{\nu } = 0.1736 {\text{km year}}^{ - 1}$ (see Appendix C), respectively. ${{\text{C}}_{\text{S}}}_{l}^{m} \left[ \cdot \right]$, ${{\text{C}}_{U}}_{l}^{m} \left[ \cdot \right]$, and ${{\text{C}}_{W}}_{l}^{m} \left[ \cdot \right]$ also include the conversion from the Schmidt quasi-normalization to the normalization adopted in our MHD dynamo model (Takahashi 2012, 2014), since we prepare $\varvec{y}_{k}$ (Eq. 38) directly comparable to the outputs of our numerical geodynamo model.

Numerical experiments

We examine the data assimilation scheme using geomagnetic data from 2004.50 to 2014.25 from the MCM3 model, covering up to 2019.25, as observations, and evaluate the resulting forecasts against data from 2014.25 to 2019.50 using the latest version of the MCM model, MCM6, covering up to 2019.50. The difference between the MCM3 and MCM6 models is as small as ~ 5 nT at the Earth’s surface on average. This hindcast can be a rigorous test of our forecast scheme since a geomagnetic jerk occurred in 2014 (Torta et al. 2015). We set the ensemble size as 960 and prepare ensemble members from a single free dynamo run for a long time corresponding to 320,000 years, where the ensemble members are taken randomly to be longer than one turn over time of core convection apart from each other. We found that iterative renewals of the ensemble members after obtaining $\bar{\varvec{x}}_{0, m + 1}$ in Eq. (9) is remarkably effective, compared with the original 4DEnVar (Liu et al. 2008). This approach enables us to deal with weak nonlinearity of the physical process in concern by shrinking the matrix ${\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{X} }}_{0,m}$ through iterations, which meets the requirement (II) for the linear approximation we discussed after Eq. (6). We refer to Nakano (2020) for the efficiency of the iterative approach in 4DEnVar. In this section, we focus on the results after five iterations (‘five’ was found enough to minimize the cost function of Eq. (12) from our experiments). We sequentially reduce $\alpha_{S}$ and $\alpha_{UW}$ (Eq. 39) through iterations, and the values in the final iteration are listed in Table 1. By the numerical experiments, we investigate dependences of our data assimilation and forecast performance on several parameters: length of the assimilation window, data weights ($\alpha_{S}$, $\alpha_{UW}$), and types of prediction approaches for the following 5 years.

Table 1 Summary of the numerical experiments

In this section, we investigate three types of forecasts:

• (i) Ensemble-weighted sum, shorten to “Ens. wei. sum” hereafter, is a forecast based on Eq. (13). After the five iterations of our data assimilation procedure, the final MHD ensemble members $\varvec{g}_{k} \left( {\varvec{x}_{0,M}^{\left( n \right)} } \right)$ for $n = 1, \ldots ,N$ and $\varvec{g}_{k} \left( {\bar{\varvec{x}}_{0,M} } \right)$ for $k = 1, \ldots ,K$ are available with the optimized weight $\hat{\varvec{w}}_{M}$. We further run MHD dynamo simulations to obtain $\varvec{g}_{k} \left( {\varvec{x}_{0,M}^{\left( n \right)} } \right)$ and $\varvec{g}_{k} \left( {\bar{\varvec{x}}_{0,M} } \right)$ for $k > K$ and calculate the field in the forecast period by Eq. (13).

• (ii) The MHD forecast is the forecast by an MHD dynamo simulation starting from a state vector given by data assimilation at time $t_{{k^{\prime}}}$ (Eq. 16). The MHD forecast differs from the Ens. wei. sum in the sense that it involves only a single MHD dynamo simulation after the data assimilation procedure. Since $t_{{k^{\prime}}}$ is arbitrary within the assimilation window, we test $k^{\prime} = 0$ (at the beginning of the window) and $k^{\prime} = K$ (at the end of the window). If the adopted linear approximation (Eqs. 3 or 17) holds precisely, the MHD forecast is equivalent to the Ens. wei. sum by Eq. (13).

• (iii) The KD forecast is the forecast by a KD simulation starting from a state vector given by assimilation at time $t_{{k^{\prime}}}$ (Eq. 16), similar to the MHD forecast. The only difference is that the KD forecast fixes the flow field in the outer core through the simulation. As for the KD forecast, we test only the case with $k^{\prime} = K$.

We test the three types of forecasts to choose the best way to generate an SV candidate model for IGRF-13SV. For comparison of our results with IGRF-12, we assume the release time of our hindcast to be 2014.75, taking into consideration that one can use magnetic data until roughly 0.50 year before the release epoch for generation of IGRF candidate models.

To assess results of our data assimilation and forecasts, we use the square root of the data misfit ${\text{d}}P$ (e.g., Whaler and Beggan 2015) defined by

$${\text{d}}P\left( t \right) = \mathop \sum \limits_{l = 1}^{{L_{\text{MF}} }} \mathop \sum \limits_{m = 0}^{l} \left( {l + 1} \right)\left[ {\left( {{g_{l}^{m}}_{\text{model}} \left( t \right) - {g_{l}^{m}}_{\text{data}} \left( t \right)} \right)^{2} + \left( {{h_{l}^{m}}_{\text{model}} \left( t \right) - {h_{l}^{m}}_{\text{data}} \left( t \right)} \right)^{2} } \right],$$

(40)

where the subscripts “model” and “data” mean that the Gauss coefficients come from our data assimilation-forecast procedure and from the observation (the MCM6 model in this study), respectively. To obtain ${g_{l}^{m}}_{\text{model}}$ and ${h_{l}^{m}}_{\text{model}}$ in the unit of nT, we use ${S_{1}^{0}}_{\text{ref}}$ in Eq. (21). At the beginning of the data assimilation, ${S_{1}^{0}}_{\text{ref}}$ is set for both the observation and the numerical geodynamo, here named ${S_{1}^{0}}_{\text{ref\_data}}$ and ${S_{1}^{0}}_{\text{ref\_model}}$. To multiply $S_{l}^{m}$ from the numerical geodynamo by the ratio of ${S_{1}^{0}}_{\text{ref\_data}} /{S_{1}^{0}}_{\text{ref\_model}}$ generates dimensional numerical magnetic fields available in Eq. (40). To assess the performance of SV forecast, we further define

$${\text{d}}P_{wo} \left( t \right) = \mathop \sum \limits_{l = 1}^{{L_{MF} }} \mathop \sum \limits_{m = 0}^{l} \left( {l + 1} \right)\left[ {\left( {{g_{l}^{m}}_{\text{model}} \left( t \right) - {g_{l}^{m}}_{\text{data}} \left( t \right) - {{\Delta }}{g_{l}^{m}}_{release} } \right)^{2} + \left( {{h_{l}^{m}}_{\text{model}} \left( t \right) - {h_{l}^{m}}_{\text{data}} \left( t \right) - {{\Delta }}{h_{l}^{m}}_{\text{release}} } \right)^{2} } \right],$$

(41)

where $\Delta {g_{l}^{m}}_{\text{release}} = {g_{l}^{m}}_{\text{model}} \left( {t_{\text{release}} } \right) - {g_{l}^{m}}_{\text{data}} \left( {t_{\text{release}} } \right)$ and a similar equation for ${{\Delta }}{h_{l}^{m}}_{\text{release}}$, and $t_{\text{release}}$ denotes the release time of the forecast. ${\text{d}}P_{wo} \left( t \right)$ measures ${\text{d}}P$ without the main field offset at the release time and purely evaluate the accuracy of SV from the release time. Table 1 summarizes all the assimilation and forecast settings with the results of $\sqrt {{\text{d}}P}$ and $\sqrt {{\text{d}}P_{wo} }$ 4.5 years after the release time.

Bias and trend terms

We first explain the effect of the bias and trend terms in Eqs. (10)–(13) on our data assimilation. We introduced those trend and bias terms, only in the final iteration and only for the magnetic field variations to reduce the data misfits. We found that the bias and trend terms represent model errors of the dynamo simulation well. Figure 5 shows three examples of the MHD forecasts with the initial condition at $k' = 0$ (Eq. 16): Case O1 for an MHD dynamo forecast without bias and trend terms, Case O2 for that containing the effect of the bias term, $- \varvec{b}$, and Case A2 for that containing the effect of the combination of the bias and trend terms, $- \varvec{b} - \varvec{a}k$, when minimizing the cost function $V\left( {\varvec{x}_{0} } \right)$ in the final iteration. One can see that the data misfits became smaller successively by introducing the bias and trend terms.

Fig. 5

Effects of the bias and trend terms on $\sqrt {{\text{d}}P}$, defined by Eq. (40). The black dotted and dashed lines and the cyan solid line indicate the MHD forecasts with $k^{\prime} = 0$ (Eq. 16), i.e., the MHD simulations given the initial condition at 2004.25 obtained by the data assimilation. The dotted black line is with neither bias nor trend, while the dashed line includes only bias. The cyan solid line takes both bias and trend into consideration in minimizing Eq. (10) at the final iteration. The blue dashed line shows the ensemble-weighted sum with bias and trend terms, corresponding to Eq. (13). See Table 1 for details of each calculation

We here briefly discuss the accuracy of the linear approximation Eq. 3 from the results in Fig. 5. We found from all of our experiments that “Ens. wei. sum” (Eq. 13) results in symmetric $\sqrt {{\text{d}}P}$ within the assimilation window with respect to the central time of the assimilation window (see Case A1 in Fig. 5). This is likely because we adopt the time-independent data covariance ${\mathbf{R}}_{k}$ (Eq. 39) through the window. On the other hand, the MHD forecasts with $k^{\prime} = 0$ (Eq. 16) often show asymmetric $\sqrt {{\text{d}}P}$ lines in the assimilation window (see Cases O1 and O2 in Fig. 5). The departures of the MHD forecast lines from the symmetry, especially at the end of the assimilation window, imply break-down of the linear approximation. This break-down probably arises from large deviations, $\varvec{x}_{0,m}^{\left( n \right)} - \bar{\varvec{x}}_{0,m}$ $\left( {n = 1, \ldots ,N} \right)$, which breaks the requirement (II) specified after Eq. (6), because the requirement (I) is satisfied by the 10-year assimilation window, much shorter than the error growth e-folding time $\tau_{e}$ of our numerical dynamo (~ 130 years). In Fig. 5, however, the MHD forecast line approaches symmetry by introducing the bias and trend terms. The MHD forecast with bias and trend terms (Case A2) fits the Ens. wei. sum (Case A1) from 2004.50 to 2012.00 very well, which indicates that the linear approximation Eq. 3 holds over 75% of the assimilation window. This implies that the introduction of the bias and trend terms is effective not only for reducing the data misfit $\sqrt {{\text{d}}P}$ but also for shrinking ${\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{X} }}_{0,m}$ (Eq. 4) and keeping the linear approximation accurate. From the small departure of Case A2 from Case A1 in Fig. 5, we conclude that both requirements for the linear approximation (I) and (II) are satisfied by the 10-year window and by the introduction of the bias and trend terms in our iterative 4DEnVar scheme, and that the linear approximation Eq. 3 is acceptable in our assimilation scheme. Hereafter, we only discuss results including both bias and trend terms in the final iteration.

Recovery of core surface flow by data assimilation

We next check how our data assimilation recovers the core surface flow. To evaluate this, we use a data misfit for the velocity field based on Holme (2015) by the expression,

$$\begin{aligned} {\text{d}}P_{UW} \left( t \right) & = \int\limits_{{r = r_{b2} }} {\left| {{\mathbf{V}}_{H} } \right|^{2} {\text{d}}S = \mathop \sum \limits_{l = 1}^{{L_{MF} }} {\text{d}}P_{UW, l} \left( t \right),} \\ {\text{d}}P_{UW, l} \left( t \right) & = \frac{{l\left( {l + 1} \right)}}{2l + 1}\mathop \sum \limits_{m = 0}^{l} \left[ {\left( {{U_{l}^{m}}_{\text{model}} \left( t \right) - {U_{l}^{m}}_{\text{data}} \left( t \right)} \right)^{2} + \left( {{W_{l}^{m}}_{\text{model}} \left( t \right) - {W_{l}^{m}}_{\text{data}} \left( t \right)} \right)^{2} } \right], \\ \end{aligned}$$

(42)

where the $U_{l}^{m}$ and $W_{l}^{m}$ are the Schmidt quasi-normalized poloidal and toroidal functions for the velocity field defined by Eq. (22). The subscripts “model” and “data” mean functions from the results of data assimilation and the observation, respectively. Note that functions with the subscript “model” are generated from outputs of the numerical geodynamo at the grid $r_{89}$, while those with “data” are prepared by the estimation of core surface flow at $r_{b2} = 3428.5$ km (see “Data 2: core surface flow” section).

Figure 4b shows the data misfit for the velocity field at $r = r_{b2}$ from the assimilation result of Case A. The top panel shows the total misfit, $\sqrt {{\text{d}}P_{UW} }$, while the each line in the bottom panel indicates the data misfit for each degree, $\sqrt {{\text{d}}P_{UW,l} } .$ The total misfit of the core surface flow, $\sqrt {{\text{d}}P_{UW} }$, is relatively large compared to the misfit of the magnetic field, $\sqrt {{\text{d}}P}$ (Eq. 40). This is caused by the two facts that both the bias and trend terms are not introduced in the final iteration for the velocity data, and that the factor for the velocity covariance matrices, $\alpha_{UW}$, is larger than that for the magnetic data, $\alpha_{S}$, in construction of ${\mathbf{R}}$ in Eq. 39 (see Table 1). Because our primary purpose is to predict the magnetic field over 5 years after the assimilation window, the role of the velocity data is complement to the magnetic data in our data assimilation.

Figure 4c shows the global maps of the velocity field for the observation (the MCM3 model) and the optimized one by the data assimilation (Case A1) within the assimilation window. The spatial patterns of the horizontal divergence and the arrows for horizontal velocity are consistent to some degree. However, the intensities of the horizontal velocity at $r = r_{b2}$ and their horizontal divergence for Case A1 are remarkably smaller than those for the MCM3 model. This may be associated with the large Elsasser number of our numerical geodynamo (${{\varLambda }}^{ *} \approx 1.778$). It seems difficult to optimize intensities of the magnetic and velocity field in the outer core simultaneously with our present dynamo parameters far from those of the actual Earth. Alternatively, this may be ascribed to the effect of magnetic diffusion. The horizontal velocity at $r = r_{b2}$ is derived from the MCM3 model, on the assumption that the contribution of magnetic diffusion to time variations in the magnetic field can be neglected. In the numerical geodynamo model, however, the induction equation containing the magnetic diffusion term as well as the motional induction and advection terms is solved. Therefore, the velocity field at $r = r_{b2}$ for Case A is probably smaller than that for the MCM3 model by the contribution of the magnetic diffusion. The discrepancy of the intensity of the velocity between the model and data remains the problem that should be solved in the future.

Length of assimilation window

We tried several lengths of the assimilation window with the maximum length of 10 years, and investigated dependence of the MHD forecasts on $t_{k}^{'}$, when the initial condition for the MHD dynamo simulation is given by the data assimilation (Eq. 16). Figure 6a and 6b shows comparison of the assimilation and forecast results between 10- and 5-year assimilation windows. As for the 10-year assimilation window (Fig. 6a), when the initial condition is given at the beginning of the assimilation window, i.e., $k^{\prime} = 0$ (at 2004.25), the MHD forecast (Case A2; the cyan line) fits the Ens. wei. sum via Eq. 13 (Case A1; the blue dashed line) only up to 2012.0 and slightly deviates from it during the prediction period after the assumed release epoch, 2014.75 (the red downward arrow). When the initial condition is given at the end of the assimilation window, i.e., $k' = K$ (at 2014.25), the MHD forecast (Case A3; the green line) fits the Ens. wei. sum well even after the release epoch of 2014.75. As for the 5-year window, both the MHD forecasts starting with initial conditions at $k' = 0$ and $k^{\prime} = K$ generate almost the same results as the Ens. wei. sum even from 2014.50 to 2019.50.

Fig. 6

Time evolutions of $\sqrt {{\text{d}}P}$ (Eq. 40) for the numerical experiments (see Table 1 for the summary). Left and right columns show results of 10-year and 5-year assimilation windows, respectively. All ordinates are $\sqrt {{\text{d}}P}$, i.e., the metric for deviation from the MCM6 model. The 10-year window spans from 2004.50 to 2014.25, while the 5-year one from 2009.50 to 2014.25. $\alpha '\left( { = 14} \right)$ in the left and $\alpha ''\left( { = 7} \right)$ in the right indicate $\alpha_{S}$ (Eq. 39) adopted in the final 5th iteration. a, b The weighted sum of ensemble members by Eq. (13) (the Ens. wei. sum) and MHD dynamo simulation starting from optimized initial conditions at the beginning ($k^{\prime} = 0$) or the end ($k^{\prime} = 0$) of the assimilation window (see Eqs. 14–16) (the MHD forecasts). c, d The dependence on the weight coefficient, $\alpha_{S}$ (Eq. 39), where the red dashed lines show the results with 0.1 $\alpha '$ and 0.1 $\alpha ''$. e, f The ensemble-weighted sum, MHD dynamo, and KD simulations starting at the end of assimilation window (the MHD and KD forecasts with $k^{\prime} = K$) for 0.1 $\alpha '$ and 0.1 $\alpha ''$. In all the panels, the black solid lines indicate $\sqrt {{\text{d}}P}$ calculated between the MCM6 model and IGRF-12. The release time for our forecast, assumed 0.5 year after the end of the assimilation window, is 2014.75, while 2015.00 for IGRF-12

These results imply that the MHD forecast with the initial condition at the end of the assimilation window ($k^{\prime} = K$ in Eq. 16) fit the corresponding Ens. wei. sum (Eq. 13) well over the forecast period regardless of the length of assimilation window. This comes from the fact that, when $k^{\prime} = K$, $\varvec{g}_{k} \left( {\varvec{x}_{0} } \right)$ is linearly approximated with respect to the deviation from $\bar{\varvec{x}}_{K,m}$ (set $k^{\prime} = K$ in Eq. 17); the accuracy of the linear approximation after the assimilation window is independent of the length of the assimilation window. When $k^{\prime} = 0$ in the MHD forecast, on the other hand, the shorter assimilation window results in better accuracy of the linear approximation over the forecast period. We can conclude that the adoption of $k^{\prime} = K$ in the MHD forecasts enables us to achieve high accuracy of the linear approximation (here Eq. 17) over the forecast period even when the assimilation window is relatively long.

Dependence on weight, $\varvec{\alpha}_{\varvec{S}}$

In our numerical experiments, we varied only $\alpha_{S}$ by fixing $\alpha_{UW}$ as listed in Table 1. Figure 6c and 6d shows the dependence of $\sqrt {{\text{d}}P}$ on $\alpha_{S}$ in Eq. (39). The two panels show the results for a given $\alpha_{S}$ (the blue dashed lines) and for the corresponding $0.1\alpha_{S}$ (the red dashed lines). We found that longer assimilation window is necessary to effectively reduce the misfit in the forecast period using smaller $\alpha_{S}$. On the other hand, when the initial conditions were obtained with too small $\alpha_{S}$, the MHD forecast did not produce reasonable outputs; the forecast was degraded when $\alpha_{S}$ was smaller than shown in Fig. 6 because the problem got unstable.

Note here that the red dashed lines in Fig. 6 (Cases A4 and B4) correspond to results with the almost smallest $\alpha_{S}$ that allows reasonable MHD forecasts for each assimilation window. We decided to adopt 10-year assimilation window for estimation of SV candidate model, because (1) the linear approximation (Eq. 3 and resulting Eq. 6) is valid over most of the assimilation window shown in Figs. 5 and 6a, and (2) smaller $\sqrt {{\text{d}}P}$ can be achieved by decreasing $\alpha_{S}$ compared with shorter windows.

MHD and KD forecasts

We investigated the three types of forecasts for the forecast period from 2014.50 to 2019.25: the Ens. wei. sum by Eq. (13), the MHD forecast, and the KD forecast. As already shown in Figs. 5 and 6, the Ens. wei. sum is a good candidate SV, which is theoretically identical to the MHD forecast with the optimized initial condition under the linear approximation assumption (Eqs. 3 or 17). Figure 6e and 6f shows comparison of the three types of forecasts for the 10- and 5-year windows, respectively. The MHD and KD forecasts were conducted with the same initial condition given at the end of the assimilation window, 2014.25 ($k^{\prime} = K$ in Eq. 16). For the 10-year window, the MHD and KD forecasts are almost the same and slightly worse than the Ens. wei. sum. For the 5-year window, the MHD forecast is almost the same as the Ens. wei. sum, while the KD forecast is remarkably worse than the other two. As far as the 10-year and 5-year assimilation windows are concerned, the following three seem to hold: (1) the Ens. wei. sum is always one of the best forecasts among the three, (2) the MHD forecast is the same as or slightly worse than the Ens. wei. sum, and (3) the KD forecast is often similar to MHD forecasts, although it is sometimes remarkably worse than the other two. We speculate that the MHD forecast is always slightly worse than the Ens. wei. sum because of a slight departure from the linear approximation over the forecast period.

We further examined the difference between the MHD and KD forecasts. Figure 7a shows the comparison of secular acceleration (SA) of the radial component of the geomagnetic field at the Earth’s surface between the MHD and KD forecasts for both 10-year and 5-year assimilation windows. SA was calculated by the equation, ${\text{SA}}\left( t \right) = \left( {F\left( {t + 0.25} \right) + F\left( {t - 0.25} \right) - 2 \cdot F\left( t \right)} \right)/\left( {0.25} \right)^{2}$, where $F\left( t \right)$ represents the geomagnetic main field and $t$ is time in year. After the calculation of SA, we applied 0.75-year (three-point) running average only to SA from the MCM6 model in order to reduce the noise. The results in Fig. 7a are the same as those in Fig. 6e, f. It is noticeable in Fig. 7a that SAs from the MHD and KD forecasts are totally different from each other; SAs from the MHD forecasts (Cases A5 and B5) generate synthetic jerks, i.e., a flip of spatial patterns of SA in sign from 2015.0 to 2017.0, and the phenomenon is more remarkable for the 5-year window. As for the KD forecasts (Cases A6 and B6), the spatial patterns of SAs do not change significantly from 2015.0 to 2017.0.

Fig. 7

a Comparison of SA, the second time derivative of the geomagnetic field at the Earth’s surface, between the MCM6 model, and the MHD and KD forecasts using 10- and 5-year assimilation windows. The initial conditions for the MHD and KD forecasts are given at 2014.25 for all forecasts. b, c Comparison of the first time derivative (SV) and second time derivative (SA) of two Gauss coefficients, $h_{32}$ and $h_{33}$, among the MCM6 model (the black line), the weighted sum of the final ensemble members by Eq. (13) (the Ens. wei. sum; the red line), the MHD forecast (the blue line), and the KD forecast (the green line) for 5- and 10-year assimilation windows. The assimilation windows are indicated by gray arrows. Panel c shows that SV of $h_{32}$ and $h_{33}$ have large discrepancies between the KD forecasts (the green lines) and the other three lines (the blue, red, and black lines) with 5-year assimilation window. The monotonic decrease and growth of SV in the KD forecasts are caused by steady negative and positive SA after the assimilation window, respectively, while oscillatory variation of SA in the MHD forecasts and the Ens. wei. sum suppress the departure of SA lines from that of the MCM6 model

We show in Fig. 7b, c the comparisons of SV and SA for two example Gauss coefficients, $h_{3}^{2}$ and $h_{3}^{3}$, which correspond to Fig. 6e, f, respectively, since we found that the large $\sqrt {{\text{d}}P}$ over the forecast period by the KD forecast with 5-year window (Case B6 in Fig. 6f) arises mainly from the misfit in degree $l = 3$ in comparison to the MHD forecast (Case B5). Top two panels in Fig. 7c, SV of $h_{3}^{2}$ and $h_{3}^{3}$ for the 5-year window, illustrate the reason for the worse KD forecast in Fig. 6f; the KD forecast (Case B6) shows rapid monotonic changes in the SV components after the initial conditions are imposed at the end of assimilation window. These are attributed to the steady positive or negative SA through the forecast period as shown in the bottom panels in Fig. 7c, which is associated with the fixed flow velocity in the KD forecast. In contrast, since the dynamic processes that allow oscillation of SA in time are present in the MHD forecasts, oscillatory SA variation of the MHD forecast results in moderate variation of SV and small departures from the observation (see Case B5 in Fig. 7c). The rapid monotonic change in SV in the KD forecast does not occur in the 10-year assimilation window case (see Case A6 in Fig. 7b). We speculate that shorter assimilation window is likely to cause large SA and SV intensities in the initial conditions at $k^{\prime} = K$ (Eq. 16) as shown in Fig. 7a, whereas it is easy to reduce $\sqrt {{\text{d}}P}$ over the assimilation window. When the short assimilation window of 5 years is adopted, the MHD forecast that allows dynamic process seems more stable rather than the KD forecast in the use of our assimilation scheme.

Performance of 5-year forecast in comparison to IGRF-12

At the end of this section, we compare the performance of our data assimilation-forecast scheme with that of IGRF-12 in terms of $\sqrt {{\text{d}}P}$ (Eq. 40) and $\sqrt {{\text{d}}P_{wo} }$ (Eq. 41) from the release time. Figure 8 shows $\sqrt {{\text{d}}P}$ and $\sqrt {{\text{d}}P_{wo} }$ from our forecasts and IGRF-12 with respect to the time from the release time, where we assume that the release time is 2014.75 for our numerical experiments and 2015.00 for IGRF-12, respectively. Table 1 summarizes the values of $\sqrt {{\text{d}}P}$ and $\sqrt {{\text{d}}P_{wo} }$ 4.50 years after the release time. In Fig. 8, we added two new results compared to Fig. 6, “extrapolation of SV of MCM6” (Case C2) and “no SV” (Case C3). In the forecast by “extrapolation of SV of MCM6”, we took instantaneous SV at 2014.25 from the MCM6 model and extrapolated it to the forecast period. The “no SV” forecast in turn provides the result literally with “no SV” after the assimilation window. The other four lines (Case C1, A4, A5, and A6) are the forecast by IGRF-12 and our best forecasts with 10-year assimilation window in Fig. 6e.

Fig. 8

Time evolutions of a $\sqrt {{\text{d}}P}$ (Eq. 40) and b $\sqrt {{\text{d}}P_{wo} }$ (Eq. 41) from the release times, 2014.75 for our numerical experiments and 2015.00 for IGRF-12. Four results in a, Cases C1, A4, A5, and A6, are the same as the results in Fig. 6e. Newly added blue and orange lines indicate the extrapolation of SV at 2014.25 of the MCM6 model, and the case without SV from 2014.25, respectively. See Table 1 for the values of $\sqrt {{\text{d}}P}$ and $\sqrt {{\text{d}}P_{wo} }$ 4.50 years after the release time

In both panels of Fig. 8, the orange lines is the worst while the blue line is the best. For the orange line, one can easily understand that any forecast schemes are much better than the result with “no SV”. As for the blue line, we should note that the SV at 2014.25 from the MCM6 model is very accurate compared to, for example, IGRF-12SV. This is because the MCM6 model was constructed by using the geomagnetic data up to 2019.50 and applying backward smoothing to determine SV at 2014.25 (Ropp et al. 2020), especially including the information on the following rapid change of SV due to the geomagnetic jerk in 2014. Strictly speaking, our numerical experiments also used the accurate SV at 2014.25 from the MCM3 model in calculation of the core surface flow. However, the effect of the accuracy of SV data at 2014.25, i.e., at the end of the assimilation window, is very small because of the homogeneous data covariance matrix ${\mathbf{R}}$ through the assimilation window (Eq. 39) and of the larger weight to the geomagnetic main field (compare $\alpha_{S}$ with $\alpha_{UW}$ in Table 1).

Our three forecasts in Fig. 8, Cases A4, A5, and A6, result in slightly worse forecast performance in terms of $\sqrt {{\text{d}}P}$ than that of IGRF-12 and almost the same in terms of $\sqrt {{\text{d}}P_{wo} }$. Since the pure performance of SV forecast for the 5 years after the release time can be measured by $\sqrt {{\text{d}}P_{wo} }$, Fig. 8b illustrates that the performance of our data assimilation-forecast scheme is comparable with that of IGRF-12SV. It is also noticeable that our numerical experiments used only data up to 2014.25, almost at the center of the geomagnetic jerk, around 2014.0 (Torta et al. 2015), while the candidate models for IGRF-12SV used data up to around 2014.50, or even up to around 2014.75 (e.g., Lesur et al. 2015), including data slightly after the jerk center. This fact implies that our data-assimilation forecast strategy has the possibility of retrieving part of information about the coming jerks from the data just before the jerks and weakening their effect on the 5-year SV forecast. Figure 8b also reveals that the IGRF-12 is clearly better than our SV forecast during the initial 2 years in the forecast period. We, therefore, speculate that our scheme may not reproduce the future magnetic jerks accurately, but can possibly provide a SV trend including jerks over the 5-year forecast period, compared to conventional SV candidate models for IGRF.

Since the Ens. wei. sum by Eq. (13) always provides the best forecast among the three forecasts through our numerical experiments, we decided to adopt it for estimation of our candidate SV model for IGRF-13, which is theoretically, and in most cases practically, equivalent to the forecast by MHD dynamo simulation starting at the end of assimilation window (the MHD forecast with $k^{\prime} = K$).

Summary of numerical experiments

We found in the application of our data assimilation and to the geomagnetic data from 2004.50 to 2014.25 that:

• A longer assimilation window works better than a shorter window, up to the assimilation window of 10 years, although we could not try assimilation windows longer than 10 years in this study.

• MHD forecast is not systematically superior to KD forecast. However, KD forecast is sometimes much worse than MHD forecast. Forecast by the Ens. wei. sum by Eq. (13) is always one of the best forecasts among the three.

• KD forecasts sometimes give worse results due to the monotonic increase/decrease in the intensity of geomagnetic field arising from the almost fixed pattern of SA since the release time, especially with the short assimilation window in the use of our data assimilation scheme. On the other hand, the MHD forecast seems stable, possibly because it can suppress large SVs by dynamic processes allowed.

• The performance of our data assimilation-forecast scheme turns out to be comparable with that of IGRF-12 for the purpose of 5-year SV forecast, even though the tested data assimilation window ends around the center of the geomagnetic jerk in 2014.0.

The SV candidate model for IGRF-13

We generated a final SV candidate model valid for the period from 2020.0 to 2025.0 based on the numerical experiments described in the previous section. We conducted the data assimilation with the 10-year assimilation window from 2009.50 to 2019.50 using the provided MCM6 model. The ensemble size was set to 960 in the same manner as the numerical experiments. We adopted values of $\alpha_{S}$ and $\alpha_{UW}$ for five iterations as listed in Table 2. In the 5th iteration, the adopted weights for $S_{l}^{m}$ and $(U_{l}^{m}$, $W_{l}^{m}$), i.e., $\alpha_{S}^{ - 1}$ and $\alpha_{UW}^{ - 1}$, are approximately 4:1 respecting each observation error.

Table 2 Weight factors for observation error covariance matrices in generation of the IGRF13-SV candidate model

Before the final iteration, we prepared the final ensemble simulation members by MHD dynamo simulations for both assimilation window and further 5.50 years starting from the end of assimilation window, 2019.50. We then forecast the future magnetic field by the weighted sum of all the ensemble members, i.e., the Ens. wei. sum by Eq. (13), for each epoch with 0.25 year interval from 2019.50 to 2025.00, which yielded time-series of Gauss coefficients, $p_{l}^{m} \left( t \right)$, for each mode from $\left( {l,m} \right) = \left( {1, 0} \right)$ through (8, 8). We generated a SV candidate model not by an averaged SV from 2020.0 to 2025.0 but by fitting of linear lines to our forecasts from 2019.50 to 2025.0. Line-fitting was applied to each time-series so as to estimate our candidate secular variation, $a_{l}^{m}$, during the 5.50 years:

$$p_{l}^{m} \left( t \right) = a_{l}^{m} \cdot \left( {t - t_{0} } \right) + p_{l}^{m} \left( {t_{0} } \right),$$

(43)

where the time origin, $t_{0}$, was set to the end of our data assimilation window, i.e., 2019.50. Figure 9 shows the fitting results for several modes, where we can confirm that the estimated SV (the blue lines) fits to the original SV forecast by the weighted sum of MHD simulation members (the red dashed lines).

Fig. 9

Comparison of Gauss coefficients between the final SV of our candidate model (the blue lines) and the forecasts by the Ens. wei. sum by Eq. (13) (the red dashed lines). The gray shadow areas behind the final SV correspond to the error provided in our SV model. The green solid lines behind the red dashed line denote the MCM6 model. The orange lines show time evolution of Gauss coefficients obtained from the MHD dynamo simulation with the initial condition at the end of the assimilation window, 2019.25 (the MHD forecast with $k^{\prime} = K$ in Eq. 16)

Whether or not the line-fit model in Eq. (43) gives a good secular variation estimate over the 5.50 years depends on each spherical harmonic mode. For some modes, it may be necessary to include higher time derivatives such as secular acceleration in order to describe the time variation in concern properly. It, therefore, is necessary to diagnose the validity of the line-fit model by examining the distribution of the resulting ensemble members for each mode, which will be described in the next section.

Error estimation of the candidate SV model

After carrying out the data assimilation, we estimated a 5-year SV candidate model by fitting Eq. (43) to the prediction using the Ens. wei. sum by Eq. (13) over the period from 2019.5 to 2025.0. We applied the same SV estimation procedure to every ensemble member and obtained 960 SV models from the ensemble. We used the standard deviation of 960 SV values as the error of our SV candidate model. Figure 10 shows two examples of the distribution of SV values including all the ensemble members. In each panel of Fig. 9, the estimated error is displayed by a gray area behind the estimated SV candidate line (the blue line).

Fig. 10

Histograms of the final distribution of ensemble members for SV ($\dot{g}_{1}^{0}$ and $\dot{g}_{2}^{0}$). The black and red dashed lines denote the range of one sigma (standard deviation) and the mean of the ensemble, respectively. The red dashed means correspond to components of the final SV candidate model

Discussion and conclusions

We derived an SV candidate model for IGRF-13 (listed in Table 3) using the 4DEnVar data assimilation technique and MHD dynamo simulations. This candidate model is the first ‘model’ contribution to the IGRF community from Japanese research groups. In this paper, we illustrated a potential of the 4DEnVar data assimilation in the field of short-term forecast of the geomagnetic field, through the numerical experiments with the existing geomagnetic data and construction of our SV candidate model for IGRF-13 SV.

Table 3 Derived SV candidate model

Our numerical experiments using the iterative 4DEnVar approach in the geomagnetic field prediction provided us important information. Introduction of the bias and trend terms in Eq. (10) effectively reduces the data misfit in the prediction of the geomagnetic field as shown in Fig. 5. We also confirmed that the bias and trend terms enhance validity of the linear approximation (Eq. 3) over the 10-year assimilation windows. In the numerical experiments, we tested three types of forecasts, the weighted sum of the ensemble members of MHD simulations extended to the forecast period by Eq. (13) (the Ens. wei. sum), the forecast by a single MHD dynamo simulation starting from the optimized initial state vector at $t_{k}^{'}$ (Eq. 16) (the MHD forecast), and the forecast by a single KD simulation from the optimized initial state vector at $t_{k}^{'}$ (the KD forecast). We found that the Ens. wei. sum is always one of the best forecasts among the tested three forecast types, and that longer assimilation window (up to 10 years) is superior to enhance the performance of SV forecast for 5 years after the assimilation window. It is also found that the KD forecasts are sometimes much worse than the MHD forecasts with short assimilation window, probably due to large SA and SV intensities in the initial condition and the lack of dynamic process that suppresses monotonic growth of the SV amplitude. This feature is just a characteristic of our current data assimilation and forecast scheme. If more stable estimation of SA at the end of assimilation windows becomes feasible in the future, the KD forecast may be the best forecast in conjunction with our data assimilation scheme.

We found that the SV forecast performance of our scheme is comparable with that of IGRF-12 in terms of $\sqrt {{\text{d}}P_{wo} }$ (Eq. 41) from the release time (see Fig. 8). However, there is still plenty of room of further improvement in our iterative 4DEnVar data assimilation. For example, we found synthetic rapid SV changes at the end of the assimilation window in the Ens. wei. sum forecast, e.g., the rapid SV change in $h_{3}^{2}$ around 2014.0 (red lines) in Fig. 7b and c, and the significant SV change in $g_{1}^{0}$ in Fig. 9 before and after the release epoch of 2020.0. These are possibly associated with the fact that the data misfit close to the edge of the assimilation window is likely larger than the center of the assimilation window (see cases of the Ens wei. sum in Fig. 6). Because this arises mainly from the time-independent data covariance matrix ${\mathbf{R}}$ (Eq. 39), one possible solution is to adopt relatively large weight, i.e., small $\alpha_{S}$ and $\alpha_{UW}$, close to the end of the assimilation window. Suppression of large SV and SA discrepancies from the observations at the end of assimilation window is one of the urgent issues that should be solved in the future.

Appendices

Appendix A: Renewal of the ensemble members

After obtaining the optimum initial condition, $\bar{\varvec{x}}_{0,m + 1}$, by;

$$\bar{\varvec{x}}_{0,m + 1} = \bar{\varvec{x}}_{0,m} + {\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{X} }}_{0,m} \hat{\varvec{w}}_{m} ,$$

(44)

a set of initial conditions, $\left\{ {\varvec{x}_{0,m + 1}^{\left( 1 \right)} \cdots \varvec{x}_{0,m + 1}^{\left( N \right)} } \right\}$, are necessary to perform MHD dynamo simulations to generate the ($m + 1$)th ensemble members. This can be achieved in a similar manner to the ensemble transform Kalman filter (Bishop et al. 2001);

$${\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{X} }}_{0,m + 1} = {\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{X} }}_{0,m} {\mathbf{T}}_{m} ,$$

(45)

where

$${\mathbf{T}}_{m} = \left( {\mathop \sum \limits_{k = 1} \left[ {{\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\varGamma } }}_{k,m}^{T} {\mathbf{R}}_{k}^{ - 1} {\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\varGamma } }}_{k,m} } \right] + \sigma_{m}^{2} \varvec{I }} \right)^{{ - \frac{1}{2} }} .$$

(46)

A set of initial conditions for the ($m + 1$)th ensemble members are obtained by:

$$\left( {\varvec{x}_{0,m + 1}^{\left( 1 \right)} \ldots \varvec{x}_{0,m + 1}^{\left( N \right)} } \right) = \left( {\bar{\varvec{x}}_{0,m + 1} \ldots \bar{\varvec{x}}_{0,m + 1} } \right) + \sqrt {N - 1} {\hat{\mathbf{X}}}_{0,m + 1} .$$

(47)

By this procedure, the ensemble gets shrunk so that the linear approximation (Eq. 3) holds better than in the previous step.

Appendix B: How to obtain $\varvec{w},\varvec{ a},\varvec{ b}$ that minimize $\hat{\varvec{J}}_{m} \left( {\varvec{w},\varvec{ b},\varvec{ a}} \right)$ in Eq. (10)

Equation (9) can be written as

$$\begin{aligned} \hat{J}_{m} \left( {\varvec{w},\varvec{b},\varvec{a}} \right) = \frac{1}{2}\left( {\begin{array}{*{20}c} {\varvec{w}^{{\mathbf{T}}} } & {\varvec{b}^{{\mathbf{T}}} } & {\varvec{a}^{{\mathbf{T}}} } \\ \end{array} } \right)\left( {\begin{array}{*{20}c} {\sigma_{m}^{2} {\mathbf{I}}} & 0 & 0 \\ 0 & {{\mathbf{P}}_{b}^{ - 1} } & 0 \\ 0 & 0 & {{\mathbf{P}}_{a}^{ - 1} } \\ \end{array} } \right)\left( {\begin{array}{*{20}c} \varvec{w} \\ \varvec{b} \\ \varvec{a} \\ \end{array} } \right) + \frac{1}{2}\mathop \sum \limits_{k = 1}^{K} \left[ {\varvec{y}_{k} - \varvec{g}_{k} \left( {\bar{\varvec{x}}_{0,m} } \right) - \left( {\begin{array}{*{20}c} {{\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\varGamma } }}_{k,m} } & {\mathbf{I}} & {k{\mathbf{I}}} \\ \end{array} } \right)\left( {\begin{array}{*{20}c} \varvec{w} \\ \varvec{b} \\ \varvec{a} \\ \end{array} } \right)} \right]^{\text{T}} {\mathbf{R}}_{k}^{ - 1} \left[ {\varvec{y}_{k} - \varvec{g}_{k} \left( {\bar{\varvec{x}}_{0,m} } \right) - \left( {\begin{array}{*{20}c} {{\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\varGamma } }}_{k,m} } & {\mathbf{I}} & {k{\mathbf{I}}} \\ \end{array} } \right)\left( {\begin{array}{*{20}c} \varvec{w} \\ \varvec{b} \\ \varvec{a} \\ \end{array} } \right)} \right]. \end{aligned}$$

(48)

Defining the following:

$$\varvec{w}^{*} = \left( {\begin{array}{*{20}c} \varvec{w} \\ \varvec{b} \\ \varvec{a} \\ \end{array} } \right),\varvec{ }{\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\varGamma } }}_{k,m}^{\varvec{*}} = \left( {\begin{array}{*{20}c} {{\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\varGamma } }}_{k,m} } & {\mathbf{I}} & {k{\mathbf{I}}} \\ \end{array} } \right), {\mathbf{P}}_{m}^{*} = \left( {\begin{array}{*{20}c} {\sigma_{m}^{ - 2} {\mathbf{I}}} & 0 & 0 \\ 0 & {{\mathbf{P}}_{b} } & 0 \\ 0 & 0 & {{\mathbf{P}}_{a} } \\ \end{array} } \right),$$

(49)

$\hat{J}_{m}$ can be regarded as a function of $\varvec{w}^{*}$. The function $\hat{J}_{m}$ is minimized at

$$\begin{aligned}\hat{\varvec{w}}^{\varvec{*}} & = \left( {\mathop \sum \limits_{k} \left[ {{\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\varGamma } }}_{k,m}^{{\varvec{*}{\text{T}}}} \varvec{ }{\mathbf{R}}_{k}^{ - 1} {\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\varGamma } }}_{k,m}^{*} } \right] + {\mathbf{P}}_{m}^{* - 1} \varvec{ }} \right)^{ - 1} \\ & \quad \times \mathop \sum \limits_{k} \left( {{\mathbf{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\varGamma } }}_{k,m}^{{\varvec{*}{\text{T}}}} {\mathbf{R}}_{k}^{ - 1} \left[ {\varvec{y}_{k} - \varvec{g}_{k} \left( {\bar{\varvec{x}}_{0,m} } \right)} \right]} \right).\varvec{ } \end{aligned}$$

(50)

Appendix C: Conversion of dimensional core flow into non-dimensional one

Fluid flow near the CMB is derived from the MCM3 model by using physical parameters corresponding to the actual core. On the other hand, dimensionless parameters are given in numerical MHD dynamo models. Hence, for data assimilation using geodynamo model outputs, core surface flows must be determined not by using actual physical parameters, but by those for numerical dynamos. In this study, we use dimensionless parameters given as follows: the Ekman number, $E = \nu /2{{\varOmega }}\left( {r_{oc} - r_{ic} } \right)^{2} = 3 \times 10^{ - 5}$, and the magnetic Prandtl number, $Pm = \nu /\eta = 2$, where $\nu$ is the kinematic viscosity of the core fluid, $\varOmega$ is the rotation rate of the mantle, $r_{oc} = 3485 {\text{km}}$ is the radius of the outer core, $r_{ic} = 0.35 \times r_{oc} = 1220 \;{\text{km}}$ is the radius of the inner core, and $\eta = \left( {\mu_{0} \sigma } \right)^{ - 1}$ is the magnetic diffusivity with the magnetic permeability of vacuum, $\mu_{0} = 4\pi \times 10^{ - 7} {\text{H m}}^{ - 1}$ and the electrical conductivity of core, $\sigma$. Then, the typical time scale given as $\tau_{\nu } = \left( {r_{oc} - r_{ic} } \right)^{2} /\nu = 13,049$ years leads to $\nu = 12.467 {\text{m}}^{2} {\text{s}}^{ - 1}$, which in turn gives $\varOmega = 4.050 \times 10^{ - 8} {\text{s}}^{ - 1}$ and $\sigma = 1.277 \times 10^{5} {\text{S m}}^{ - 1}$. Fluid flow can be scaled by $\left( {r_{oc} - r_{ic} } \right)/\tau_{\nu } = 5.504 \times 10^{ - 6} {\text{m s}}^{ - 1} = 0.1736 {\text{km year}}^{ - 1}$.

The Ekman number used in Matsushima (2015) is defined as $E^{\prime} = \nu /\varOmega r_{oc}^{2} = E \cdot 2\left( {r_{oc} - r_{ic} } \right)^{2} /r_{oc}^{2} = 2.534 \times 10^{ - 5}$. Thickness of the Ekman boundary layer is, therefore, $\delta_{E} = r_{o} E^{ '1/2} = 17.544 {\text{km}}$. It should be noted here that the effect of magnetic field reduces the boundary layer thickness. In our numerical dynamo model, grid points in the radial direction inside the outer core are given as

$$r_{k} = \frac{1}{2}\left[ {r_{o} + r_{i} - \left( {r_{o} - r_{i} } \right)\cos \left( {\frac{k\pi }{{N_{r} }}} \right)} \right],$$

where $N_{r} = 99$ and $k = 0, 1, \ldots , N_{r}$. As shown in Fig. 4a, $r_{b1} = r_{oc} - \xi_{1}$ and $r_{b2} = r_{oc} - \xi_{2}$ should be defined to be $\xi_{1} < \delta_{E}$ and $\xi_{2} \gg \delta_{E}$. In this study, we adopted $r_{b1} = 3475.9 {\text{km}}$ and $r_{b2} = 3428.5 {\text{km}}$ corresponding to $r_{95}^{*}$ and $r_{89}^{*} ,$ respectively.