Math behind KIM#
Consider a vector of inputs \(\mathbf{X} = [X_1,...,X_{N_x}]\) and a vector of outputs \(\mathbf{Y} = [Y_1,...,Y_{N_y}]\). The objective is to build up a mapping function \(f\) from \(\mathbf{X}\) to \(\mathbf{Y}\), such that \(f: \mathbb{R}^{N_x} \rightarrow \mathbb{R}^{N_y}\), based on \(N_e\) pairs/realizations of \(\mathbf{X}\) and \(\mathbf{Y}\). However, it is oftentimes hard to develop an accurate composite \(f\), given the limited training data, partially due to the high computational cost of model ensemble simulation. Instead, it would be desirable to develop a separate inverse mapping \(f_j\) for each \(Y_j \in \mathbf{Y}\) by using a reduced space \(\mathbf{X}^S_j \subseteq \mathbf{X}\) that is most relevant to \(Y_j\), such that \(f_j: \mathbb{R}^{N_{x_j}} \rightarrow \mathbb{R}\) (see examples in Jiang et al. [5] and Wang et al. [9]. The identification of \(\mathbf{X}_j\) for a given \(Y_j\) involves a two-step filtering as follows.
Step 1: Filtering by global sensitivity analysis. We first perform a mutual information-based global sensitivity analysis to narrow down a subset \(\mathbf{X}^{S_1}_j\), each of which shares zero information with \(Y_j\) such that:
where \(I(X_i;Y_j)\) is the mutual information between \(X_i\) and \(Y_j\) [2]. Based on the \(N_e\) realizations, \(I\) is calculated on the joint probability of \(X_i\) and \(Y_j\) using either binning method or k-nearest-neighbor method. Following Jiang et al. [5], a statistical significance test is performed to identify the significant \(I\) (i.e., \(I(X_i;Y_j) \neq 0\)) with a significance level of \(1-\alpha\) on 100 bootstrap samples.
Step 2: Filtering by redundancy check. Then, we conduct a further assessment that filtered out any model output in \(\mathbf{X}^{S_1}_j\) whose dynamics are redundant to \(Y_j\) given the knowledge of other outputs. This is achieved through a conditional independence test using conditional mutual information [2] given as:
where \(\mathbf{X}^{S_1}_j \backslash X_i\) is the remaining set of \(\mathbf{X}^{S_1}_j\) by excluding \(X_i\); \(I(X_i;Y_j|\mathbf{X}^{S_1}_j \backslash X_i)\) is the conditional mutual information between \(X_i\) and \(Y_j\) conditioning on \(\mathbf{X}^{S_1}_j \backslash X_i\). \(I(X_i;Y_j|\mathbf{X}^{S_1}_j \backslash X_i) = 0\) indicates that \(X_i\) and \(Y_j\) are independent given the knowledge of \(\mathbf{X}^{S_1}_j \backslash X_i\). However, calculating \(\mathbf{X}^{S_1}_j \backslash X_i\). \(I(X_i;Y_j|\mathbf{X}^{S_1}_j \backslash X_i)\) faces the curse of dimensionality due to the potential high dimension in \(\mathbf{X}^{S_1}_j\). To address this, we leverage the idea of Peter-Clark algorithm for causal inference detection [8] to evaluate the zeroness of a high-dimensional conditional mutual information by gradually adding conditioning variables. Specifically, we approximated \(I(X_i;Y_j|\mathbf{X}^{S_1}_j \backslash X_i)\) via \(I(X_i;Y_j|\mathbf{X}^{S_2}_j)\), where \(\mathbf{X}^{S_2}_j\) is a subset of \(\mathbf{X}^{S_1}_j \backslash X_i\) with a predefined maximum cardinality \(K\) (i.e., three by default). Rather than exhausting every subset of a given cardinality, \(\mathbf{X}^{S_2}_j\) is chosen greedily as the \(k\) members of \(\mathbf{X}^{S_1}_j \backslash X_i\) with the highest Step 1 sensitivity, for \(k = 1, ..., K\) in increasing order. Candidates \(X_i\) are visited from least to most sensitive based on the calculated \(I(X_i;Y_j)\) in Step 1; at each \(k\), we conducted a statistical significance test on assessing \(I(X_i;Y_j|\mathbf{X}^{S_2}_j) = 0\), and \(X_i\) is removed from \(\mathbf{X}^{S}_j\) as soon as one such test is not significant (i.e., \(I(X_i;Y_j|\mathbf{X}^{S_2}_j) = 0\)), after which the candidate set used to build \(\mathbf{X}^{S_2}_j\) for subsequent tests is updated accordingly.
Step 3: Uncertainty aware estimation by training ensemble neural networks. For each parameter \(Y_j\), we train an ensemble of fully-connected neural networks by varying the hyperparameters, including the number of hidden layers, the number of hidden neurons, and the learning rate. We split the \(N_e\) model realizations into training, validation, and testing dataset. For each model inference, the ensemble learning enables the predictions through weighted mean \(\mu_w\) and weighted standard deviation \(\sigma_w\) as:
where \(N_e\) is the number of ensemble neural networks; \(\tilde{y_k}\) is the estimation by the \(k\) th neural network; \(w_k\) is the weight to the \(k\) th prediction and is calculated through the corresponding loss value in the validation dataset \(\mathcal{L}_{k,\text{val}}\), such that \(w_k = \frac{1/\mathcal{L}_{k,\text{val}}}{\sum^{N_e}_{k=1} 1/\mathcal{L}_{k,\text{val}}}\).
When evaluating the estimation on the test dataset, we further quantified the bias and uncertainty of the prediction as:
where \(E\) is the expectation operator and \(y\) is the true value.