Exploring Model Expansion Interactively

The following notes attempt to explain the main ideas of my paper "Identifiability and Falsifiability: Two Challenges for Bayesian Model Expansion" using interactive plots and an emphasis on intuition. When I began this project, I wanted to understand the methodological consequences of model expansion.

Model expansion is often motivated by observing some feature of our observed data (e.g. as quantified by a test statistic) which cannot be reproduced in simulated data generated from our current model. This sort of model expansion is thus a natural component, explicitly or implicitly, of many statistical modeling workflows (see Gelman, et al. 2020; Box's loop in Blei 2014; and Betancourt 2020). In practice, expansion is usually achieved by adding new parameters to the base model. Some examples are given in the following table.

In particular, expanding models in this way necessitates increasing the dimension of the parameter space. We should then ask what consequences this increasing dimension might have for our statistical methods. From a computational vantage point, it is well-understood how high dimensionality can hurt us (through curses of dimensionality) and how we can sometimes regain control in these settings (through high dimensional regularities like concentration of measure). As a result, computational methods (e.g. the Hamiltonian Monte Carlo sampler in Stan) have been carefully developed to handle these situations.

But what about our (seemingly) simpler tools for Bayesian inference and model checking? Take for example the default summary output from Stan, consisting of marginal posterior means and standard deviations. The practical usefulness of this output depends on the standard deviations being not too large, or our inferences may be too uncertain to support substantively interesting conclusions. Similarly, the usefulness of the posterior predictive $p$-value for model checking depends on its power under nearby alternatives to our current model.

The primary goal of this project is to understand how such properties (marginal uncertainty, test power) scale under the process of model expansion. It is almost certainly impossible to give a fully general answer, as the relevance of any theoretical analysis will depend on how faithfully the assumptions of that analysis map onto whatever applied modeling scenario we find ourselves in. With these necessary limitations in mind, my main argument is that model expansion tends to increase our uncertainty about parameters and decrease our power to detect misfit.

A Simple Regression

The main ideas of this argument can be presented in an extremely simple regression example. Imagine that we begin with a simple linear regression of a single predictor:

where $j\in \{1,2\}$, $x_1=(0,1)$, and ${\sigma }_b$ is taken large enough that our prior is weakly informative for the coefficient. Then, imagine that we expand this regression by adding a second predictor $x_2$ with coefficient ${\beta }_2$ independent of ${\beta }_1$ and with an identical prior:

The following interactive plots examine the behavior of the expanded model under different assumptions about the correlation between $x_1$ and $x_2$. All distributions are recentered at 0 to make for easier comparisons of their scales. The left panel plots the marginal posterior against the marginal prior for ${\beta }_1$. The right panel plots the sampling distribution for an independent replication/resample $y_1^{\mathrm{rep}}$ of $y_1$ at the posterior mean of the parameters, i.e. $p(y_1^{\mathrm{rep}}|{\hat{\beta }}_1,{\hat{\beta }}_2)$ against the posterior predictive distribution of $y_1^{\mathrm{rep}}$, i.e. $p(y_1^{\mathrm{rep}}|y_1,y_2)$.

The comparison in the left panel is clearly related to our ability to make useful inferences about ${\beta }_1$, since a posterior that is much narrower than the prior represents a greater information gain from observing the $y_j$. It is likely less clear how the comparison in the right panel relates to the power of model diagnostics. Such a connection can be argued formally, but here we will focus on intuition. Note that if the sampling distribution $p(y_1^{\mathrm{rep}}|{\beta }_1,{\beta }_2)$ had no dependence on the parameters, then the two distributions in the right panel would be equal and we would have no uncertainty about the true sampling distribution (assuming model correctness). Likewise, dissimilarity between the distributions in the right panel indicates greater posterior uncertainty about the true sampling distribution for $y_1$. When this uncertainty is large, the model contains many different plausible descriptions of the data generating process. Thus, in any check of the model, it has as many different attempts to explain the observed data, decreasing the power of the check by making it easier for the model to pass.

Thus, for purposes of parameter identification, we want the two distributions in the left panel to be as dissimilar as possible; and for purposes of model checking, we want the two distributions in the right panel to be as similar as possible. Varying the predictor correlation from 0 to 1, you can see that the best case for both of these goals recovers the relationships between the plotted distributions that exist in the base model. At all other correlations, the relationships between the plotted distributions are worse than in the base model. Furthermore, the best-cases for each panel occur at opposite extremes: 0 correlation is best for parameter identification and total correlation is best for model checking. Thus, in this simple case, model expansion degrades the performance of our basic inference and evaluation tools, and expansions which limit the damage to inference tend to more severely damage evaluation, and vice versa.

Parameter Uncertainty and Model Expansion

In general, expansion will not inevitably degrade inference and evaluation as it does in this example, but the general tendency will remain. In order to see this generalization, we must first give a more formal definition of model expansion.

Intuitively, this definition just requires that an expanded model embed the base model as a special case. This gives us a well-defined way to talk about a base and expanded model as sharing some parameters (i.e. $\mathbf{\theta }$ in the above definition), allowing us to draw direct comparisons with respect to these parameters. Furthermore, all of the examples of model expansion that were given in the table above can be seen to fit this formal definition.

It follows immediately from the definition that the posterior distribution of an expanded model also includes the posterior of the base model as a special case, again just by conditioning on ${\mathbf{\lambda }}_0$. With this in mind, we can see how expanded models tend to exhibit increased uncertainty for the parameters they share with the base model. The following interactive figure demonstrates this idea. The top panel displays a joint posterior distribution between the new parameters $\lambda$ (y axis) and the shared parameters $\mathbf{\theta }$ (x axis). The controls allow adjusting both the dependence of the conditional means

$$\mathbb{E}[\theta |\lambda ,\mathbf{y}]$$

on $\lambda$ as well as the marginal posterior $p(\lambda |\mathbf{y})$. The base model's posterior distribution is obtained by conditioning on $\lambda =0$. This distribution is constant across the various attainable joint posterior distributions, so that all such joint distributions represent expansions of the same base model. Finally, the bottom panel displays the marginal posterior distributions over $\theta$ for the base an expanded model, and the heatmap next to the controls represents the difference in uncertainty between these two distributions. Blue values represent cases where the expanded model has greater uncertainty, and red values represent cases where the base model has greater uncertainty (where uncertainty is measured by the information entropy of the marginal distributions).

As is apparent from examining the changes in the marginal for different configurations of the joint posterior (or, equivalently, from the heatmap), most of the possible expanded models exhibit greater uncertainty about $\theta$ than the base model. Furthermore, this visualization shows us why we should expect this to be the case in general. Suppose we shift the marginal posterior $p(\lambda |\mathbf{y})$ towards values for which the conditional posteriors $p(\theta |\lambda ,\mathbf{y})$ have greater confidence (i.e. we move up on the heatmap). Even for such marginal posteriors, we can still easily end up with greater marginal uncertainty about $\theta$ since the aforementioned conditional posteriors need not place the majority of their probability over the same intervals. Visually, this is seen by then increasing the dependence of the conditional expectations on $\lambda$ (i.e. by moving to the right on the heatmap).

Sampling Distribution Uncertainty and Model Expansion

Another simple example illustrates why model uncertainty about the (true) sampling distribution $p(\mathbf{y}|\cdot )$ tends to increase through model expansion. For our base model, we assume the components of $\mathbf{y}$ are sampled i.i.d. from a normal population with known scale:

We then expand this model by allowing the population scale to be unknown as well, arriving at the specification:

The $\mathrm{NG}$ prior is the Normal-Gamma prior, which is conjugate for this problem. The marginal prior on $\theta$ matches the prior in the base model, and ${\mu }_{{\sigma }^2}$ is the prior mean of the population variance ${\sigma }^2$. The (average) accuracy with which we can estimate the unknown mean $\theta$ is controlled by the average "noise level" $r={\mu }_{{\sigma }^2}/J$, where $J$ is the sample size (i.e. the dimension of $\mathbf{y}$). With this setup, we can compare the posterior uncertainty about the true sampling distribution between base and expanded models. The following figure plots the change in this uncertainty from base to expanded model against the sample size. The average noise level can be adjusted by moving the slider.

Two features are immediately apparent from the plot. First, the difference tends to increase with the sample size, such that it is very difficult to achieve a negative difference for larger sample sizes. Second, to achieve negative differences for even moderate sample sizes, the average noise level must be substantially larger than 1. We can make sense of this by dividing the change in sampling distribution uncertainty into two parts. On one hand, for any fixed distribution over the population mean $\theta$, increasing the typical size of $\sigma$ will "blur" the distributions $p(\mathbf{y}|\theta ,\sigma )$ together (as the differences in their means will be swamped by their larger variance). This makes the plausible sampling distributions more similar and puts downward pressure on the plotted uncertainty difference. Symmetrically, this effect could equally put upward pressure on the uncertainty difference if the typical size of $\sigma$ were to decrease.

On the other hand, including $\sigma$ as a parameter creates an additional degree of freedom in terms of which the sampling distributions $p(\mathbf{y}|\theta ,\sigma )$ may vary. In other words, for any value of $\theta$, there was only one possible sampling distribution in the base model, but there is a potentially wide variety of possibilities in the expanded model, depending on the value of $\sigma$. This additional degree of freedom places upward pressure on the uncertainty difference.

It is this effect of the additional degree of freedom that accounts for the fact that the uncertainty difference is almost always positive in this plot. Adjusting the slider, it is easy to see that the noise level must be much high than 1 (which is the noise level in the base model with $J=1$) in order to offset this additional degree of freedom and push the uncertainty difference into negative territory. Furthermore, it is this addition of degrees of freedom which creates the tendency for sampling distribution uncertainty to increase in general.

An Uncertainty Tradeoff in General

In the last two examples, I have attempted to convey intuition for why uncertainty about both the parameters and sampling distribution tends to increase through model expansion. However, in both cases, the increase in uncertainty was not inevitable, and we might hope that if we are sufficiently alert or clever, we can avoid the worst of these consequences by choosing the more favorable expansions (assuming these are compatible with our understanding of the world). This is where the third feature of our first example comes in the play, namely the apparent tradeoff between parameter and sampling distribution uncertainty. At this point, you may rightly be wondering whether this feature of the regression example also generalizes to other cases reliably. A simple heuristic that suggests this pattern should hold with some generality is given by the observed Fisher information with respect to $\theta$. In particular, suppose we have a scalar $\theta$ and consider how this quantity varies with $\mathbf{\lambda }$ in the expanded model:

If, for typical $(\lambda )$, this quantity is larger than the observed Fisher information in the base model, then we expect relatively lower marginal uncertainty about $\theta )$ in the expanded model. But large values of this second derivative also indicate that the sampling distribution $p(\mathbf{y}|\theta ,\mathbf{\lambda })$ changes rapidly with $\lambda$, increasing the "strength" of the additional degree of freedom and putting upward pressure on the uncertainty about the sampling distribution. And clearly the reverse tradeoff exists if this second derivative is smaller than the base model's observed information. Furthermore, if $\mathbf{\theta }$ is a vector, then we can make similar arguments involving the spectrum of the corresponding information matrices. It is of course interesting to ask when these ideas can be made rigorous, and the main argument of my paper presents a first attempt at doing so.

Sampling Distribution Uncertainty and Model Checking

I will close this post with a few words on why we should care about our uncertainty about the true sampling distribution. In fact, I think there are a few distinct reasons, but for now I will focus on how large levels of uncertainty can produce undesirable behavior in the posterior predictive $p$-value. Again my focus will be on the intuition that we can extract from a simple model. Consider modeling two scalar data points $y_1,y_2$ as an i.i.d. sample from a $t$ distribution with fixed degrees of freedom $d$ and unknown location $\theta$.

We will assume our data points are of the form $(y_1,y_2)=(-y,y)$ for various levels of $y>0$. As $y$ increases, the model becomes increasingly misspecified, as the sampling distribution has fixed variance equal to $1$. This is evident from the top-right panel of the following figure, which plots the observed data point against the posterior predictive distribution of the model. As we increase the magnitude of $y$ with the slider, the observed data becomes increasing atypical under the posterior predictive distribution.

We can then compute posterior predictive $p$-values $p_T$ for the test statistics $T(y_1,y_2)=y_1$ and $T(y_1,y_2)=y_2$. By symmetry, these $p$-values are equal, and are indicated by the dashed line in the top-left panel. These are also plotted against the magnitude of $y$ in the bottom panel. We observe two unpleasant features of these $p$-values. First, they never drop below $0.05$, a common nominal threshold for rejection. Furthermore, they actually begin increasing again after the magnitude of $y$ is large enough, even though the overall model fit becomes worse.

A clearer picture of the model fitness is painted by the conditional $p$-values $p_T(\theta )$, defined as the tail probabilities with respect to the individual sampling distributions $p(y_1^{\mathrm{rep}},y_2^{\mathrm{rep}}|\theta )$ rather than the posterior predictive $p(y_1^{\mathrm{rep}},y_2^{\mathrm{rep}}|y_1,y_2)$. As the notation suggests, these depend on $\theta$, and the top-left panel plots these against $\theta$ for each of our two test statistics. From this we can see that the usual $p$-value remains above our rejection threshold because each statistic becomes highly typical at the extremes of one or the other posterior mode as $y$ becomes large. But both statistics are vanishingly unlikely over most of the posterior mass, and there are no values of $\theta$ for which the statistics are simultaneously typical for $y$ greater than $4$.

While all of the above observations were already obvious from the top-right panel, examining these conditional $p$-values reveals that the poor performance of the posterior predictive $p$-value can be explained by the variation in the model fitness over $\theta$. Such extreme variation in fitness over the parameters is only possible when uncertainty about the true sampling distribution is sufficiently large. Said another way, if the sampling distributions that were plausible under the posterior were all very similar, then the conditional $p$-values computed with respect to them could not vary so severely over the support of the posterior. Only in cases like this where sampling distribution uncertainty is large can severe misfit hide in local regions of parameter space.