Hypotheses Testing

As outlined in the short tutorial, we can test specific hypotheses regarding the BLP, GATES, and RATEs to validate the estimated heterogeneity. This article explores these hypotheses and their implications.

The notation is the same as in the short tutorial.

BLP

Consider the heterogeneity parameter $\beta_2 = Cov [ \tau ( X_i ), \hat{\tau} ( X_i ) ] / Var [ \hat{\tau} ( X_i ) ]$. Notice that $Cov [ \tau ( X_i ), \hat{\tau} ( X_i ) ] = 0$ in two cases:

• If $\tau ( x) = \tau$ for all $x$ (that is, if the effects are homogeneous);
• If $\hat{\tau} ( \cdot )$ is pure noise uncorrelated to $\tau ( \cdot )$ (that is, if our CATE estimates are really bad).

Consequently, $\beta_2 = 0$ either if the effects are homogeneous or our CATE estimates are unreliable. On the other hand, $Cov [ \tau ( X_i ), \hat{\tau} ( X_i ) ] = Var [ \hat{\tau} ( X_i ) ]$ if $\hat{\tau} ( x ) = \tau ( x)$ for all $x$ (that is, if we have “perfect” CATE estimates). Therefore, $\beta_2 \approx 1$ if our CATE estimates are reliable.

We can thus consider the hypothesis $\beta_2 = 0$ as a test for effect heterogeneity and the reliability of our CATE estimates. If the effects are homogeneous, or if our estimates are unreliable (or if both conditions hold), then $\beta_2$ is close to zero, and we should fail to reject our hypothesis. Conversely, if the effects are heterogeneous and our estimates are accurate, then $\beta_2$ is close to one, and we should reject our hypothesis.

If we estimate the BLP by one of the strategies outlined in the short tutorial and using only the validation sample, then $\hat{\beta}_2$ exhibits well-behaved asymptotic properties conditioned on the training sample. This enables us to employ standard tools for inference, such as conventional confidence intervals and $p$-values.

GATES

One could compare the estimated GATES to validate how various groups respond differently to the treatment. However, disparities in the point estimates can emerge merely due to estimation noise.

A more appropriate method for assessing the presence of systematic heterogeneity is to test the hypothesis that all GATES are identical, namely $\gamma_1 = \gamma_2 = \dots = \gamma_K$. Alternatively, we can test whether the difference in the GATES for the most and least affected groups is statistically significant, that is, $\gamma_K = \gamma_1$.

If we estimate the GATES by one of the strategies outlined in the short tutorial and using only the validation sample, then $\hat{\gamma}_k$ exhibits well-behaved asymptotic properties conditioned on the training sample. This enables us to employ standard tools for inference, such as conventional confidence intervals and $p$-values.

RATE

Notice that $TOC ( u; \hat{\tau} ) = 0$ for any $u \in (0, 1]$ in two cases:

• If $\tau ( x) = \tau$ for all $x$ (that is, if the effects are homogeneous);
• If $\hat{\tau} ( \cdot )$ is pure noise uncorrelated to $\tau ( \cdot )$ (that is, if our CATE estimates are really bad).

Consequently, $\theta_{\alpha} ( \hat{\tau} ) = 0$ either if the effects are homogeneous or our CATE estimates are unreliable.

We can thus consider the hypothesis $\theta_{\alpha} ( \hat{\tau} ) = 0$ as a test for effect heterogeneity and the reliability of our CATE estimates. If the effects are homogeneous, or if our estimates are unreliable (or if both conditions hold), then $\theta_{\alpha} ( \hat{\tau} )$ is close to zero, and we should fail to reject our hypothesis. Conversely, if the effects are heterogeneous and our estimates are accurate, then $\theta_{\alpha} ( \hat{\tau} )$ is large enough so that should reject our hypothesis.

If we estimate the RATE by the strategy outlined in the short tutorial and using only the validation sample, then $\hat{\theta}_{\alpha} ( \hat{\tau} )$ exhibits well-behaved asymptotic properties conditioned on the training sample. This enables us to employ standard tools for inference, such as conventional confidence intervals and $p$-values.