Fisher's Exact test and Log Odds test ST551 Lecture 22

Fisher’s Exact test

For \(2\times 2\) tables

Setting: two independent samples

\(Y_1, \ldots, Y_n\) i.i.d from Bernoulli\((p_Y)\)
\(X_1, \ldots, X_m\) i.i.d from Bernoulli\((p_X)\)

Null hypothesis: \(p_Y = p_X\) Homegeneity of proportions

Test statistic: Probability of observed table conditional on margins

p-value: Sum of probability of all tables as or more extreme than observed table.

Data from last time

##        ate_breakfast
## cat_dog no yes
##    cats  6   9
##    dogs  6  14

A smaller (in sample size) example

##        ate_breakfast
## cat_dog no yes
##    cats  4   4
##    dogs  3   4

A smaller (in sample size) example with margins

##        ate_breakfast
## cat_dog no yes Sum
##    cats  4   4   8
##    dogs  3   4   7
##    Sum   7   8  15

Under null, there is no difference in the probability of eating breakfast between people who prefer cats and people who prefer dogs, what is the probability of observing this exact table, conditioning on the margins?

Working out the test statistic

Under the null \(p_Y = p_X = p\).

Conditional on row sums \(R_1 = a + b\) and \(R_2 = c + d\):

\(n_{12} \sim Binomial(R_1, p)\)
\(n_{22} \sim Binomial(R_2, p)\)

But we also need \(n_{12} + n_{22} = C_2\).

I.e. What is \(P(n_{12} = b \mid n_{12} + n_{22} = C_2)\)?

Turns out to be: \[ = \frac{{a+b \choose b} {c + d \choose d}}{{N \choose b+d}} \] a.k.a Hypergeometric Distribution

Derivation

\(P(n_{12} = b) = {a+b \choose b} p^b(1-p)^{a}\)

\(P(n_{22} = d) = {c + d \choose d} p^d(1-p)^{c}\)

\(P(n_{12} + n_{22} = b + d) = {N \choose b+d} p^{b + d}(1-p)^{a+c}\)

\[ \begin{aligned} P(n_{12} = b \mid n_{12} + n_{22} = b + d) &= \frac{P(n_{12} = b, n_{12} + n_{22} = b + d)}{P(n_{12} + n_{22} = b + d)} \\ &= \frac{P(n_{12} = b, n_{22} = d)}{P(n_{12} + n_{22} = b + d)} \\ &= \frac{{a+b \choose b} p^b(1-p)^{a}{c + d \choose d} p^d(1-p)^{c}}{{n \choose b+d} p^{b + d}(1-p)^{a+c}}\\ &=\frac{{a+b \choose b} {c + d \choose d}}{{N \choose b+d}} \end{aligned} \]

Example: Test statistic

##        ate_breakfast
## cat_dog no yes Sum
##    cats  4   4   8
##    dogs  3   4   7
##    Sum   7   8  15

Under null, probability of seeing this table, given margins:

\[ \text{Prob} = \frac{{8 \choose 4} {7 \choose 4}}{{15 \choose 8}} = 0.38 \]

p-values

More extreme? Depends on alternative:

• \(p_Y > p_X\), greater \(n_{12}\)
• \(p_Y < p_X\), smaller \(n_{12}\)
• \(p_Y \ne p_X\), less likely tables

Example: p-values lower alternative

\(H_A: p_Y < p_X\)

More extreme tables would have \(n_{12} =\) 3, 2, 1 or 0

Your turn: Complete the table if \(n_{12} = 3\):

Prob = \[ \frac{{\phantom{a+b} \choose \phantom{b}} {\phantom{c + d} \choose \phantom{d}}}{{\phantom{n} \choose \phantom{b+d}}} = 0.18 \]

Example: p-values lower alternative

\(H_A: p_Y < p_X\)

p-value \(= 0.381 + 0.183 + 0.03 + 0.001 = 0.595\)

p-values lower alternative

Example: p-values greater alternative

\(H_A: p_Y > p_X\)

More extreme tables would have \(n_{12} =\) 5, 6, 7 or 8

Your turn: What are the more extreme tables?

p-value \(= 0.381 + 0.305 + 0.091 + 0.009 + 0 = 0.786\)

Example: p-values two-sided alternative

Which tables are less likely than that observed?

In R

Careful with direction (rows permuted in R)

x <- xtabs(~ cat_dog + ate_breakfast, data = class_data)
fisher.test(x, alternative = "greater")

## 
##  Fisher's Exact Test for Count Data
## 
## data:  x
## p-value = 0.397
## alternative hypothesis: true odds ratio is greater than 1
## 95 percent confidence interval:
##  0.3796782       Inf
## sample estimates:
## odds ratio 
##   1.535689

Sampling for binomial proportions

• Multinomial Obtain a sample of \(N\) units, and cross classify according to the grouping variable \(G\) and the binary response variable \(Y\).

  • Can estimate all probabilities, \(P(Y_i = 1)\), \(P(G_i = 1)\), \(P(Y_i = 1 | G_i = 0)\) etc.

• Two-sample Obtain two samples, one of size \(n\) from group 1 (\(G_i = 0\)), and one of size \(m\) from group 2 (\(G_i = 1\)).

  • Can’t estimate \(P(G_i = 1)\) or \(P(Y_i = 1)\) but can estimate \(P(Y_i = 1 | G_i = 0)\).

Retrospective studies

Sometimes, particularly if an outcome is very rare, it’s hard to observe any successes \(Y_i = 1\), with Multinomial or Binomial sampling.

E.g.
Response variable \(Y =\) struck by lightning (Yes/No)
Grouping variable \(G =\) regular golfer (Yes/No)

Lightning strikes are very rare events, so even with rather large samples we may not observe anybody who has been struck.

In a retrospective sample we get \(s\) units where \(Y_i=1\) and \(r\) units where \(Y_i=0\) then classify them to the groups \(G_i\).

Example

Retrospective studies

We can no longer estimate \(P(Y_i = 1 \mid G_i = 1)\) (e.g. P(getting hit by lightening | regular golfer) = P(L|G)) since we don’t have a representative sample of golfers.

Means we can’t estimate:

• Risk difference: \(P(Y_i = 1 \mid G_i = 1)\) - \(P(Y_i = 1 \mid G_i = 0)\)
• Relative risk: \(\frac{P(Y_i = 1 \mid G_i = 1)}{P(Y_i = 1 \mid G_i = 0)}\)

We can still estimate \(P(G_i = 1 | Y_i = 1)\), which means we can estimate the odds ratio.

\[ \frac{P(L \mid G) / (1 - P(L \mid G))}{P(L \mid NG) / (1 - P(L \mid NG))} = \frac{P(G \mid L) / (1 - P(G \mid L))}{P(G \mid NL) / (1 - P(G \mid NL))} \]

Example

Estimate of \(P(G_i = 1 | Y_i = 1) = d/C_2\)
Estimate of \(P(G_i = 1 | Y_i = 0) = c/C_1\)

Estimate of Odds(G|L) = \(\frac{d/C_2}{1 - d/C_2} = \frac{d}{b}\)

Estimate of Odds(G|NL) = \(\frac{c/C_1}{1 - c/C_1} = \frac{c}{a}\)

Estimate of Odds ratio: \(\frac{ad}{bc}\)

Properties of the sample odds ratio

Sample odds ratio \[ \hat{\omega} = \frac{ad}{bc} \]

The log of this estimate is asymptotically Normal

\[ \log(\hat{\omega}) \, \dot \sim \, N\left(\log(\omega), \frac{1}{a} + \frac{1}{b} + \frac{1}{c} + \frac{1}{d}\right) \]

Log Odds ratio test

Leads to test of \(H_0: \omega = 1\) using statistic

\[ Z = \frac{\log(\hat{\omega}) - \log(1)}{\sqrt{\frac{1}{a} + \frac{1}{b} + \frac{1}{c} + \frac{1}{d}}} = \frac{\log(\hat{\omega}) }{\sqrt{\frac{1}{a} + \frac{1}{b} + \frac{1}{c} + \frac{1}{d}}} \]

Under \(H_0\) has an approximate N\((0, 1)\) distribution.

\((1 - \alpha)100\)% confidence interval for \(\log(\omega)\): \[ \log(\hat{\omega}) \pm z_{1-\alpha/2} \sqrt{\frac{1}{a} + \frac{1}{b} + \frac{1}{c} + \frac{1}{d}} \]

(Exponentiate both endpoints to get CI for \(\omega\))

Which test?

All three tests, Chi-square, Fisher’s Exact, and Log Odds ratio can be used in all sampling schemes (Multinomial, two sample Binomial and Retrospective).

Which probabilities can be estimated depends on sampling scheme.