Fisher's Exact test and Log Odds test ST551 Lecture 22

Fisher’s Exact test

For 2×2 tables

Setting: two independent samples

Y1,,Yn i.i.d from Bernoulli(pY)
X1,,Xm i.i.d from Bernoulli(pX)

Null hypothesis: pY=pX Homegeneity of proportions

Test statistic: Probability of observed table conditional on margins

p-value: Sum of probability of all tables as or more extreme than observed table.

Data from last time

##        ate_breakfast
## cat_dog no yes
##    cats  6   9
##    dogs  6  14

A smaller (in sample size) example

##        ate_breakfast
## cat_dog no yes
##    cats  4   4
##    dogs  3   4

A smaller (in sample size) example with margins

##        ate_breakfast
## cat_dog no yes Sum
##    cats  4   4   8
##    dogs  3   4   7
##    Sum   7   8  15

Under null, there is no difference in the probability of eating breakfast between people who prefer cats and people who prefer dogs, what is the probability of observing this exact table, conditioning on the margins?

Working out the test statistic

Under the null pY=pX=p.

Conditional on row sums R1=a+b and R2=c+d:

n12Binomial(R1,p)
n22Binomial(R2,p)

But we also need n12+n22=C2.

I.e. What is P(n12=bn12+n22=C2)?

Turns out to be: =(a+bb)(c+dd)(Nb+d) a.k.a Hypergeometric Distribution

Derivation

P(n12=b)=(a+bb)pb(1p)a

P(n22=d)=(c+dd)pd(1p)c

P(n12+n22=b+d)=(Nb+d)pb+d(1p)a+c

P(n12=bn12+n22=b+d)=P(n12=b,n12+n22=b+d)P(n12+n22=b+d)=P(n12=b,n22=d)P(n12+n22=b+d)=(a+bb)pb(1p)a(c+dd)pd(1p)c(nb+d)pb+d(1p)a+c=(a+bb)(c+dd)(Nb+d)

Example: Test statistic

##        ate_breakfast
## cat_dog no yes Sum
##    cats  4   4   8
##    dogs  3   4   7
##    Sum   7   8  15

Under null, probability of seeing this table, given margins:

Prob=(84)(74)(158)=0.38

p-values

More extreme? Depends on alternative:

  • pY>pX, greater n12
  • pY<pX, smaller n12
  • pYpX, less likely tables

Example: p-values lower alternative

HA:pY<pX

More extreme tables would have n12= 3, 2, 1 or 0

Your turn: Complete the table if n12=3:

No Yes
Cats 8
Dogs 7
7 8 15

Prob = (a+bb)(c+dd)(nb+d)=0.18

Example: p-values lower alternative

HA:pY<pX

No Yes
Cats
Dogs
No Yes
Cats 6 2
Dogs 1 6
No Yes
Cats 7 1
Dogs 0 7

p-value =0.381+0.183+0.03+0.001=0.595

p-values lower alternative

Example: p-values greater alternative

HA:pY>pX

More extreme tables would have n12= 5, 6, 7 or 8

Your turn: What are the more extreme tables?

p-value =0.381+0.305+0.091+0.009+0=0.786

Example: p-values two-sided alternative

Which tables are less likely than that observed?

In R

Careful with direction (rows permuted in R)

x <- xtabs(~ cat_dog + ate_breakfast, data = class_data)
fisher.test(x, alternative = "greater")
## 
##  Fisher's Exact Test for Count Data
## 
## data:  x
## p-value = 0.397
## alternative hypothesis: true odds ratio is greater than 1
## 95 percent confidence interval:
##  0.3796782       Inf
## sample estimates:
## odds ratio 
##   1.535689

Sampling for binomial proportions

  • Multinomial Obtain a sample of N units, and cross classify according to the grouping variable G and the binary response variable Y.

    • Can estimate all probabilities, P(Yi=1), P(Gi=1), P(Yi=1|Gi=0) etc.
  • Two-sample Obtain two samples, one of size n from group 1 (Gi=0), and one of size m from group 2 (Gi=1).

    • Can’t estimate P(Gi=1) or P(Yi=1) but can estimate P(Yi=1|Gi=0).

Retrospective studies

Sometimes, particularly if an outcome is very rare, it’s hard to observe any successes Yi=1, with Multinomial or Binomial sampling.

E.g.
Response variable Y= struck by lightning (Yes/No)
Grouping variable G= regular golfer (Yes/No)

Lightning strikes are very rare events, so even with rather large samples we may not observe anybody who has been struck.

In a retrospective sample we get s units where Yi=1 and r units where Yi=0 then classify them to the groups Gi.

Example

Yi=0 Yi=1 Total
Gi=0 a b R1
Gi=1 c d R2
Total C1=r C2=s N=r+s

Retrospective studies

We can no longer estimate P(Yi=1Gi=1) (e.g. P(getting hit by lightening | regular golfer) = P(L|G)) since we don’t have a representative sample of golfers.

Means we can’t estimate:

  • Risk difference: P(Yi=1Gi=1) - P(Yi=1Gi=0)
  • Relative risk: P(Yi=1Gi=1)P(Yi=1Gi=0)

We can still estimate P(Gi=1|Yi=1), which means we can estimate the odds ratio.

P(LG)/(1P(LG))P(LNG)/(1P(LNG))=P(GL)/(1P(GL))P(GNL)/(1P(GNL))

Example

Yi=0 Yi=1 Total
Gi=0 a b R1
Gi=1 c d R2
Total C1=r C2=s N=r+s

Estimate of P(Gi=1|Yi=1)=d/C2
Estimate of P(Gi=1|Yi=0)=c/C1

Estimate of Odds(G|L) = d/C21d/C2=db

Estimate of Odds(G|NL) = c/C11c/C1=ca

Estimate of Odds ratio: adbc

Properties of the sample odds ratio

Sample odds ratio ω^=adbc

The log of this estimate is asymptotically Normal

log(ω^)˙N(log(ω),1a+1b+1c+1d)

Log Odds ratio test

Leads to test of H0:ω=1 using statistic

Z=log(ω^)log(1)1a+1b+1c+1d=log(ω^)1a+1b+1c+1d

Under H0 has an approximate N(0,1) distribution.

(1α)100% confidence interval for log(ω): log(ω^)±z1α/21a+1b+1c+1d

(Exponentiate both endpoints to get CI for ω)

Which test?

All three tests, Chi-square, Fisher’s Exact, and Log Odds ratio can be used in all sampling schemes (Multinomial, two sample Binomial and Retrospective).

Which probabilities can be estimated depends on sampling scheme.