{"id":5800,"date":"2012-11-26T07:49:25","date_gmt":"2012-11-26T06:49:25","guid":{"rendered":"http:\/\/idiolect.org.uk\/notes\/?p=5800"},"modified":"2012-12-06T22:58:53","modified_gmt":"2012-12-06T21:58:53","slug":"testing-bootstrapping","status":"publish","type":"post","link":"https:\/\/idiolect.org.uk\/notes\/2012\/11\/26\/testing-bootstrapping\/","title":{"rendered":"Testing bootstrapping"},"content":{"rendered":"

Update: This post used an incorrect implementation of the bootstrap, so the conclusions don’t hold. See this correction<\/a><\/strong><\/p>\n

This surprised me. I decided to try out bootstrapping as a method of testing if two sets of numbers are drawn from different distributions. I did this by generating sets of numbers of size m from two ex-gaussian distributions which are identical except for a fixed difference, d<\/p>\n

\r\n    s1=randn(1,m)+exp(randn(1,m));\r\n    s2=randn(1,m)+exp(randn(1,m))+d;\r\n<\/pre>\n
All code is matlab. Sorry about that.<\/p>\n
Then, for each pair of numbers I apply a series of different tests for if the distributions are different.
\n1. Standard t-test (0.05 significance level)
\n2. Is the mean(s1)\n3. Bootstrapping using mean as the test statistic (0.05 significance level)
\n4. Bootstrapping using the median as the test statistic (0.05 significance level)<\/p>\n
I used Ione Fine’s pages on bootstrapping as a guide<\/a>. The bootstrapping code is:<\/p>\n
\r\nfunction H=bootstrap(s1,s2,samples,alpha,method)\r\n\r\nfor i=1:samples\r\n    \r\n    boot1=s1(ceil(rand(1,length(s1))*length(s1)));\r\n    boot2=s2(ceil(rand(1,length(s2))*length(s2)));\r\n    \r\n    if method==1\r\n        a(i)=mean(boot1)-mean(boot2);\r\n    else\r\n        a(i)=median(boot1)-median(boot2);    \r\n    end\r\n    \r\nend\r\n\r\nCI=prctile(a,[100*alpha\/2,100*(1-alpha\/2)]);\r\n\r\nH = CI(1)>0 | CI(2)<0;\r\n<\/pre>\n
I do that 5000 times for each difference, d, and each sample size, m. Then I take the average answer from each test (where 1 is 'conclude there distributions are different' and 0 is 'don't conclude the distributions are different'). For the case where d > 0 this gives you a hit rate, the likelihood that the test will tell you there is a difference when there is a difference. For d = 0.5 you get a difference that most of the tests can detect the majority of the time as long as the sample is more than 50. For the case where d = 0, you can calculate the false alarm rate for each test (at each sample size). <\/p>\n
From these you can calculate d-prime as a standard index of sensitivity and plot the result. Sttest, Smean, Sbootstrap and Sbootstrap2 are matrices which hold the likelihood of the four tests giving a positive answer for each sample size (columns) for two differences, 0 and 0.5 (the rows):<\/p>\n
\r\nfigure(1);clf\r\nplot(measures,norminv(Sttest(2,:))-norminv(Sttest(1,:),0,1),'k')\r\nhold on\r\nplot(measures,norminv(Smean(2,:))-norminv(Smean(1,:)),'r')\r\n%plot(measures,norminv(Smedian(2,:))-norminv(Smedian(1,:)),'c--')\r\nplot(measures,norminv(Sbootstrap(2,:))-norminv(Sbootstrap(1,:)),'m')\r\nplot(measures,norminv(Sbootstrap2(2,:))-norminv(Sbootstrap2(1,:)),'g')\r\nxlabel('Sample size')\r\nylabel('sensitivity - d prime')\r\nlegend('T-test','mean','bstrap-mean','bstrap-median')\r\n<\/pre>\n
Here is the result (click for larger):<\/p>\n
\"\"<\/a><\/p>\n
What surprised me was:<\/p>\n