{"id":5821,"date":"2012-11-30T08:53:00","date_gmt":"2012-11-30T07:53:00","guid":{"rendered":"http:\/\/idiolect.org.uk\/notes\/?p=5821"},"modified":"2012-12-06T22:59:20","modified_gmt":"2012-12-06T21:59:20","slug":"bootstrap-update","status":"publish","type":"post","link":"https:\/\/idiolect.org.uk\/notes\/2012\/11\/30\/bootstrap-update\/","title":{"rendered":"Bootstrap update"},"content":{"rendered":"

Update: This post used an incorrect implementation of the bootstrap, so the conclusions don’t hold. See this correction<\/a><\/strong><\/p>\n

Mike<\/a> suggested that I alter the variance of the underlying distibutions. This makes total sense, since it matches what we are usually trying to do in psychological research – detect a small difference in a lot of noise. So I made the underlying distibutions look a lot like reaction time distributions, with a 30ms difference between them. The code is<\/p>\n

\r\n    t0=200;\r\n    s1=t0+25*(randn(1,m)+exp(randn(1,m)));\r\n    s2=t0+25*(randn(1,m)+exp(randn(1,m)))+d;\r\n<\/pre>\n
Where m is the sample size, and d is either 0 or 30. For a very large sample, the distributions look like this:<\/p>\n
\"\"<\/a><\/p>\n
After a discussion with Jim<\/a> I looked at the hit rate and false alarm rate separately. For the simple comparison of means, the false alarm rate stays around 0.5 (as you’d predict). For the other tests it drops to about 0.05. The simple comparison of means is so sensitive to a true difference, however, that the dprime can still be superior to that of the other tests. Which suggests dprime is not a good summary statistic to me, rather than that we should do testing simply by comparing the sample means.<\/p>\n
So I rerun the procedure I described before<\/a>, but with higher variance on the underlying samples. <\/p>\n
\"\"<\/a><\/p>\n
The results are very similar. The bootstrap using the mean as the test statistic is worse than the t-test. The bootstrap using the median is clear superior. This surprises me. I had been told that the bootstrap was superior for nonparametric distributions. In this case it seems as if using the mean as a test statistic eliminates the potential superiority of bootstrapping.<\/p>\n
This is still a work in progress, so I will investigate further and may have to update this conclusion as the story evolves.<\/p>\n","protected":false},"excerpt":{"rendered":"
Update: This post used an incorrect implementation of the bootstrap, so the conclusions don’t hold. See this correction Mike suggested that I alter the variance of the underlying distibutions. This makes total sense, since it matches what we are usually trying to do in psychological research – detect a small difference in a lot of […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[5,9],"tags":[],"class_list":["post-5821","post","type-post","status-publish","format-standard","hentry","category-psychology","category-science"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5KQtW-1vT","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/idiolect.org.uk\/notes\/wp-json\/wp\/v2\/posts\/5821"}],"collection":[{"href":"https:\/\/idiolect.org.uk\/notes\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/idiolect.org.uk\/notes\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/idiolect.org.uk\/notes\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/idiolect.org.uk\/notes\/wp-json\/wp\/v2\/comments?post=5821"}],"version-history":[{"count":4,"href":"https:\/\/idiolect.org.uk\/notes\/wp-json\/wp\/v2\/posts\/5821\/revisions"}],"predecessor-version":[{"id":5845,"href":"https:\/\/idiolect.org.uk\/notes\/wp-json\/wp\/v2\/posts\/5821\/revisions\/5845"}],"wp:attachment":[{"href":"https:\/\/idiolect.org.uk\/notes\/wp-json\/wp\/v2\/media?parent=5821"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/idiolect.org.uk\/notes\/wp-json\/wp\/v2\/categories?post=5821"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/idiolect.org.uk\/notes\/wp-json\/wp\/v2\/tags?post=5821"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}