Economists Do It With Models

Warning: “graphic” content…

Bookmark and Share
Small Steps In The War Against Publication Bias…

November 19th, 2013 · 4 Comments
Behavioral Econ · Fun With Data

As a behavioral economist, this is probably the the most exciting thing I will read all week. From the American Economic Association:

Dear AEA member:

The AEA has launched a new site to register randomized control trials (RCTs). The AEA encourages all investigators to register new and existing RCTs. Registration is entirely voluntary and is not currently linked to or required for submission and publication in the AEA journals.

The site is available at https://www.socialscienceregistry.org

On this site, you can register your forthcoming, ongoing, or even completed RCTs, with as little or as many details as you wish. The site will also permit you to store and make publicly-available additional information on your RCTs (reports, articles, data, and code). We believe that this will prove to be a very valuable resource for investigators to share their work and the site will be widely used by those who wish to find out about on-going and completed studies.

The registry is characterized by:

1) Simplicity and flexibility: Registering a trial is straightforward with only a minimal number of required fields. There is considerable flexibility to provide additional material at the time of registration or at any point in the life of the study. Materials can also be hidden from public view until completion of the study, or be made accessible only with the permission of the PI.

2) Adjustability and memory: Any registry entry can be amended by the PI at any point, but the registry keeps track of all versions.

3) Ability to work as a research portal for your RCTs: The registry can serve as an access point for collaborators, other scholars, students, and the general public providing links to data sets, survey instruments, experimental findings, and experimental protocols.

To register a trial, the PI simply needs to enter the following information: PI name, project title, study location, project status, keyword(s), abstract, trial start and end dates, intervention start and end dates, proposed outcome(s), experimental design, whether the treatment is clustered, planned number of clusters, planned number of observations, and IRB information. Optional fields allowing the PI to customize and enhance the information made available include details on sponsors and partners, survey instruments, an analysis plan, and other supporting documents. Help is available if the PI encounters any problem.

The AEA registry system will provide the PI with reminders to update the registration of an RCT at appropriate points in the trial’s lifecycle. For example, the submitted end date will trigger an email asking the PI to enter post-trial information. If the trial has been extended, the PI can update the trial with the new end date.

We encourage you to explore the registry and to register your RCTs.

The committee on the registry for social experiments
Larry Katz (chair)
Esther Duflo
Pinelopi Goldberg
Duncan Thomas

So why is this important? I think this pretty much sums it up:

I’ve written about this before- in general, a finding is only considered statistically significant if there is less than a 5 percent chance that the observed result would have happened by random chance. (Hence the use of the 0.05 value in the cartoon.) But think this through a bit- if something has a 5 percent chance of happening randomly, then, on average, that result will be observed one time out of 20 even if there is nothing systematic going on. Therefore, it doesn’t mean a whole lot to see one result that has less than a 5 percent probability of occurring by random chance unless you know that there aren’t a whole bunch of studies out there that tried the same thing and didn’t get the observed result.

This registry is a fantastic development since, if used properly, it will keep track of all of those non-result studies that would otherwise be hidden in researchers’ desk drawers or on their hard drives and therefore be unobservable to someone trying to determine the validity of the research that is actually published. I say “if used properly,” since it’s only helpful to the degree that we can be confident that everyone is actually registering their experiments. Given this, I’m somewhat surprised that the architects of the system didn’t make pre-experiment registration a prerequisite for publishing in an AEA journal, but I’d be willing to bet that that will be coming eventually. Baby steps, I suppose.

If you still want to think more about publication bias and the reliability of the scientific results that you see, check out the following:

(You should know that I spent about 30 minutes figuring out how to disable the annoying autoplay feature. You’re welcome. You can also see the video directly here, especially since I can’t seem to get the sizing to work properly…but consider yourself forewarned about the autoplay issue.) Granted, the conclusions in the video depend heavily on the number used for the proportion of hypotheses that are actually true as well as the assumed rate of false negatives, but the concept is still worth pondering.

Tags: Behavioral Econ · Fun With Data

4 responses so far ↓

Leave a Comment