Economists Do It With Models

Warning: “graphic” content…

Bookmark and Share
Small Steps In The War Against Publication Bias…

November 19th, 2013 · 10 Comments
Behavioral Econ · Fun With Data

As a behavioral economist, this is probably the the most exciting thing I will read all week. From the American Economic Association:

Dear AEA member:

The AEA has launched a new site to register randomized control trials (RCTs). The AEA encourages all investigators to register new and existing RCTs. Registration is entirely voluntary and is not currently linked to or required for submission and publication in the AEA journals.

The site is available at

On this site, you can register your forthcoming, ongoing, or even completed RCTs, with as little or as many details as you wish. The site will also permit you to store and make publicly-available additional information on your RCTs (reports, articles, data, and code). We believe that this will prove to be a very valuable resource for investigators to share their work and the site will be widely used by those who wish to find out about on-going and completed studies.

The registry is characterized by:

1) Simplicity and flexibility: Registering a trial is straightforward with only a minimal number of required fields. There is considerable flexibility to provide additional material at the time of registration or at any point in the life of the study. Materials can also be hidden from public view until completion of the study, or be made accessible only with the permission of the PI.

2) Adjustability and memory: Any registry entry can be amended by the PI at any point, but the registry keeps track of all versions.

3) Ability to work as a research portal for your RCTs: The registry can serve as an access point for collaborators, other scholars, students, and the general public providing links to data sets, survey instruments, experimental findings, and experimental protocols.

To register a trial, the PI simply needs to enter the following information: PI name, project title, study location, project status, keyword(s), abstract, trial start and end dates, intervention start and end dates, proposed outcome(s), experimental design, whether the treatment is clustered, planned number of clusters, planned number of observations, and IRB information. Optional fields allowing the PI to customize and enhance the information made available include details on sponsors and partners, survey instruments, an analysis plan, and other supporting documents. Help is available if the PI encounters any problem.

The AEA registry system will provide the PI with reminders to update the registration of an RCT at appropriate points in the trial’s lifecycle. For example, the submitted end date will trigger an email asking the PI to enter post-trial information. If the trial has been extended, the PI can update the trial with the new end date.

We encourage you to explore the registry and to register your RCTs.

The committee on the registry for social experiments
Larry Katz (chair)
Esther Duflo
Pinelopi Goldberg
Duncan Thomas

So why is this important? I think this pretty much sums it up:

I’ve written about this before– in general, a finding is only considered statistically significant if there is less than a 5 percent chance that the observed result would have happened by random chance. (Hence the use of the 0.05 value in the cartoon.) But think this through a bit- if something has a 5 percent chance of happening randomly, then, on average, that result will be observed one time out of 20 even if there is nothing systematic going on. Therefore, it doesn’t mean a whole lot to see one result that has less than a 5 percent probability of occurring by random chance unless you know that there aren’t a whole bunch of studies out there that tried the same thing and didn’t get the observed result.

This registry is a fantastic development since, if used properly, it will keep track of all of those non-result studies that would otherwise be hidden in researchers’ desk drawers or on their hard drives and therefore be unobservable to someone trying to determine the validity of the research that is actually published. I say “if used properly,” since it’s only helpful to the degree that we can be confident that everyone is actually registering their experiments. Given this, I’m somewhat surprised that the architects of the system didn’t make pre-experiment registration a prerequisite for publishing in an AEA journal, but I’d be willing to bet that that will be coming eventually. Baby steps, I suppose.

If you still want to think more about publication bias and the reliability of the scientific results that you see, check out the following:

(You should know that I spent about 30 minutes figuring out how to disable the annoying autoplay feature. You’re welcome. You can also see the video directly here, especially since I can’t seem to get the sizing to work properly…but consider yourself forewarned about the autoplay issue.) Granted, the conclusions in the video depend heavily on the number used for the proportion of hypotheses that are actually true as well as the assumed rate of false negatives, but the concept is still worth pondering.

Tags: Behavioral Econ · Fun With Data

10 responses so far ↓

  • 1 Chris // Nov 19, 2013 at 8:05 pm

    A network good then.

  • 2 bdbd // Nov 20, 2013 at 11:41 am

    When will there be some misbehavioral economists?

  • 3 Small Steps In The War Against Publication Bias | economists do it with models | Utopia - you are standing in it! // Apr 26, 2014 at 3:38 am

    […] via Small Steps In The War Against Publication Bias…. […]

  • 4 Jacinto // Oct 25, 2014 at 2:01 pm

    I think this is one of the most vital info for me. And i am glad reading yor article.
    But want to remark on some general things, The site style is great, the articles is really
    excellent : D. Good job, cheers

  • 5 modern fashion // Dec 28, 2014 at 2:02 pm

    Fashion in simple terms is just a mode of self
    expression. Tribal hair accessories are equally important in your tribal ethnic looks.

    The skirts are combined with tight-fitting tops, T-shirts, or long-sleeved blouses — often glittering with sparkly appliqué,
    crystals and sequins.

  • 6 firearms training course // Jan 15, 2016 at 3:31 pm

    As a gun proprietor, I’m fantastic with background checks and even think that perhaps a class
    on firearms be advisable for all owners.

  • 7 gaming center houston // Feb 8, 2016 at 11:14 am

    On a Mac, the Sport Heart app is included with Mac OS X.
    As of OS X 10.eleven El Capitan, this app is protected with System Integrity
    Protection and can’t usually be deleted.

  • 8 lucky number slevin trailer music // Feb 17, 2016 at 11:36 am

    The trick works as a result of there are literally only eight numbers someone
    is allowed to select from — thirteen, 15, 17, 19, 31,
    35, 37 and 39. And by mentioning the number 13 within the preamble, you set them
    off that one.

  • 9 hip hop songs 2015 // Mar 6, 2016 at 11:36 pm

    Charge songs to hear them roughly usually, and even mix two or
    more of your favourite channels together.

  • 10 bike parts shop // Mar 26, 2016 at 4:10 pm

    Big strikes and massive mountains are often solely accessible by
    long, rewarding climbs; after all of the onerous work, all mountain bikes
    come into their very own when the path points down and issues get scary.

Leave a Comment