Economists Do It With Models

Warning: “graphic” content…

Bookmark and Share
Small Steps In The War Against Publication Bias…

November 19th, 2013 · 24 Comments
Behavioral Econ · Fun With Data

As a behavioral economist, this is probably the the most exciting thing I will read all week. From the American Economic Association:

Dear AEA member:

The AEA has launched a new site to register randomized control trials (RCTs). The AEA encourages all investigators to register new and existing RCTs. Registration is entirely voluntary and is not currently linked to or required for submission and publication in the AEA journals.

The site is available at

On this site, you can register your forthcoming, ongoing, or even completed RCTs, with as little or as many details as you wish. The site will also permit you to store and make publicly-available additional information on your RCTs (reports, articles, data, and code). We believe that this will prove to be a very valuable resource for investigators to share their work and the site will be widely used by those who wish to find out about on-going and completed studies.

The registry is characterized by:

1) Simplicity and flexibility: Registering a trial is straightforward with only a minimal number of required fields. There is considerable flexibility to provide additional material at the time of registration or at any point in the life of the study. Materials can also be hidden from public view until completion of the study, or be made accessible only with the permission of the PI.

2) Adjustability and memory: Any registry entry can be amended by the PI at any point, but the registry keeps track of all versions.

3) Ability to work as a research portal for your RCTs: The registry can serve as an access point for collaborators, other scholars, students, and the general public providing links to data sets, survey instruments, experimental findings, and experimental protocols.

To register a trial, the PI simply needs to enter the following information: PI name, project title, study location, project status, keyword(s), abstract, trial start and end dates, intervention start and end dates, proposed outcome(s), experimental design, whether the treatment is clustered, planned number of clusters, planned number of observations, and IRB information. Optional fields allowing the PI to customize and enhance the information made available include details on sponsors and partners, survey instruments, an analysis plan, and other supporting documents. Help is available if the PI encounters any problem.

The AEA registry system will provide the PI with reminders to update the registration of an RCT at appropriate points in the trial’s lifecycle. For example, the submitted end date will trigger an email asking the PI to enter post-trial information. If the trial has been extended, the PI can update the trial with the new end date.

We encourage you to explore the registry and to register your RCTs.

The committee on the registry for social experiments
Larry Katz (chair)
Esther Duflo
Pinelopi Goldberg
Duncan Thomas

So why is this important? I think this pretty much sums it up:

I’ve written about this before– in general, a finding is only considered statistically significant if there is less than a 5 percent chance that the observed result would have happened by random chance. (Hence the use of the 0.05 value in the cartoon.) But think this through a bit- if something has a 5 percent chance of happening randomly, then, on average, that result will be observed one time out of 20 even if there is nothing systematic going on. Therefore, it doesn’t mean a whole lot to see one result that has less than a 5 percent probability of occurring by random chance unless you know that there aren’t a whole bunch of studies out there that tried the same thing and didn’t get the observed result.

This registry is a fantastic development since, if used properly, it will keep track of all of those non-result studies that would otherwise be hidden in researchers’ desk drawers or on their hard drives and therefore be unobservable to someone trying to determine the validity of the research that is actually published. I say “if used properly,” since it’s only helpful to the degree that we can be confident that everyone is actually registering their experiments. Given this, I’m somewhat surprised that the architects of the system didn’t make pre-experiment registration a prerequisite for publishing in an AEA journal, but I’d be willing to bet that that will be coming eventually. Baby steps, I suppose.

If you still want to think more about publication bias and the reliability of the scientific results that you see, check out the following:

(You should know that I spent about 30 minutes figuring out how to disable the annoying autoplay feature. You’re welcome. You can also see the video directly here, especially since I can’t seem to get the sizing to work properly…but consider yourself forewarned about the autoplay issue.) Granted, the conclusions in the video depend heavily on the number used for the proportion of hypotheses that are actually true as well as the assumed rate of false negatives, but the concept is still worth pondering.

Tags: Behavioral Econ · Fun With Data

24 responses so far ↓

  • 1 Chris // Nov 19, 2013 at 8:05 pm

    A network good then.

  • 2 bdbd // Nov 20, 2013 at 11:41 am

    When will there be some misbehavioral economists?

  • 3 Small Steps In The War Against Publication Bias | economists do it with models | Utopia - you are standing in it! // Apr 26, 2014 at 3:38 am

    […] via Small Steps In The War Against Publication Bias…. […]

  • 4 Jacinto // Oct 25, 2014 at 2:01 pm

    I think this is one of the most vital info for me. And i am glad reading yor article.
    But want to remark on some general things, The site style is great, the articles is really
    excellent : D. Good job, cheers

  • 5 modern fashion // Dec 28, 2014 at 2:02 pm

    Fashion in simple terms is just a mode of self
    expression. Tribal hair accessories are equally important in your tribal ethnic looks.

    The skirts are combined with tight-fitting tops, T-shirts, or long-sleeved blouses — often glittering with sparkly appliqué,
    crystals and sequins.

  • 6 firearms training course // Jan 15, 2016 at 3:31 pm

    As a gun proprietor, I’m fantastic with background checks and even think that perhaps a class
    on firearms be advisable for all owners.

  • 7 gaming center houston // Feb 8, 2016 at 11:14 am

    On a Mac, the Sport Heart app is included with Mac OS X.
    As of OS X 10.eleven El Capitan, this app is protected with System Integrity
    Protection and can’t usually be deleted.

  • 8 lucky number slevin trailer music // Feb 17, 2016 at 11:36 am

    The trick works as a result of there are literally only eight numbers someone
    is allowed to select from — thirteen, 15, 17, 19, 31,
    35, 37 and 39. And by mentioning the number 13 within the preamble, you set them
    off that one.

  • 9 hip hop songs 2015 // Mar 6, 2016 at 11:36 pm

    Charge songs to hear them roughly usually, and even mix two or
    more of your favourite channels together.

  • 10 bike parts shop // Mar 26, 2016 at 4:10 pm

    Big strikes and massive mountains are often solely accessible by
    long, rewarding climbs; after all of the onerous work, all mountain bikes
    come into their very own when the path points down and issues get scary.

  • 11 work lyrics meaning // May 8, 2016 at 2:00 am

    This could contain an appointment at an area Work and Income service
    centre to complete the registration course of.

  • 12 new music videos mtv // May 18, 2016 at 12:16 am

    Debut performance of the complete works from their new album, Bonjour, to be released by Cantaloupe Music.

  • 13 marketing mix product // May 20, 2016 at 1:41 pm

    There are many steps to marketing after getting established a
    product or service that satisfies and need or want.

  • 14 data centers delivered // Jul 2, 2016 at 3:27 pm

    It solely has a 12-megapixel digital camera, for instance, no expandable storage and a fairly meager
    1,860mAh battery.

  • 15 dekoratyvinis tinkas fasadui // Jul 5, 2016 at 11:35 am

    Pakalnutė dėl puošnių žiedų ir nuostabaus aromato,
    auginama kaip dekoratyvinis augalas darželiuose,

  • 16 babies r us hours westbury // Jul 17, 2016 at 2:50 am

    Having docs that will perform the orders of the parents to have
    a designer child with out telling them the full
    penalties of the actions adjustments the values and the duties
    of a doctor and this is no completely different from a salesman attempting to sell
    a product to a couple.

  • 17 baby fashionista boutique // Jul 24, 2016 at 1:39 pm

    Government bodies, analysis establishments and couples considering PGD ought to take these
    values into consideration earlier than creating a designer child.

  • 18 income property show fake // Jul 28, 2016 at 9:50 am

    Your tax return might be scanned many instances before
    you file to ensure you’ll not miss any invaluable tax credit or deductions.

  • 19 presentes criativos para homens namorados // Jul 31, 2016 at 6:43 pm

    Comumente, utensílios para futebol costumam agradar, bem como jogos
    para videogames também surpreendem.

  • 20 c3320254410348046768 // Aug 4, 2016 at 5:37 pm

    This listing is what we take into account to
    be the best vegetarian and vegan restaurants in New York City.

  • 21 animation maker no download // Aug 25, 2016 at 6:55 pm

    Developed in partnership with Hollywood’s largest studios, the Walmart Entertainment Facebook App will enable Walmart’s
    greater than 26 million Facebook fans to read and submit child pleasant movie opinions,
    watch exclusive behind-the-scenes film content material and even vote on what
    films the retailer will characteristic – all the things from limited releases to Walmart $5 film
    bin affords.

  • 22 logo generator png // Aug 28, 2016 at 10:35 pm

    Don’t compromise a transparent graphic emblem by applying special effects comparable to drop shadows, bevelling or

  • 23 tiket bus lorena online // Sep 17, 2016 at 3:38 am

    Aha..mmg tak boleh reserving online…name no ni :
    020-835 45816 (No China) Address : Clivia @ Jun Zi Lan Hotel,
    Number 823, The North of Jiefang Road, Guangzhou China.

  • 24 pl/sql training in pune // Dec 22, 2016 at 4:38 am


Leave a Comment