The Citation Gap: Results of a Self-Experiment

by on 2013-08-16 in Duck- 12 Comments

Both because of the unexpected direction yesterday took, and because I haven’t worked through my thoughts about any number of pressing current events, I thought I’d write about an experiment that I’ve been engaging in with my recent academic papers. You might recall the Maliniak, Powers, and Walter paper (soon to be out with International Organization) on citations and the gender gap. As Walter reported at Political Violence @ a Glance:

…. articles written by women in international relations are cited significantly less than articles written by men. This is true even if you control for institutional affiliation, productivity, publication venue, tenure, topic, methodology and anything else you can think of. Our hunch was that this gender citation gap was due to two things: (1) women citing themselves less than men, and (2) men tending to cite other men more than women in a field dominated by men.

After the wide-ranging discussion prompted by the piece, I decided to try to increase the number of women that I cited.

The approach I took was straightforward. In the first cut, I substituted ‘obligatory generic citations’ to male authors with ones to female authors. These are the kinds of citations where the precise details of the argument don’t matter, but rather the reference genuflects toward a literature or a line of argument. Many, but not all of these, could include the lines “see, for example….” On the second cut, I focused on articles where the details of the argument mattered more, and tried to see if I could find an appropriate piece written by a woman.

The results were an improvement, but still not anything to shout about. The piece that got the most rigorous treatment saw more than a doubling of the number of references to articles, books, and chapters with at least one female author, but that only involved a shift from around 10% to 25%.

Still, I think the simple step is worth taking. After all, there’s no reason references to the importance of identity, norms, military power, trade interdependence, or whatever must be to the same male authors that we reflexively cite over and over (and over and over) again. 

Moreover, the exercise reinforced my suspicion (one widely held) that some of the “gender gap” in citations is rooted in syllabi, i.e., that our sense of whom we need to cite for a particular argument is based on whom our instructors placed on the syllabus for a particular week. This suggests that with a little bit of effort, and the downstream effects of having a higher percentage of active female scholars, the problem is likely to become less intense over time.

Print article

  • Minority CP scholar

    Interesting. Have you tried to do the same with minority scholars and scholars from outside Europe and N. America? Or is IR really as white as it appears? Not being in the field, I don’t know. But I am struck by how white this blog is (and other similar blogs like PV@G)) and how white most IR literature appears. Or is this just not a relevant issue to you and others? I guess the muted reaction to Rathburn’s racist piece on APSA in Congo versus the voluminous reaction to the jailbait piece by the same author is revealing.

  • DuckPM

    I wouldn’t describe the reaction to Rathbun’s piece as “muted”, and I’m not sure you can make a fair comparison between the reactions to the two pieces since (in my limited observations) the Duck has apparently become more prominent and institutionalized than it was in the past.

    Imputing racism based on a non-discussion of racism in a piece about gender dynamics is certainly an interesting conversational tactic. The issue you raise is a very real one, or rather a series of real ones, and I guess it’s time for me to write a main-page post about my own reactions to precisely the issues that both you and Dan raise.

  • blerg

    I think your response is a little over-defensive. Minority CP Scholar was asking an honest question (as I read it) about a big and important cognate issue. I suspect s/he’s right about minority scholars being even more marginalized than women scholars in our field, and about the fact that, to the extent we think about inclusiveness, we’re better at thinking about including women scholars than thinking about including minority scholars. I agree that the comparison between the Congo piece and yesterday’s is probably not good data, but what’s the harm in saying, “No, I really haven’t tried that with minority/global South/etc. scholars and yeah, it might reflect something about the discipline. We should do better.”

  • Minority CP scholar

    I look forward to it. To be clear (as blerg pointed out) I did not accuse anyone of racism though you seemed to see that in my brief response. Ignoring contributions by minority IR scholars could have many explanations, many of which may have nothing to do with racism. I see that possibility and hope that you do as well the next time somebody points out (as blerg already did) that the fairly systematic exclusion of non-white voices in the study of IR in the US doesn’t appear to be as much of a concern as the equally distressing exclusion of womens’ voices.

  • DuckPM

    I’ll retract my imputation of that accusation in your response. It has stimulated a rather more interesting line of argument. I am deeply convinced, as I believe you are, that the absence of minority voices is bad for the discipline not just as a profession (e.g. a place where people pick up paychecks) but also and more importantly as a scholarly enterprise (e.g. a place where we are supposed to think true thoughts about the world). The question of North American/non-North American origin, however, appears to me to be different–at least if we are talking about the origin **of ideas** and not **of people**. Social sciences seem more closely tied to the societies that birth them, and consequently noncitation to scholars outside a field seems more defensible (or less indefensible, which isn’t quite the same thing) than failure to cite scholars within a field.

  • Dan Nexon

    The reaction to the Congo post was harsh enough that Brian pulled it. But you’re right that this is way more intense.

    There are a number of things going on here, and they deserve a longer response.

    1. As far as I know, we don’t know one way or another if there’s a race/ethnicity/nationality citation gap. Data on the demographic characteristics of authors is pretty poor. You might recall that I had a post about this sometime back in which I called for feedback about how we could improve the situation, as I very much want to generate data that will help us better understand all kinds of issues. Very few people responded, either there or by email.

    That data is *much* worse for race/ethnicity than for sex, because (a) the degree of error in extrapolating race/ethnicity/national origin from names is much higher and (b) the labor/return ratio for improving that data post hoc is much worse. That, plus the demographics and politics of the discipline, help explain why the “sex gap” has gotten so much attention.

    I should note that collecting race/ethnicity data gets even harder to deal with in a cross-national setting, because categories shift a great deal. You need to generate conditional surveys, such that you can feed Americans into different answers than Nigerians, French, Germans, or Japanese. I’ve been told that ScholarOne doesn’t support this, which is creating a massive headache for our efforts to figure out how to collect this info. Note: if anyone wants to volunteer to help, send me an email.

    So the quick answer is that I did this exercise for sex because I was reacting to the aforementioned study, we don’t know if there’s a citation gap for race/ethnicity/nationality, we don’t know which race/ethnicity/minority groups might be under-cited, etc. Frankly, given how poorly represented American minorities are, for example, in international studies, my immediate reaction is that trying to cite more is nice, but amounts to a band-aid on a deep structural wound that needs to be dealt with in a much more systematic and intensive way.

    I’m not sure how you want me to address the issue of representation on the blogsphere. We’ve certainly asked people from a variety of different backgrounds. Not everyone wants to blog, or blog with us, etc.

  • DuckPM

    As you may see, it’s posted now on the main site. I intend this to be more a framework for discussion than a definitive statement, but I welcome reactions to it. You are entirely right that we need to have a conversation, and if not now, when?

  • Experimental pedant

    I am glad to hear you are doing this.

    Somewhat small nomenclature point. This isn’t really an “experiment” as none of it is random. It is a change in behavior after hearing about the fact that fewer women are cited (your knowledge of that is the intervention). Yes I’m pedantic.

  • Dan Nexon

    Let’s see if I can out-pedant the Experimental Pedant.

    “Experiment” is a broad term that covers any attempt to manipulate a variable in order to test a relationship. You’re confusing questions of proper experimental *procedure* with what defines the activity itself. So, by the standards of good experimental design, mine was a major #fail. But it was still an experiment, insofar as I wanted to see how two interventions influenced the number of papers with at least one woman author cited in my manuscripts.

  • Adrienne LeBas

    Just to clarify: Brian pulled his follow-up post to the APSA-in-Congo piece, but the original post is still up on the Duck. I remember the comments thread on the follow-up being pretty long.

  • Jmulich

    This is a very interesting observation, Dan. Having a foot in each disciplinary camp of IR and History, I’m surprised by how much it seems that IR is lacking behind. There are definitely still major gender issues in History as a discipline, but the citation gap doesn’t seem anywhere near as bad is it is in IR. As pure anecdote, I tried counting the citations to secondary sources in my latest article/project, and it came out to just around 40% female scholars. Granted, this might be a result of my specific subfield (global history), which is somewhat younger and more in flux than a field like American history, but I still think there is a significant difference in the gender imbalance between History, as a discipline, and IR.

  • zenpundit

    “Or is IR really as white as it appears?”

    Almost certainly, yes – as are most fields academic outside of gender and ethnic studies programs and education. Engineering may be more diverse than IR

    We must also note the vast size of American higher education as an employer relative to our population compared to similar advanced nations is a factor. Many jobs are at third and fourth tier universities or community colleges but they add up.

    For developing and MIC nations – many of which feature authoritarian regimes – will they emphasize graduating more scientists, engineers, mathematicians, medical doctors, computer scientists, teachers, agronomists etc. with their scarce resources or more political scientists who are apt to criticize the government and analyze what it is doing?