I was recently asked to review some grant applications for several social science research projects of relevance to national security. The applications are interesting and I’ve enjoyed seeing the proposal process from a new perspective. However, I’ve noted a common theme regarding the handling of data that I think is intuitively problematic with respect to theory building given my own interest in epistemology and the robustness of any claims that can be made from our research.
The problem that interests me is the coding of empirical data. I see great emphasis on developing common coding schemes and ensuring intercoder reliability—both worthy and necessary goals. However, as someone who is interested in the robustness of any claims about social systems, I think the potential gains of alternative points of view regarding common cases get lost. During my assessments of the research strategies, I can’t help but wonder what might occur if a proposer made no corrections for intercoder diversity, and generated multiple coded data sets based on the same empirical sources from different coders or coding processes, or to get even more diversity different sources regarding the same cases?
In the context of individual research projects, it makes sense that the greatest gains are to be made from growing data sets and analyzing them using one’s selected methodology in an effort to generate and test a particular theory. However, taking a step back from the singular project, I think there are greater gains to be found by identifying how robust particular theories or arguments are about a phenomenon given multiple interpretations of the data. Thus, I’m left wondering what would come out of a research design with alternative coders and coding schemes were employed against the same data or cases, and then, following the same process of modeling and inference from the structured data, the assessments produced by these different data sets were compared? What kinds of relationships would prove invariant or robust to whims of coding or alternative interpretations, and what would be contingent?
As far as I know, I’ve only seen this point raised one time in Ian Lustick’s piece about multiple versions of history and the problem of selection bias in political science, and I know that this is a common problem in meta-analysis, although I don’t think the approach is based on comparing the consequences of disparate inputs even though consistency/inconsistency among outputs are assessed. I’m thinking this would be something worth revisiting if it hasn’t been addressed and operationalized since.
Great points, Aaron. This is a good intersection point between the budding quantitatively-driven computational humanities and the much longer traditions of social science – and perhaps one thing we might be able to contribute to you guys rather than how it so often happens in reverse. I generally try not to plug my own work, but you might be interested in a piece I co-authored recently that touches on just this subject, if perhaps framed differently, but to very much the same effect and purpose as you describe here.
http://llc.oxfordjournals.org/content/early/2012/05/25/llc.fqs015.full (section on data choice)
Also, thank you for pointing me to that issue of selection bias; it’s something philosophers of history have been struggling against for some time now, as well, just one meta-level down; how does the intrinsic selection bias of the historical process affect the availability of our primary sources?