Sociologist, Scientist? Toward Transparency, Accountability, and a Sharing Culture

With the help of the designer Brigid Barrett, I have a new website at philipncohen.com, and a redesigned blog to match (which you’re looking at now). We decided on the tagline, “Sociologist / Demographer” for the homepage photo. It’s true I am those two things, but I also like how they modify each other, a type of sociologist and a type of demographer. First some reflections, then a little data.

I shared the website on Twitter, and wrote this in a thread:

Having “sociologist” attached to your name is not going to signal scientific rigor to the public in the way that other discipline labels might (like, I think, “demographer”). A lot of sociologists, as shown by their behavior, are fine with that. Your individual behavior as a researcher can shape the impression you make, but it will not change the way the discipline is seen. Until the discipline — especially our associations but also our departments — adopts (and communicates) scientific practices, that’s how it will be. As an association, ASA has shown little interest in this, and seems unlikely to soon.

A substantial portion of sociologists rejects the norms of science. Others are afraid that adopting them will make their work “less than” within the discipline’s hierarchy. For those of us concerned about this, the practices of science are crucial: openness, transparency, reproducibility. We need to find ways at the sub-discipline level to adopt and communicate these values and build trust in our work. Building that trust may require getting certain publics to see beyond the word “sociologist,” rather than just see value in it. They will see our open practices, our shared data and code, our ability to admit mistakes, embrace uncertainty, and entertain alternative explanations.

There are other sources of trust. For example, taking positions on social issues or politics is also a way of building trust with like-minded audiences. These are important for some sociologists, and truly valuable, but they’re different from science. Maybe unreasonably, I want both. I want some people to give my work a hearing because I take antiracist or feminist positions in my public work, for example. And also because I practice science in my research, with the vulnerability and accountability that implies. Some people would say my public political pronouncements undermine not just my science, but the reputation of the discipline as a whole. I can’t prove they’re wrong. But I think the roles of citizen and scholar are ultimately compatible. Having a home in a discipline that embraced science and better communicated its value would help. A scientific brand, seal of approval, badges, etc., would help prevent my outspokenness from undermining my scientific reputation.

One reply I got, confirming my perception, was, “this pretence of natural science needs to be resisted not indulged.” Another wrote: “As a sociologist and an ethnographer ‘reproducibility’ will always be a very weak and mostly inapplicable criterion for my research. I’m not here to perform ‘science’ so the public will accept my work, I’m here to seek truth.” Lots of interesting responses. Several people shared this old review essay arguing sociology should be more like biology than like physics, in terms of epistemology. The phrase “runaway solipsism” was used.

I intended my tweets to focus on the open “science practices” which which I have been centrally concerned, centered on scholarly communication: openness, transparency, replicability. That is, I am less interested in the epistemological questions of what is meaning and truth, and solipsism, and more concerned with basic questions like, “How do we know researchers are doing good research, or even telling the truth?” And, “How can we improve our work so that it’s more conducive to advancing research overall?”

Whether or not sociology is science, we should have transparency, accountability, and a sharing culture in our work. This makes our work better, and also maybe increases our legitimacy in public.

Where is ASA?

To that end, as an elected member of the American Sociological Association Committee on Publications, two years ago I proposed that the association adopt the Transparency and Openness Promotion Guidelines from the Center for Open Science, and to start using their Open Science Badges, which recognize authors who provide open data, open materials, or use preregistration for their studies. It didn’t go over well. Some people are very concerned that rewarding openness with little badges in the table of contents, which presumably would go mostly to quantitative researchers, would be seen as penalizing qualitative researchers who can’t share their data, thus creating a hierarchy in the discipline.

So at the January 2019 meeting the committee killed that proposal so an “ad hoc committee could be established to evaluate the broader issues related to open data for ASA journals.” Eight months later, after an ad hoc committee report, the publications committee voted to “form an ad hoc committee [a different one this time] to create a statement regarding conditions for sharing data and research materials in a context of ethical and inclusive production of knowledge,” and to, “review the question about sharing data currently asked of all authors submitting manuscripts to incorporate some of the key points of the Committee on Publications discussion.” The following January (2020), the main committee was informed that the ad hoc committee had been formed, but hadn’t had time to do its work. Eight months later, the new ad hoc committee proposed a policy: ask authors who publish in ASA journals to declare whether their data and research materials are publicly available, and if not why not, with the answers to be appended in a footnote to each article. The minutes aren’t published yet, but I seem to remember us approving the proposal (minutes should appear in the spring, 2021). So, after two years, all articles are going to report whether or not materials are available. Someday. Not bad, for ASA!

To see how we’re doing in the meantime, and inspired by the Twitter exchange, I flipped through the last four issues of American Sociological Review, the flagship journal of the association, to assess the status of data and materials sharing. That is, 24 articles published in 2020. The papers and what I found are listed in the table below.

There were six qualitative papers and three mixed qualitative/quantitative papers. None of these provided access to research materials such as analysis code, interview guides, survey instruments, or transcripts — or provided an explanation for why these materials were not available. Among the 15 quantitative papers, four provided links to replication packages, with the code required to replicate the analyses in the papers. Some of these used publicly available data, or included the data in the package, while the others would require additional steps to gain access to the data. The other 11 provided neither data nor code or other materials.

That’s just from flipping through the papers, searching for “data,” “code,” “available,” reading the acknowledgments and footnotes, and so on. So I may have missed something. (One issue, which maybe the new policy will improve, is that there is no standard place on the website or in the paper for such information to be conveyed.) Many of the papers include a link on the ASR website to “Supplemental Material,” but in all cases this was just a PDF with extra results or description of methods, and did not include computer code or data. The four papers that had replication packages all linked to external sites, such as Github or Dataverse, which are great but are not within the journal’s control, so the journal can’t ensure they are correct, or that they are maintained over time. Still, those are great.

I’m not singling out papers (which, by the way, seem excellent and very interesting — good journal!), just pointing out the pattern. Let’s just say that any of these authors could have provided at least some research materials in support of the paper, if they had been personally, normatively, or formally compelled to do so.

Why does that matter?

First, providing things like interview guides, coding schemes, or statistical code, is helpful to the next researcher who comes along. It makes the article more useful in the cumulative research enterprise. Second, it helps readers identify possible errors or alternative ways of doing the analysis, which would be useful both to the original authors and to subsequent researchers who want to take up the baton or do similar work. Third, research materials can help people determine if maybe, just maybe, and very rarely, the author is actually just bullshitting. I mean literally, what do we have besides your word as a researcher that anything you’re saying is true? Fourth, the existence of such materials, and the authors’ willingness to provide them, signals to all readers a higher level of accountability, a willingness to be questioned — as well as a commitment to the collective effort of the research community as a whole. And, because it’s such an important journal, that signal might boost the reputation for reliability and trustworthiness of the field overall.

There are vast resources, and voluminous debates, about what should be shared in the research process, by whom, for whom, and when — and I’m not going to litigate it all here. But there is a growing recognition in (almost) all quarters that simply providing the “final” text of a “publication” is no longer the state of the art in scholarly communication, outside of some very literary genres of scholarship. Sociology is really very far behind other social science disciplines on this. And, partly because of our disciplinary proximity to the scholars who raise objections like those I mentioned above, even those of us who do the kind of work where openness is most normative (like the papers below that included replication packages), can’t move forward with disciplinary policies to improve the situation. ASR is paradigmatic: several communities share this flagship journal, the policies of which are serving some more than others.

What policies should ASA and its journals adopt to be less behind? Here are a few: Adopt TOP badges, like the American Psychological Association has; have their journals actually check the replication code to see that it produces the claimed results, like the American Economic Association does; publish registered reports (peer review before results known), like all experimental sciences are doing; post peer review reports, like Nature journals, PLOS, and many others do. Just a few ideas.

Change is hard. Even if we could agree on the direction of change. Brian Nosek, director of the Center for Open Science (COS), likes to share this pyramid, which illustrates their “strategy for culture and behavior change” toward transparency and reproducibility. The technology has improved so that the lowest two levels of the pyramid are pretty well taken care of. For example, you can easily put research materials on COS’s Open Science Framework (with versioning, linking to various cloud services, and collaboration tools), post your preprint on SocArXiv (which I direct), and share them with the world in a few moments, for free. Other services are similar. The next levels are harder, and that’s where we in sociology are currently stuck.

For some how-to reading, consider, Transparent and Reproducible Social Science Research: How to Do Open Science, by Garret Christensen, Jeremy Freese, and Edward Miguel (or this Annual Review piece on replication specifically). For an introduction to Scholarly Communication in Sociology, try my report with that title. Please feel free to post other suggestions in the comments.

Four 2020 issues of American Sociological Review

—

This post was previously published on familyinequality.wordpress.com under a Creative Commons License.

***

If you believe in the work we are doing here at The Good Men Project and want a deeper connection with our community, please join us as a Premium Member today.

Premium Members get to view The Good Men Project with NO ADS. Need more info?

A complete list of benefits is here.

—

Photo credit: PHILIP N. COHEN (Author)

The post Sociologist, Scientist? Toward Transparency, Accountability, and a Sharing Culture appeared first on The Good Men Project.