2 Years ReproducibiliTea Aalto
Today, 2 years ago we started the ReproducibiliTea chapter Aalto (then AERIS). In this post I want to briefly reminisce over that time, about what I learned, and what tips I would give to others who have or might want to start a journal club.
We started the journal club (JC) in March 2020. Originally, my supervisor had floated the idea to start a reading group for the lab; to get some more reading done and do a little team building. I can’t remember why exactly, but I suggested making it a ReproducibiliTea journal club. A few weeks later - with support and blessing from my supervisor - we had our first session, opening with the Manifesto for Reproducible Science.
The original idea was to have the JC on Fridays: To round out the work week before retiring to a bar. However, we only had two or three physical meetings before the university went into lockdown and we had to move online. We have not moved back since (with only some brief attempts at a hybrid mode in what seemed to be two weeks between Delta and Omicron).
With this Sunday we have now met almost weekly for two years and have discussed about 70 papers. We average about 4 to 6 participants. Most of us work in a computer science department with similar publication venues but very different research topics and backgrounds. We are psychologists, computer scientists, philosophers, neuroscientists and more; among our regulars we have master students, PhD candidates, post docs and professors.
As such, as a place where any coworker can join and contribute, the JC has been unique for me in the last two years and found it hugely beneficial for me and my work. Here are a few take aways I had.
Of blind spots and new friends
I think that as scientists - while we naturally become experts in our domain - we are susceptible to developing blind spots. We do not necessarily broaden our understanding of a topic but deepen it, with many areas of research drifting into our the periphery. To some degree this is probably unavoidable and during our day to day work, we do not necessarily notice if we gain blind spots as our direct colleagues tend to specialize in similar things.
I found that a JC can work to mitigate this to some extent: For us having people from many different corners of science created a space in which we meet and discuss with people who have different scientific practices and goals, who publish to different communities, and who employ different theories and methods. Not to mention that everyone brings papers from their respective discipline as well.
Additionally, people usually do not specialize in Open Science. Few people are meta-scientists and the field exploded in size and complexity in the last decade. So no one is really that well versed in open science specifically and discussing an Open Science paper can be somewhat of a free for all. There are no real experts, everyone is equally welcome to chime in and bring their interpretation, real life connections and views to the table.
While it does take continuous work to ensure that the JC is a space where everyone really can and wants to speak freely - I have certainly failed to ensure this in the past - I found the ReproducibiliTea format a good starting point for creating such a space at a university.
Not all papers are good
Okay, that many papers are not that great might be self-evident - hey, it is arguably the core point of the replication crisis. However, especially as an early, early ECR (read: late masters, early PhD student) I found it very difficult to actually decide if the paper I was reading was bad. If I didn’t understand what a paper was talking about, if I found their conclusions confusing or not backed by data, I usually blamed myself. After all, I thought it more likely that I just lacked the ability to correctly read the paper than there being a mistake and neither authors, nor editors nor reviewers noticed.
Only when I started discussing papers with others - listening to other people spelling out their problems with a paper or hearing them share my own gripes - I gained the self-confidence to decide if a paper (or an aspect of it) was bad. It became easier for me to differentiate when it was me who didn’t get it or if it was the paper that had dropped the ball. Having discussions about a paper, how it created arguments and where it failed (or succeeded) helped me immensely in writing logically coherent and deriving research questions.
Of course - and to be fair - sometimes it is totally just me who doesn’t get a paper, but in those cases there are now other people who can show me where and how I was wrong.
Lets talk about OS
Starting out doing Open Science can be difficult. Especially when never having learned it or when not having thought about OS from the very start of a project, it can be complicated to suddenly having to implement OS practices: For example, making the analysis script available and understandable to others when not having it written like that in the first place can be a horrible experience. Likewise post hoc wanting to publish your data without getting consent from the participants or not being able to anonymize it can be neigh impossible (or lead to exhausting conversations with university lawyers). At the same time, the amount of Open Science literature out there can be overwhelming and getting into it can seem like a bad time investment when one could be reading papers directly applicable to their work instead.
The JC can act as a guide to get into Open Science with the soft social pressure of regular meetings and it can mainstream Open Science for you, i.e., make it a normal part of your research process and something you start to consider from the very conception of a study.
Last but not least, our JC became a nice place to hang out with people. Even if I had found that reading the papers was a waste of time, I think in these last two years it is hard to understate how important it can be to just regularly hang out with people. I - and several others who joined the JC - did never experience a university culture at Aalto as everybody was at home. The JC became a sort of surrogate for that. It became a place to exchange grievances, to share frustrations and to have a laugh with colleagues.
What I learned about running a JC
Here are a few things that I learned about running the JC. Some things might be self-evident, some things might be different to your experience, still I found these thoughts useful when planning out the sessions.
Low barrier to entry
One thing I try to make sure is that no one has to actually read the paper to partake in the discussion. This might be contradictory to the idea of a journal club, but I want allow people to join who did not have time to prepare and facilitate that people can freely drop in and out or spontaneously decide to join. Further, after two years we tend to no longer really read papers that where meant to introduce people to Open Science (as can be found in this awesome reading list: https://rpt-rl.netlify.app/) so any newcomers run the risk of being a bit lost.
The best and simplest way I found to offer every participant a decent baseline to join the discussion - regardless of reading the paper and previous knowledge - was giving a brief summary of the paper. My summaries usually contain an overview of the core arguments and attempt to link the paper and authors to other papers we read in the past. I also collect the summaries on our OSF repo for future reference.
While this does shift quite a lot of work onto me - after all I have to both read the paper closely and prepare an informal presentation - I haven’t mind so far as I found it to be a good deterrent against becoming lazy (which I tend to do). Also - thankfully - there are a lot of enthusiastic regulars who happily take over presentation duty every now and again.
Know what kind of JC you are
Be aware of what your regulars (and irregulars) expect and want from your JC. One question we ask ourselves sometimes is if we want to invite the authors of a paper to the discussion. In fact, in February we did have our first ever author joining us. I found that having the author present is fun and enlightening. However, it does change how the JC is conducted.
Our main mode is being a small, informal group. This leads to a pretty free, unguided discussion that more than once has shifted away from the paper to random things we were otherwise occupied with. When an author (or guests in general) are present, doing this would be quite rude. I mean, imagine being invited to a discussion about a work and your hosts suddenly start complaining about their deadlines. Further, having the author present naturally shifts the discussion away from the paper’s quality towards the papers goal. I.e., we discuss less what the paper does well and what it doesn’t and instead focus more on the results and take-aways of the paper.
Moreover, having the author present naturally shifts the modus of the session from a more round table style to a talk format with the author presenting their work and afterwards answering questions. A great format, but not always what your JC members want. In fact, several of my regulars directly stated that they joined for the round table, and not to listen to talks.
This is not to say, that a JC should not feature talks. In fact, several ReproducibiliTea Journal Clubs are talk series and I understand them to be quite successful. In fact, so far I have not visited a ReproducibiliTea Talk that was not super interesting.
Another type of papers that in my opinion require a change in modus from the round table are those whose goal it is to teach you - the researcher - a new technique. For example, how to use Docker for reproducible computation. In my experience, these papers are not easily discussed but lend themselves more to a workshop session which - in turn - benefits from things like practice assignments, prepared software, etc. Again: Totally possible to do under the umbrella of a ReproducibiliTea JC, but it will require special preparation and potentially caters to a different subset of people.
At the moment I am considering of branching our JC out a bit. Keeping the weekly JC but spinning off into a local talk series. It would be great to get some funding to maybe even invite people in person again once the pandemic is truly over. Another idea is to suggest a ReproducibiliTea seminar for master students. ReproducibiliTea Aalto, however, I want to keep unchanged for now. We are a small group for round table discussions. That is what we want and that is what we go to the JC for.
There is enough to talk about!
One fear I had starting out was that one day we might run out of topics to discuss. I now highly doubt it could ever happen.
Instead, there is the opposite problem: Given the amount of possible things to read, how does one decide on what to feature? In our JC, we set up a system in which everyone can recommend papers to read and every month we vote on a subset of papers to read next month. Besides this saving me from having to write a curriculum, this has the benefit that we only discuss papers we have already indicated interest in (by having either voted for or even recommended them) which consequently makes it more likely that people come to the meeting.
This is how I set the system up: I have a Zotero libary in which I collect all papers recommended by others and the ones I stumble across myself. During the last week of every month I set up a survey in the slack channel with a selection of papers from the library. Everyone in the channel may vote for as many papers as there are meetings in the next month. During the last session I briefly go over which where the four most voted for papers, set the schedule and ask who wants to present them. In general, I would highly recommend having a slack/discord/teams/whatever channel to facilitate collaboration on the JC and to allow for small talk between members. I find those more effective/quick than E-Mail lists and it better facilitates being a community.
Do a positive month every know and again
Reading Open Science/Metascience papers can be a bit depressing. The constant reminder of science imploding is definitely putting urgency into improving your own work, but it is exhausting to be face to face with it all the time. Hence, one of our non-regularly returning segment are “positivity months” (I am still workshopping a better name). During these months we are focusing specifically on papers that are good, non-meta science papers. Papers that show that besides all the problems in science there are works that are just great pieces of science. Those papers do not need to be perfect, but just something we personally like for whatever reason. Papers that do something really well, papers that we enjoy reading.
For me one such paper - the one I brought the last time we did a positivity month- was Przybylsky et al.’s 2014 paper on aggression and games which I believe to be a great example of asking a question and systematically trying to isolate an answer. In fact, it is the paper that convinced me to become a scientist.
Conclusion
So yeah. These were some thought I have after doing two years of ReproducibiliTea Aalto. It is fun. 10/10, can highly recommend (though I am obviously biased as I have also since joined the Global ReproducibiliTea steering Committee).
So here is to the next 2 years. Thank you to everyone who joins and joined. To those who have since moved on or will move on soon. It was great time so far.
