Thanks for accepting to review a package for rOpenSci! This chapter consists of our guidelines to prepare, submit and follow up on your review.
You might contact the editor in charge of the submission for any question you might have about the process or your review.
Please strive to complete your review within 3 weeks of accepting a review request. We will aim to remind reviewers of upcoming and past due dates. Editors may assign additional or alternate reviewers if a review is excessively late.
rOpenSci’s community is our best asset. We aim for reviews to be open, non-adversarial, and focused on improving software quality. Be respectful and kind! See our reviewers guide and code of conduct for more.
If you use our standards/checklists etc. when reviewing software elsewhere, do tell the recipients (e.g. journal editors, students, internal code review) that they came from rOpenSci, and tell us in our public forum, or privately by email.
Volunteering as a reviewer
Thank you for your desire to participate in rOpenSci software peer-review as a reviewer!
Please fill our volunteering form.
If you see a current submission that is particularly relevant to your interests please email info@ropensci.org, including the name of the package, the URL to the submission issue and the name of the assigned editor. However, keep in mind that reviewer invitations are kept at the editor’s discretion, and the editor might well have already emailed people. Please do not volunteer for all issues, and do not volunteer via GitHub interface.
For other ways to contribute, refer to rOpenSci contributing guide.
Preparing your review
Reviews should be based on the latest GitHub version on the default branch, unless otherwise indicated by package authors. All submissions trigger a detailed report on package structure and functionality, generated by our pkgcheck package. If the package has changed substantially since the last checks, you may request a re-check with the command @ropensci-review-bot check package. Note that when installing the package to review it, you should make sure you have all dependencies available (for instance run pak::pak()).
General guidelines
To review a package, please begin by copying our review template (or our review template in Spanish) and using it as a high-level checklist. In addition to checking off the minimum criteria, we ask that you provide general comments addressing the following:
- Does the code comply with general principles in the Mozilla reviewing guide?
- Does the package comply with the rOpenSci packaging guide?
- Are there improvements that could be made to the code style?
- Is there code duplication in the package that should be reduced?
- Are there user interface improvements that could be made?
- Are there performance improvements that could be made?
- Is the documentation (installation instructions/vignettes/examples/demos) clear and sufficient? Does it use the principle of multiple points of entry i.e. takes into account the fact that any piece of documentation may be the first encounter the user has with the package and/or the tool/data it wraps?
- Were functions and arguments named to work together to form a common, logical programming API that is easy to read, and autocomplete?
- If you have your own relevant data/problem, work through it with the package. You may find rough edges and use-cases the author didn’t think about.
Please be respectful and kind to the authors in your reviews. Our code of conduct is mandatory for everyone involved in our review process. We expect you to submit your review within 3 weeks, depending on the deadline set by the editor. Please contact the editor directly or in the submission thread to inform them about possible delays.
We encourage you to use automated tools to facilitate your reviewing. These include:
- Checking the initial package report generated by our
@ropensci-review-bot.
- Checking the package’s logs on its continuous integration services (GitHub Actions, Codecov, etc.)
- Running
devtools::check() and devtools::test() on the package to find any errors that may be missed on the author’s system.
- Seeing whether tests’ skipping is justified (e.g.
skip_on_cran() tests that do real API requests vs. skipping all tests on one operating system).
- If the package is not submitted via the default/main branch, remember to switch to the submitted review branch before starting your review. In this case, you will also have to search the package locally, as GitHub search is limited to the default branch. Further, documentation hosted on a
pkgdown website is not necessarily up-to-date, and we recommend to inspect the package’s documentation locally by running pkgdown::build_site().
Reviewers may also re-generate package check results from @ropensci-review-bot at any time by issuing the single comment in a review issue: @ropensci-review-bot check package.
Off-thread interactions
If you interact with the package authors and talked about the review outside a review thread (in chats, DMs, in-person, issues in the project repository), please make sure that your review captures and/or links to elements from these conversations that are relevant to the process.
Experience from past reviewers
First-time reviewers may find it helpful to read (about) some previous reviews. In general you can find submission threads of onboarded packages here. Here are a few chosen examples of reviews (note that your reviews do not need to be as long as these examples):
You can read blog posts written by reviewers about their experiences via this link. In particular, in this blog post by Mara Averick read about the “naive user” role a reviewer can take to provide useful feedback even without being experts of the package’s topic or implementation, by asking themselves “What did I think this thing would do? Does it do it? What are things that scare me off?”. In another blog post Verena Haunschmid explains how she alternated between using the package and checking its code.
As both a former reviewer and package author, and now editor, Adam Sparks wrote “[write] a good critique of the package structure and best coding practices. If you know how to do something better, tell me. It’s easy to miss documentation opportunities as a developer, as a reviewer, you have a different view. You’re a user that can give feedback. What’s not clear in the package? How can it be made more clear? If you’re using it for the first time, is it easy? Do you know another R package that maybe I should be using? Or is there one I’m using that perhaps I shouldn’t be? If you can contribute to the package, offer.”
Helper package for reviewers
If working in RStudio, you can streamline your review workflow by using the pkgreviewr package created by associated editor Anna Krystalli. Say you accepted to review the refnet package, you’d write
remotes::install_github("ropensci-org/pkgreviewr")
library(pkgreviewr)
pkgreview_create(pkg_repo = "embruna/refnet",
review_parent = "~/Documents/workflows/rOpenSci/reviews/")
and
- the GitHub repo of the
refnet package will be cloned.
- a review project will be created, containing a notebook for you to fill, and the review template.
- note that if the package is not submitted via the default/main branch, you need to switch to the submitted branch before starting your review.
Feedback on the process
We encourage you to ask questions and provide feedback on the review process on our forum.
Submitting the Review
- When your review is complete, paste it as a comment into the package software-review issue.
- Additional comments are welcome in the same issue. We hope that package reviews will work as an ongoing conversation with the authors as opposed to a single round of reviews typical of academic manuscripts.
- You may also submit issues or pull requests directly to the package repo if you choose, but if you do, please comment about them and link to them in the software-review repo comment thread so we have a centralized record and text of your review.
- Please include an estimate of how many hours you spent on your review afterwards.
Review follow-up
Authors should respond within 2 weeks with their changes to the package in response to your review. At this stage, we ask that you respond as to whether the changes sufficiently address any issues raised in your review. We encourage ongoing discussion between package authors and reviewers, and you may ask editors to clarify issues in the review thread as well.
You’ll use the approval template.