14 Peer Review Guide

Peer review is invaluable for ensuring the quality of educational content, and is integral to the production of textbooks, just as it is for scholarly monographs and journals. Its presence signals to a prospective adopter that the work has passed through rigorous quality control, and that its content is suitable for use in the classroom.

This is especially significant when working with OER, as the quality, comprehensiveness, clarity, and currency of open textbooks and open educational resources (OER) is often called into question by naysayers. OER, due to their low-cost nature and ease of creation/publication, are mistakenly perceived as low-quality. Peer review is important to dispel these notions, and to encourage wider use and adoption of the book – which is ultimately the goal of most projects. Not only does it give a public indicator of quality to potential adopters, but experience tells us reviewers very often end up adopting the text they’ve reviewed themselves, so it’s great ‘advertising.’

External perception aside, peer review is fundamentally a means for you to receive valuable feedback on your book’s content and make it stronger. It’s a chance for you to share your book with subject experts and ensure that the content is appropriate, accurate, and adequately covers the material.

Why peer review?

If you’re reading this guide, chances are you’ve decided that you want your book to go through peer review. But how do you actually go about organizing this process? The first step should be to decide who is in charge. It may be that you’re happy to coordinate the process, but if you need help, or you don’t want to have contact with your reviewers, we recommend recruiting a review coordinator. Whoever acts as the coordinator, they will be responsible for preparing the review guide, managing recruitment, tracking reviewers’ progress, and relaying information between the reviewers and the authors, editors, and/or project managers as needed.

Once you have a coordinator (and have briefed them about your project if they’re new to the team) there are a few things that you and your team should discuss. The first is to consider what you would like to get out of the peer review process – are their particular expectations or goals? For instance, you may want peer reviewers to ensure that the book’s content and tone are consistent for an undergraduate audience at the first-year level. Or you may want to assess whether there are any major gaps in the content that need fleshing out, etc. Setting these goals will make it easier to prepare a guide for reviewers, recruit the right experts, and ensure that you get something useful out of this process.

You should also be clear about how important the external perception of the review is to you – some potential adopters might expect formal, double-anonymous review, whereas others just want to know there have been several sets of eyes on the content along the way. There are good arguments for both sides of this, and only you can know what’s right for your project.

Finally, while you have your team together, you should discuss at what stage you would like to share the content with reviewers – before it has been edited, or after. One advantage of sharing content with reviewers after it has been edited is that the reviewer can focus more on subject matter, structure, tone, etc. than looking for grammatical errors. On the other hand, conducting review and incorporating feedback after a substantial edit does introduce the possibility of more errors requiring a second round of corrections. This choice might be affected by the resources you have access to as a project. Do note, though, that other project tasks could be taking place while review is ongoing (such as accessibility checks, glossary development, permissions, etc.), so there’s no need to bring you project to a halt for this stage.

Types of peer review and tools: what’s right for you?

Before you begin looking for reviewers, it’s important to decide what kind of peer review you would like to conduct on your project. In academic and educational publishing, there are many terms that are used for different types of peer review. At Rebus, we like to keep things simple, so for us, the biggest choice is whether your review will be anonymous or not.

In an anonymous peer review process, authors and reviewers’ identities will be unknown to one another and, potentially, editors. Conversely, if you don’t want to keep the process anonymous, these identities will be known, which might allow authors, editors, and reviewers to be in communication if need be. When deciding which is right for you, look back on the goals that you had set with your team earlier – if your priority is to follow a largely formal/traditional process for the perceived “prestige” value (we think all kinds of review are equally valuable and prestigious, but the reality is that opinions and perceptions differ), you might want to keep things anonymous. Or, if you want to be able to engage collaboratively with and publicly credit volunteers, you may want to know who they are from the start. What’s important is that you have experts looking through the book, and identifying areas for improvement.

You should also decide whether you would like reviewers to read through the full-text or a chapter or another section of the book, such as a part. Some combination of these can also be valuable – most often with 1-2 reviewers for each chapter, then another 1-2 for the full text. Full-text reviewers would be able to better comment about the structure of the book, whereas chapter or part-level reviewers would be able to provide more granular comments about those sections. For a full-text review, you will need to wait until all the content has come in, whereas chapter-level review could take place on a rolling basis as content is submitted.

Also consider how many reviewers you would like to have for each section, remembering that you can always adjust if you have more or less of a response than you expect. Typically, academic and scholarly works will have a minimum of 2 peer reviewers. However, this number can vary based on your project’s discipline, the topics covered in the book, the reviewers’ areas of expertise, available formats, audience, intended use (e.g. in classrooms or as a library resource), and whether the reviewer is asked to look at the full-text or a single chapter/section.

Lastly, you should take a look at the different tools available for reviewers to provide feedback. You may want reviewers to look at content in Google Docs, and leave comments or edits using the Suggestion mode. This option might be ideal if you are asking reviewers to look at content that has not been edited yet. Or, you might want reviewers to look at the content and leave feedback using a web annotation tool like Hypothes.is. This option would be preferred if your content is already formatted, in Pressbooks, for instance. Alternatively, you could ask reviewers to write a memo and share it with the review coordinator via email. We’ve prepared some templates with instructions for reviewers using either Google Docs or Hypothes.is, which you are welcome to adapt for your review guide.

Creating a review guide

Once all the major decisions regarding the review process have been made, you can turn your attention to preparing guidelines for reviewers. This may be something the review coordinator takes on, or might be a team effort, with input from the project managers and editors.

If you like, you could start with our review guide templates (one for reviewers using Google Docs or another for those using Hypothes.is), and fill in information about your project. Make sure to include details about the audience, guiding questions specific to the project (these should tie into the goals discussed earlier), deadlines (multiple, if review is taking place on a rolling basis), tools and how to provide feedback, and compensation (if any) or other ways that reviewers will be credited or recognized in the project and final text.

It’s important to include a few lines about reviewer etiquette, to remind reviewers that they are critiquing the work that someone may have pored hours into writing. Conversely, be sure to thank reviewers for their service! Reviewers are often involved because they believe in the value of the project, especially if they’re volunteers, and as the one in charge, you can make sure they enjoy the experience and feel appreciated.

This sample review guide follows the structure outlined above, and includes a separate list of project specific questions that reviewers should keep in mind. However, the structure can be adapted — this example review guide incorporates project specific guiding questions with the core components of review. Both example guides are licensed CC BY, and you can refer to or adapt each as suits your project’s needs.

Recruiting reviewers

Finding peer reviewers can be a challenging task, but far from impossible. Support staff from the University Libraries can help manage the peer review process for your OER project.

The key to successful recruitment is having a clear and concise call. Your call should be short, informative, and direct interested candidates to the right place. You should aim to include the following details: a brief description of the project, with a link to the public listing page; the intended audience; criteria for reviewers based on what you’d like to get out of the review process; and clear directions to volunteer. Here’s a sample call posted on a public forum, and an example of various kinds of copy used for emailing potential volunteers.

You can also include a few lines encouraging members of underrepresented groups within the community and people working in different contexts to your own to apply (e.g. different regions/countries, different kinds of institutions etc.) – having a wide range of perspectives assessing your content will help to ensure that it meets the needs of all students in the long run!

Before sending out the calls, you should also be prepared to track responses from potential reviewers. Use the team’s preferred tools, such as Google Sheets, to collect a list of interested candidates, confirmed reviewers, contact information, deadlines, and status. Doing so will make it easier for you to oversee the process, and not have to sift through emails or chats for an update. We’re prepared a review tracking template that you could copy and adapt for your project.

Once the call and the tracking sheet are created, you’re ready to send out your call! A great place to start is mailing lists or listservs that are specific to your field or discipline. You can also share the call with the Rebus Community, on social media, and in your personal networks (and encourage others on the team to do the same). A good option is to have each team member write up a list of all the people they think are qualified and who might be interested, and send them each an email asking them to act as a reviewer. The personal touch goes a long way! It may take a week or two for you to start receiving responses, so be patient, but if you find that you haven’t had the response you hoped for, you could also send out cold calls. This involves some good old fashioned internet sleuthing to find faculty you think might be suitable for the role. Sending out cold calls is time consuming, and can have low response rates, but if you pick your candidates wisely (making sure their expertise aligns with the project), you might be surprised at how willing people are to be involved.

Managing the review process

As responses start filtering in, you can update your tracking spreadsheet and assess whether the respondents are right for the role. Make sure to respond to reviewers who have been selected, and offer a note to the ones who haven’t! You can confirm details with selected reviewers, sharing a copy of the review guide and setting a clear deadline for completion of the review (we recommend allowing at least 4-6 weeks).

During this exchange, you should also (of course) share the content with the reviewers, or get this to them as soon as possible. Ideally, you’d like to send content to reviewers immediately after they express their interest in the task, so as to avoid any waiting around.

Once all the details are confirmed and content has been sent, you can let the reviewers start the process and largely leave them be. However, we do recommend a series of check-ins leading up to the agreed upon deadline to make sure that the reviewer is all set, that they don’t have any questions, and to make sure they are still committed to completing the task. You can check-in as many (or few) times as you like. We recommend doing so:

• 2-3 weeks after you confirm the details with them
• A week before the deadline
• One (business) day after the deadline if they haven’t submitted
• Then as often as needed to get a response from them before it is clear they are no longer participating in the project!

The next step is to collect the reviews as they roll in, and pass them along to authors or editors. Be sure to send thanks to the reviewers for their time and feedback. You should encourage them to adopt the book when it is available for classroom use, and let them know you’ll share updates about the project with them as things progress.

Depending on the nature of your project and the type of peer review process you have selected, you can also ask reviewers if they would like to communicate with authors or editors as changes are being implemented. And finally, it’s also be good to ask reviewers to share their feedback about the process overall, so you can keep it in mind for the rest of the project!

After peer review

Of course, the ‘peer review’ stage doesn’t end after the review has been submitted, and you’ve shared your thanks with the reviewers for their work. The obvious next step is to figure out how to incorporate the feedback! This is most often done by the authors themselves, with the editors or project managers coordinating, but sometimes the editors may take on the changes themselves. There are also inevitably decisions around which suggestions to take, and which to defer or leave aside. We’ll have some guidance on how to handle this portion of the review in future.

Aside from working on the content, there are a last couple of threads to tie up after the review, but they aren’t very intensive, as you’ll see.

First of all, we suggest you send ongoing updates about the project to reviewers, especially when the book is released. More often than not, they’re excited to hear about the progress, and (very importantly!) they could be potential adopters once the book is out. Share the announcement of the book’s release with them, and encourage them to spread the word in their institutions and networks – as reviewers, they’re in the perfect position to vouch for the text.

Once all the reviewers have shared their feedback and changes have been made to the content, it’s good to include a Review Statement in the book back matter. This is an excellent way to indicate to potential adopters that the book has undergone peer review and is a high quality resource. It’s also a great chance to list reviewers (if agreed that they’ll be credited) and publicly recognize their work on the project. Here’s a review statement template that you can work with, and you can take a look at an example statement from another open textbook.

Finally, it’s a nice gesture to send a physical copy of the book to the reviewer as thanks, if you’re printing copies of the book, and have the capacity to do so. Alternatively, a thank you card or personal email can do the trick – or an offer to return the favour and review their future open textbook!

Other things to consider

While the major aspects of the peer review process are covered above, there are a few last things that might come up. Many open textbook projects don’t have much in the way of funding, and often none at all, so financial compensation for reviewers won’t always be possible. While we do encourage you to look for sources of funding for peer review, we fully understand that this simply may not be possible in all cases. Reviewers may ask about compensation outright, as it is fairly common to be paid for review of other scholarly works, so make sure you have an answer for them. If no funds are available, you could let them know of any other recognition you could provide them, and to reiterate the importance and potential impact of your project. Money isn’t the only motivator, so share a bit more about the project and see whether this inspires them to participate.

At the same time, remember that reviewers are people too! They might have their own reasons for not participating, stepping down from the project, or requesting delays. While you have a timeline that you’d like to follow, try to be understanding as the inevitable obstacles or delays come up. Look out for the people involved in your project, and if anyone does have to drop out of the job they signed up for, think about how else you might be able to keep them engaged.