Reviewing Guide

Thanks for agreeing to be a reviewer for the Participatory Design Conference 2016.

Reviews are very important to us. When doing a review you are helping the PD community, the track chairs, and paper authors. The PDC review process for 2016 is a double-blind review, two-round process. All PC members will be acknowledged for their effort on the conference website (www.pdc2016.org) and in the proceedings.

Each submission will be reviewed by at least three reviewers from the Program Committee complemented by a meta-review by one of these three people. In the first round, papers will either be rejected or invited to ‘revise & resubmit’ based on whether or not a paper has sufficient quality to become an excellent paper if revised according to the suggestions by the reviews. Please note that invitation to ‘revise & resubmit’ is not a promise or guarantee of acceptance – this will be decided based on re-review of the revised and re-submitted paper.
This guide is intended to provide assistance to reviewers, especially those new to the craft. We hope it will also be helpful for prospective authors in preparing their papers.

For your convenience the guide is also available as a PDF: LINK

What to look for
PDC research papers and short papers should present original, unpublished ideas and research that advance the field of Participatory Design (PD) and reflect on its potential future development. Compared to Research (or Full) Papers, Short Papers may offer a more limited discussion of related work, or they may, for example, provide a novel design, method or theoretical concepts, without a full evaluation or with less detailed explanation. Both are reviewed within the same rigorous process and should be judged by how well the paper provides a strong contribution to the field.

The role of reviewing
The purpose of reviewing a paper is to assure quality. We need to encourage the author(s) of submissions to deliver improved versions of their paper, one that in inspiring and stimulating ways can becoming a part of and advances the published record of the PD research community. Thus, we also need to filter out low-quality submissions, in a way that helps their authors understand the basis for their rejection and provide some guidance for improvement. We hope the PD community sees reviewing as a mutually rewarding form of collaboration that produces constructive reviews of a quality that we all wish others would produce for us as authors.
With this in mind a good review should:

  1. provide the chairs and PC with a detailed assessment of a paper’s qualities and limitations, as well as its acceptability and relevance for the conference, and;
  2. inform the prospective author(s) what could to be done to improve the paper,
    irrespective of its acceptance to the conference. Authors deserve constructive feedback in a positive tone.

The PDC reviewing form consists of a written review and numeric ratings. The numeric ratings are presented as check boxes on a 5 point scale. 1 is the lowest rating and 5 the highest.

The Written Review
The written comments are the essential part of a review and should not be overlooked.
Comments should clearly reflect the numeric scores. A review with explicit indication of the basis for the ratings makes the review credible and helpful for the authors.
When writing comments, begin with a short statement expressing in your own terms the main argument of the paper and the contribution to the PD field today – both what the authors claim it to be and what you assess it to be. Give an indication of how significant the contribution is. Do this while identifying both positive and negative features of the paper. If you find the contribution, the methodology, the literature review, the quality of writing or other issues with the paper in need of improvement please be explicit about this. Give a clear indication of the basis of the numerical score you assign in each of the rating categories, especially where you give a particularly high or low score, i.e. 5 or 1. See the next section for questions you might ask yourself in coming up with these scores and providing rationales in the written review.
Close your written comments by stating your overall recommendation and why. The numeric overall rating question asks whether you would argue (strongly) for or against acceptance. Highlight here for the PC the main arguments you would make for supporting this score. For round 1 the overall score should reflect whether a paper has sufficient quality to become excellent if revised, or whether this seems out of scope within the timeframe. For round 2, the overall score should reflect whether the paper should be recommended for acceptance at PDC 2016.
Do also identify inadequacies and if possible offer authors additional perspectives and/or resources. If your opinion is that the paper isn’t appropriate for PDC, is there some other venue to consider? Are there steps that the authors could take to improve their chances next time?
The author(s) will read your Written Review closely with interest. If there are remarks that are appropriate for the PC alone, use the later Comments to Committee box for these. The author(s) will not see these comments.

The Numeric Ratings
Here are some helpful questions to keep in mind when assigning scores and justifying them in the written comments.

Contribution to Participatory Design
Is the paper within the scope of PD? Does it deal with collaborative and participatory aspects of design?
Is the article sufficiently novel and interesting to warrant presentation and publication?
Does it contribute to the field? In which ways? What is special? What requires more elaboration? Have the authors identified relevant points? Is the context explained?

Literature review
While research papers can often benefit from an explicit Literature review section, more important is whether the author shows a good grasp of the prior work that is directly relevant to the topic of the paper.
Short papers will benefit from the citation of relevant and appropriate literature while the need for an explicit literature review section should be evaluated according to the aim of the paper (conceptual, experiential, etc…).

Quality of writing, structure and presentation
Is the writing clear and correct?
Would the writing benefit from a careful review by a native/experienced English speaker?
Is the paper a good read? Can you follow it well? How might it be better presented?

Description of methodology and/or robustness of argument
Are claims clearly stated, and supported with appropriate evidence and argumentation?
Are the research tools, techniques and methods adopted appropriately, well understood and properly applied? Could they be improved? Would other approaches likely be more fruitful?
Are you persuaded by the findings and conclusions?

Capability to draw interest in the audience
Stimulating and rewarding in person presentations and subsequent discussions are the lifeblood of the PD community. Is this paper likely to earn audience attention? Can you suggest aspects of the paper to highlight in the presentation?

Overall Rating
Considering all the factors relevant for a strong contribution to PD, is the paper of sufficient quality to be presented at the conference and be published?
Few papers are perfect in every aspect. Are any shortcomings likely to be addressed adequately before the authors submit a final version, or are they too major to take the chance?

Remember

  • Respect the deadlines. The reviewing process is complex and time driven. Late or missing reviews throw off the schedule, and can lead to frustration and extra work for others.
  • Treat the papers you receive for review as confidential documents. Treat author’s work and offer comments like you would want your own work read and reviewed.
  • Authors are required to use a submission template and have certain page limits (10pp, full papers; 4pp, short papers), check that the submission complies with this.
  • Check possible conflicts of interest (e.g. you have close personal or working relations with the presumed authors, you have a direct financial or other interest at stake). There are some special settings/features in the PDC conference management system that assist PC members and chairs to identify conflicts of interest; make sure to review those settings. If you are assigned a paper where your review would create a possible conflict of interest, please return the assignment. It would be helpful if you can suggest someone else who can offer an expert review.

 

Thank you.
PDC2016 Chairs