MICCAI Conference Review Process

Context
General
A note on conflicts of interest, double submissions, plagiarism and ethical issues
Stage 0: Call for PC Members
Stage 1: Reviewer Database
Stage 2: Intention to Submit
Stage 3: Paper Submission
Stage 4: Additional PC Member’s Enrolment
Stage 5: Paper Allocation to PC Members
Stage 6: Paper Allocation to Reviewers
Stage 7: Review
Stage 8: Early Paper Decisions and Rebuttal Process
Stage 9: Primary Scoring of Borderline Papers
Stage 10: Acceptance Process
Stage 11: Oral Decision Process (1 week)
Stage 12: Official Notification to Authors

context

Context

The purpose of this document is to define the Review Process for the MICCAI conference. The document has been initiated and is endorsed by the MICCAI 2017, 2018, 2019 and 2020 organizing committees, to be passed on each year, from successive MICCAI organizing committees, with the goal to refine and include feedback from the full set of future conference organising committees to create a “sliding window” and consistency between years.

The document is heavily inspired by the previous Review Process developed by S. Ourselin, D. Hawkes, N. Navab, P. Golland, N. Ayache, and G. Fichtinger. The current review process version was prepared jointly by the MICCAI 2017, 2018, 2019, 2020 committees with the following people involved: L. Joskowicz, M. Descoteaux, D.L. Collins, L. Meier-Hain, P. Jannin, S. Duchesne, C. Davatzikos, A.F. Frangi, J.A. Schnabel, D. Shen, T. Liu, A. Martel, D. Mateus, D. Stoyanov, M. Reyes, D. Racoceanu. We are grateful for all their contributions.

The policies set within are born of a desire to enhance the previous process, by specifically:

  • simplifying the process to increase compliance and enforceability;
  • lowering the burden on participants;
  • increasing review quality and retention;
  • clarifying roles and responsibilities to increase understanding from all participants;
  • decreasing the appearance of arbitrariness in the decision-making process, by being more transparent; and
  • reducing costs.

As in previous MICCAI conferences, the goal of the MICCAI Conference review process is to select the best papers in each discipline. This selection should be fair, taking into consideration the specialized nature of our discipline and the size of our community; efficient, in not wasting valuable time, effort and funding from our peers; and just, in exploiting the consensus of peer comments.

context

General

This summary is made public so that all participants understand the review process and can plan accordingly.
The MICCAI 2019 review process will be overseen by the following individuals:

Program Co-Chairs:

  • Terry Peters, Robarts/Western University, Canada
  • Larry Staib, Yale University, USA
  • Sean Zhou, United Imaging Intelligence, China
  • Caroline Essert, University of Strasbourg, France
  • Pew-Thian Yap, University of North Carolina at Chapel Hill, USA

Associate to Program Chairs:

  • Ali Khan, Robarts/Western University, Canada

Submission Platform Manager:

  • Jackie Williams

General Chairs:

  • Dinggang Shen, University of North Carolina at Chapel Hill, USA
  • Tianming Liu, University of Georgia, USA

The Program Chair will create two bodies of participants that are essential to the review process:

  • The Program Committee (PC), composed of Area Chairs (ACs); and
  • The College of Reviewers (Reviewers)

The following documents supplement the following guide, and will be made available on the MICCAI 2019 conference website:

A note on conflicts of interest and plagiarism
Conflicts of interest are inevitable in a community as networked as ours. To handle such conflicts, the MICCAI Program Chairs will rely on authors’, reviewers’ and area chairs’ professional behaviour, augmented with automated means.

The main check for conflicts will be done using Researcher.cc, which is used by CMT, as well as other well-known conferences such as CPVR, NIPS, and ICML. Researcher.cc requires only a Google scholar profile (https://scholar.google.com) and DBPL profile (http://dblp.uni-trier.de) for ACs, reviewers and authors. No additional information is necessary. Using these two sources, Researcher.cc extracts institutional information and checks automatically for conflict.

For MICCAI 2019, we will use and/or automatically create Researcher.cc profiles from all names and emails exported from the MICCAI 2018 CMT system. Reviewers will then receive an email with detailed instructions to double-check their information and make changes if needed. Authors must also make sure they have a Researcher.cc account, and provide their Google scholar and DBPL profiles when submitting their papers to manage conflicts between authors and reviewers. All authors are required to use their institutional email address.

In previous conferences, situations have arisen when authors have modified the author list of papers at any stage from the final submission deadline all the way to the final camera ready or final journal submission in MICCAI-related special issues. While there may be justifiable reasons to do this (e.g., additional experiments requested that require additional clinical collaborators), any change in the author list should be justified in writing to the Program Chairs prior to submission. In particular, it is unacceptable to include anyone as co-author who has been directly or indirectly involved with the manuscript at any stage on the decision process either as a reviewer, area chair or as part of the organizing committee; this includes any direct follow-on publications (e.g., MICCAI special journal issues).

We further wish to highlight our policies about double submission and plagiarism. In line with past MICCAI conferences, all MICCAI 2019 submissions must be original and cannot already be published or considered for publication elsewhere (with the explicit exception of arXiv.org as a form of prepublication of MICCAI contributions, however, authors should be aware that this may compromise the anonymity of their MICCAI submission and therefore may inadvertently affect the review process). Plagiarism is a serious offence that consists of using wordings or results of someone else’s publication without giving credit or providing appropriate referencing. Reviewers and Area Chairs can identify such acts of plagiarism, and will refer these to the MICCAI OC who will investigate, and in severe cases refer these on to the MICCAI Society for determining further action.
MICCAI conference organizers also should coordinate with the MICCAI Society on ethical issues such as: i) which individuals are currently barred from any role, and ii) alert the Board to any apparent unethical behavior.

context

Stage 0: Call for PC Members

The MICCAI Society should maintain a database of past Area Chairs and reviewers. Excellent ACs and reviewers should be identified after each event. The MICCAI 2019 organizing committee will build on the list provided by MICCAI 2018. As in the previous two years, MICCAI 2019 will use CMT to maintain this database. The information about the number of times a member has been an AC will be kept indefinitely. The information on excellence of reviewers or ACs will be maintained in the system.

MICCAI 2019 also will issue a call for participation in the PC via the newsletter and mailshots. Individuals will submit relevant information regarding their past participations in MICCAI and other similar conferences, as well as other biographical details.

Individuals will be chosen from the database, the received invitations and other sources, and invited to join the MICCAI 2019 PC. When they apply, ACs will need to register with Researcher.cc, update their Toronto Paper Matching System entries, create/update their CMT profiles and associate themselves as being either MIC, CAI or MICCAI. They will need to use their institutional email address. The MICCAI Program Chairs and Co-Chairs will then select the program committee using the following guidelines:

  • The PC composition should be representative of all MICCAI topics, and with a range of seniority;
  • There will be 30 ACs initially recruited in this phase, balanced between MIC, CAI and MICCAI experts, with a spread of expertise based on past MICCAI history;
  • For MICCAI 2019, we will aim to have the best PC committee possible while aiming for gender parity (50% male, 50% female) and balanced demographics.

A webinar may be held to explain the review process to ACs and secure their commitment early in the process.

context

Stage 1: Reviewer Database

The list of potential reviewers is based on the active reviewers on the last MICCAI conference, to which are added all first and last authors at that conference. Further, student reviewers must be at least enrolled at Ph.D. and have two or more published articles in a related field to qualify as reviewers. ACs will contribute to updating and expanding the list of existing reviewers.
Each potential reviewer will be invited to review for MICCAI. By accepting to review:

  • the reviewers commit to review a maximum of 6 papers
  • the reviewers will be asked to create or update their CMT and Researcher.cc profiles
  • the reviewers will be required to use their institutional/work address as primary contact

context

Stage 2: Intention to Submit

Authors will provide an intention to submit one week prior to the paper submission deadline. Authors must provide: i) a list of co-authors and their affiliations, ii) Google scholar and DBPL profiles for each co-author where possible, iii) the title of the paper, and iv) the abstract of the paper. This information will be used to start building the assignment of papers to ACs and potential reviewers. Authors and co-authors must use their institutional/work email address.

context

Stage 3: Paper Submission

Authors will submit papers in CMT. The MICCAI Conference review process will be double-blind, i.e. the names of the authors will be hidden from the Area Chairs and Reviewers, and the names of Reviewers and Area Chairs will not be revealed to the authors. To achieve this, papers must be properly anonymized before submission. At the discretion of the Program Chairs, a paper may undergo an outright rejection when it is in blatant breach of these anonymization rules. Area Chairs and Reviewers may bring concerns about such a breach to the attention of the Program Chairs.

Each paper must be submitted with at least three keywords selected from the CMT system. Authors also must identify to which stream the paper belongs, i.e., either MIC, CAI, or MICCAI. The keywords, stream, and the paper itself will be used to generate suggested reviewers using an automated paper matching system embedded in the CMT system (see Stages 5 and 6).

context

Stage 4: Additional AC’s Enrolment

The goal of this step is to ascertain the breadth and depth of expertise required within the PC in response to the initial submissions from authors. Given that it is almost impossible to accurately predict which domains will be most represented in any conference edition, this step will allow for the adjustment of the PC composition.

The Program Chair will then invite approximately 10-20 additional PC members having an identified expertise from the database of potential ACs and reviewers. These individuals must be free from conflict and could be drawn from the larger community. Those who agree to join, must update their CMT profiles and associate themselves as being MIC, CAI or MICCAI, but at this point the balance between these three groups will be proportional to their representation in the submissions.

If required, a webinar will be held for all PC members to provide early statistics and explain the process once again.

context

Stage 5: Paper Allocation to ACs

Each paper will be assigned to an AC automatically by the CMT system, based on the ranked lists provided by the authors and the Toronto Paper Matching System. (Note that the AC remains blind to the paper authorship.) The Program Chairs will check the assignments to make sure all papers received a good assignment.

The essential role of the AC is to move the paper through the review process, up until the decision by the PC. ACs will use their knowledge of the topic and of the appropriate reviewers to ensure the best (most informative) reviews.

For MICCAI 2019, a maximum of 20 papers will be allocated to each AC. This is necessary to ensure a proper statistical distribution in the ranking to follow (cf. Stage 7). The number of ACs will be adjusted accordingly. Thus, for approximately 50 ACs, MICCAI 2019 is expecting to handle around 1,000 submitted papers.

context

Stage 6: Paper Allocation to Reviewers

The goal of the paper allocation is to find the most appropriate reviewers in terms of expertise for a given paper. This step is achieved in 2 phases using the CMT system:

In phase 1, the CMT system will provide a list of potential reviewers for each paper to the AC. This ordered list will be generated based on keywords, Toronto Paper Matching System (TPMS), and other tools use by the computer vision community. Using their expertise and judgement, the AC will re-order this list as needed and can add additional reviewers to the list (including those suggested by the authors, if deemed reasonable). The TPMS system will re-optimize matching of reviewers to papers based on a large weighting on the ACs suggested ranked list of reviewers (at least 6 reviewers on the list). The optimization will consider the ordered list from the AC, while load balancing across all papers, reviewers, and ACs. Each reviewer will be assigned an initial list of 25 papers.

In phase 2, reviewers will bid for papers by categorising them into “I am interested in reviewing this paper”, “I can review this paper if need be”, “I am not interested in reviewing this paper”, “I have a conflict with this paper”). The TPMS system will then re-optimize matching of reviewers to papers based on the ranked list provided by the reviewers, while load balancing across all papers, reviewers and ACs. Note that the reviewers and secondary ACs remain blind to the paper authorship at all times. CMT permitting, primary ACs will be blinded to the final set of reviewers.

This procedure has the following advantages over previous methods:

  • Better matching of paper and reviewer; should yield better reviews
  • Better conflict identification and resolution
  • Reduced workload for AC
  • The streamlined process saves several steps (reviewer acceptance, reassignment of papers), and thus saves time in the process
  • In line with other conferences, thus leveraging existing reviewer experience with CMT
  • Automated procedure with fixed dates ensures that procedure is more likely to stay on track
  • Potential race condition for reviewers is avoided

The PC will ensure ACs have re-ordered reviewers on time and can provide additional opinions if a borderline situation cannot be resolved by the AC PC members.

context

Stage 7: Review

The goal of the review step is to provide constructive criticism of a submitted paper.

The reviewers will:
Provide a comprehensive, fair review

  • Provide a composite score
  • Recommend papers for orals and awards
  • Self-declare their expertise for each paper (passing knowledge, knowledgeable, expert)
  • Rank papers to provide additional assessment

The AC will log in frequently and react to the reviewers’ actions: remind reviewers to login and download the papers, and especially ask for a more detailed or fair review. This is critical. Should the AC be unsatisfied with the quality of a review, and failing to get further feedback from the reviewer, then the AC can ask for additional reviewer(s) input on the paper, beyond the original three reviewers. Any reviewer who does not provide a quality review will be identified in the reviewer database.

For MICCAI 2019, we aim to run a “live fire” trial run of both stages 5 and 6 in the weeks preceding the final deadline for paper submission. We will use approximately 40 papers submitted ahead of the deadline for this purpose. They will be assigned to their AC, who will then assign reviewers, as per the instructions set forth above. The goal of this trial run is to iron out any bugs in the CMT – with approximately 160 users – while ensuring that all PC members are familiar with the process. While these will be real reviews, the papers will be kept accessible for the reviewer for them to change their reviews once the rest of the conference papers are completed.

context

Stage 8: Early Paper Decisions and Rebuttal Process

The OC will implement clear accepts, borderlines and clear rejects papers based on consistent reviews (all reviewers recommend borderline accept or above; or borderline reject and below, respectively). Borderline papers will enter Stage 9 (below). The goal of the rebuttal process is to provide authors, in particular authors of borderline papers, the opportunity to highlight possible misinterpretations or inaccuracies in the reviewer’s findings, and inform the AC’s final recommendation for scoring:

• Reviews of the papers are sent to the authors
• Authors have one (1) week to submit their rebuttal
• Only the ACs will read the rebuttal during this process

context

Stage 9: Paper Scoring of Borderline Papers

The goal of this step is to provide a numerical score and rank for each paper assigned to the AC.
After rebuttals are entered, each AC will provide for the borderline papers a recommendation to either “accept” or “reject”. ACs will further provide a meta-review for each assigned paper, consistent with their recommendation and based on the assessment of the reviews and rebuttal.

Further, each AC will:

  • rank all papers allocated to him or her
  • select between 0-2 nominations for oral papers
  • recommend papers for Student Awards
  • recommend 2-3 Reviewer Awards

context

Stage 10: Acceptance Process

The goal of the decision process is to establish the final list of accepted papers.
Prior to this stage, the Program Chairs will update the ACs regarding the total number of orals and selected papers to be awarded at the conference after consultation with the Organizing Committee and the MICCAI Board.

For MICCAI 2019, the overall acceptance rate is aimed at 35%.
The top 20% papers by will be accepted based on consistent “accept” reviews and high scores.

  • For 20 papers per AC, this represents 4 papers per AC
  • For 1,000 submissions, with 50 PC members, this represents a maximum of 200 papers

The bottom 50% papers will be rejected based on consistent “reject” reviews and low scores:

  • For 1,000 submissions, with 50 PC members, this represents a maximum of 500 papers

For MICCAI 2019, the next step will be to select 15% of all submitted papers, from the remaining 30% of all submitted papers in the “grey area” (i.e., select one out of two remaining articles). Each AC will be assigned the remaining set of ~6 papers from two other ACs, in addition to their remaining ~6 papers emerging from the first round (i.e. initial 20 minus the top 20% and bottom 50%). This assignment of so-called borderline papers will be made by the Program Chairs. ACs will be asked to read the papers, the reviews, and the rebuttal, and recommend either acceptance or rejection at 50% level (to be confirmed by OC). For 1,000 submissions and 50 ACs, this represents 150 papers.

After having switched to the CMT, TPMS and Researcher.cc systems for MICCAI 2017 and 2018, and after having collected significant data before, during, and after the PC meetings, we will again replace the PC meeting by a set of dedicated teleconferences between the MICCAI 2019 OC and the ACs to inform about the overall results and statistics of the review process to discuss and resolve any arising issues, and to collect their feedback.

context

Stage 11: Oral Decision Process (1 week)

The goal of this step is to select the best submitted papers that justify an oral presentation.

Prior to this stage, the Program Chair will update the ACs of the number of orals at the conference. This will be determined by the Organizing Committee, in consultation with the MICCAI Board. For MICCAI 2019 the number of orals is currently set at approximately 35, plus spotlight talks from Young Investigator Award nominees (if not already selected for a full oral presentation). Of this number, the Organizing Committee reserves the right to identify the theme and composition of up to ~4 oral sessions out of a total of ~10. Moreover, there will minimally be 1 CAI oral session.

Three teleconferences will be held, in three time zones (GMT 8h) to discuss the outcome of the MICCAI review process, statistics and any arising issues, and to collect feedback and recommendations for any further improvements of the process.

context

Stage 12: Official Notification to Authors

The goal of this step is to inform authors about the outcome of the review process.

The Program Chair will issue the following, via email, to all authors:

  • Complete statistics for the process (ACs, number of submissions, number of reviewers, number of reviews, number of accepted papers, number of orals); and
  • The author’s acceptance (oral, poster) or rejection status for their paper.

A complete list of acceptance and orals will then be drafted for dissemination as a program. The Program Chairs will record each complaint and follow up.