Will the data shared by PC chairs be private?
Data shared by PC chairs with CLOSET is more private than any review management system (RMS). This is due to the following reasons. First, CLOSET is developed and maintained by only one person. Only he can see the data sent by PC chairs. Typically, any industrial-strength RMS has a team of individuals to manage the system and hence multiple individuals may see sensitive data of a venue. Second, unlike existing RMS, CLOSET deliberately does not have any web interface. Hence, there is no possibility of privacy or security breaches through such interface. Third, we can only view sensitive data of reviewer assignments and submission titles/authors in CLOSET. It does not demand access to review and discussion data. Hence, CLOSET sees significantly less information associated with a review proces than any RMS. In summary, if one is concerned about privacy issues in CLOSET then they should also be concerned about privacy of data in any existing RMS.
Why is CLOSET not integrated with commercial RMS such as CMT?
Good news! CLOSET output can now be ingested by CMT for automatic reviewer assignment (please click here for details). To this end, PC chairs need to request CMT to enable ingestion of COI detected by CLOSET. CMT will then enable an interface for uploading the CLOSET COI file. CMT then uses the COI detected by CLOSET along with COIs detected by CMT and author-specified COIs for reviewer assignment. We have tested it in recent conferences and the CLOSET-CMT interface is working well.
No such integration is possible yet with other RMS (e.g., EasyChair).
Note that CLOSET is not integrated within a RMS. Hence, relevant submission details still need to be exported to CLOSET before the output can be uploaded to CMT. Any integration within an RMS is a decision that lies solely on the RMS providers and not us. CLOSET is built in NTU. We are happy to work towards this integration goal if an RMS is willing to satisfy NTU's legal and financial requirements for shifting out software to a commercial entity.
Do PC chairs incorporate all recommendations made by CLOSET during reviewer assignment?
CLOSET is an enabling technology to support PC chairs to make fairer reviewer assignments. We generate an array of reports and recommendations to this end and our work ends here. PC chairs incorporate these recommendations at their own discretion. Note that PC chairs may request to run CLOSET multiple times to refine the assignments (manually) iteratively based on the recommendations from it. Most of the times we are not privy to the final assignments made by PC chairs.
Is there a publicly-available technical report/paper on the internals of CLOSET?
CLOSET is a complex data-driven system with several components working together toward the goal of fairer review process. The internals of many of these components are not disclosed in order to maintain privacy and security of the software. In the future, we may publish a high-level architecture of CLOSET but implementation details of various components will not be published. It is also infeasible to publish a high-quality paper on these components without revealing experimental results on highly-sensitive data.
How can I contribute to CLOSET?
Under the hood, we are addressing some very interesting problems at the intersection of psychology and data-centric computing in order to enhance fairness and quality of review process. If you are interested in solving some of these problems or you have a solution that you may think will benefit CLOSET please contact us.
CLOSET takes significant effort to build and manage especially with evolving requirements from PC chairs of different venues. If you wish to donate financially to support the cause of CLOSET, feel free to contact us :).