SC has been a leader in tangible progress towards scientific rigor, through its pioneering practice of enhanced reproducibility of accepted papers. The SC22 initiative builds on this success by continuing the practice of using appendices to enhance scientific rigor and transparency.
The Reproducibility Initiative impacts technical papers and their submission and peer review in the following ways:
- The Artifact Description (AD) Appendix will be auto-generated from author responses to a standard form embedded in the online submission system. It will have greater emphasis on linking to the broader products resulting from research. The AD is required but an indication that no artifacts were created or used is acceptable.
- For accepted papers, SC22 will evaluate and reward the availability and functionality of automated Artifact Evaluation (AE) so that reviewers can easily reproduce a paper’s key results and claims with provided artifacts. While authors are strongly encouraged to make all artifacts publicly available, AE is open to artifacts that are not. Accepted papers with available artifacts will be honored with the appropriate ACM badges.
The Reproducibility Initiative accomplishes its work through the following three subcommittees:
- Artifact Description/Artifact Evaluation Appendices Subcommittee
- Reproducibility Challenge Subcommittee
- Journal Special Issue Subcommittee
SC Reproducibility Subcommittees
This subcommittee has the following tasks within its charge:
- Manage and/or develop reproducibility FAQ and other resources as aid to authors during submission. Could include one-on-one assistance with prospective authors.
- Develop and publicly release a rubric for reviewing appendices. This release will occur prior to the paper submission deadline.
- Review the appendices of submitted papers according to the rubric. Provide a justified assessment to the PC who will incorporate this recommendation into their final determination.
- Nominate Best Reproducibility Advancement Award finalists and select the winner. See the Best Reproducibility Advancement Award page for details.
This subcommittee has the following tasks within its charge:
- Select a paper accepted to the previous conference to be used as source of the Reproducibility Challenge in the Student Cluster Competition (SCC) of the next conference;
- Work with the authors of the selected paper to build the challenge benchmark for the SCC teams;
- Continue working until the time of the conference in crafting the rules and requirements for the challenge, and ensure compatibility with the diverse hardware used by the SCC teams;
- Convene a subgroup to collect publish-quality reports from each student team who participated in the reproducibility challenge of SCC. The reports in this collection will have a citable DOI and be made available through an archive venue.
AD/AE Appendix Form: Its Role in a Paper and Its Review
All manuscripts submitted to the SC Technical Papers program must contain an AD Appendix. The AD Appendix describes the significant research products and other evidence needed for greater scientific rigor and transparency of the scientific conclusions of the manuscript. If a manuscript’s scientific findings have no significant supporting research products or require no additional evidence in support of its scientific conclusions, the AD Appendix can easily be generated to reflect this fact. That is, the AD form is mandatory but the author makes the determination whether additional evidence is needed for their particular scientific contribution.
The AE Appendix is optional but authors are strongly encouraged to provide AE resources and workflows that reviewers can easily evaluate for availability, functionality, and their support for claimed key results. Even though the AD/AE review process strongly encourages publicly available artifacts, it is open to artifacts that are not.
Role of AD/AE Appendix during review: Submissions to SC are double-blind reviewed. SC Papers Committee (PC) reviewers will have only the information in the AD/AE form available to them that does not compromise the double-blind protection.
The AD/AE review criteria are posted as part of AD/AE Appendix Process & Badges. The AE badges are the rubric as to how far an artifact has gone along in the reproducibility spectrum (see description of the badges).
Additional notes to authors:
- The online submission form accommodates both Artifact Description (required) and Artifact Evaluation (optional).
- The appendices are intended for supplemental material only. Specifically, the research or experimental methodology is critical to understanding the scientific contribution so must be included in the body of the paper.
- Please contact us with your questions.
AD/AE Appendix Process & Badges
The goal of the SC Reproducibility Committee is to encourage and promote reproducible research within the SC community. To that end, we aim to assist SC authors in providing us with the necessary documentation that describes your artifact and help us evaluate it so we can assign a badge to the artifact.
For full details on the AD/AE process including AD/AE criteria, submission timeline, a description of artifact badges, and tips on organizing your research object see:
Infrastructure for Artifact Evaluation
Authors of accepted papers who applied for artifact badges require infrastructure to share their artifacts with the AD/AE Committee for review. To facilitate artifact sharing, the SC22 Reproducibility Initiative is collaborating with Chameleon Cloud, CloudLab, Jetstream via XSEDE, SDSC, and TACC to provide such an infrastructure. Authors of submitted papers will be able to prepare their artifacts so that if their submissions get accepted, the AD/AE Committee can perform single-blind reviews of their artifact badge applications.
Reproducibility Infrastructure Webinar Series
The SC22 Reproducibility Initiative and the infrastructure collaborators are organizing a webinar for authors who are interested in using this infrastructure. The webinar will be recorded so authors in multiple time zones can view it and will be posted here.
Extreme-scale Scientific Software Stack (E4S)
Monday, March 14, 2022 at 9 am PDC (UTC-7)
Title: Using Containers for Reproducibility of Scientific Results for AD/AE Submission
Speaker: Sameer Shende, University of Oregon
Abstract: Containers have ushered in a new era of reproducibility of scientific results. This talk will focus on using and building custom containers that support CPU and GPU platforms on x86_64, ppc64le, and aarch64 with support for NVIDIA, Intel, and AMD GPUs. It will demonstrate techniques for deploying these base and full-featured images from E4S to build tools that can be used in SC22 papers for AD/AE submission.
Friday, March 18, 2022 at 7 am CST (UTC-5)
Title: Using Chameleon Cloud for Reproducibility for AD/AE Submission
Speakers: Isabel Brunkan and Michael Sherman, Chameleon Cloud
Abstract: Chameleon, an NSF-funded computer science and systems research testbed, provides bare metal reconfigurability options for CPUs, GPUs, and FPGAs, with consumer and enterprise level SSDs, NVMe, NVDIMM and RDMA storage options, InfiniBand networking, ARM ThunderX2 processors, an edge testbed with Raspberry Pis and Jetson Nanos and even more unique configurations available. This talk will cover the basics of interacting with Chameleon and reproducibility options including using the Jupyter Notebook integration to package experiments, a testbed-integrated repository to help you share and discover experiments, and Daypass, which allows you to share your experiment via a link or QR code.
Slides: Coming Soon
Monday, March 21, 2022 at 10 am EST (UTC-4)
Title: Jetstream2 – Accelerating Cloud Computing via Jetstream
Speaker: Jeremy Fischer, Indiana University
Abstract: Jetstream2 is a category I production cloud resource that is part of the National Science Foundation’s Innovative HPC Program. The project’s aim is to accelerate science and engineering by providing “on-demand” programmable infrastructure built around a core system at Indiana University and four regional sites. Jetstream2 is an evolution of the Jetstream platform, which functions primarily as an Infrastructure-as-a-Service cloud. The lessons learned in cloud architecture, distributed storage, and container orchestration have inspired changes in both hardware and software for Jetstream2. These lessons have wide implications as institutions converge HPC and cloud technology while building on prior work when deploying their own cloud environments. In this talk, we will cover the transition to Jetstream2, the available resources, accessing Jetstream2 resources, and how to use them.
History of the SC Reproducibility Initiative
Paper authors applied for up to three reproducibility badges (available, functional, reproducible). SC collaborated with NSF Cloud infrastructure providers for preparing and examining artifacts. Accepted papers were listed with their badges in the program Schedule. The SC Steering Committee approved the SC Best Reproducibility Advancement Award and members of the technical program and the reproducibility initiative committees selected an Award winner.
SC expanded the Initiative to transparency and reproducibility to reflect scientific rigor through disclosure particularly in research involving AI. The AD Appendix was streamlined for reduced researcher burden and to align with open science principles. An additional track undertook a formal survey of community sentiment about SC reproducibility with the objective of publishable results. The majority of survey participants who went through the AD/AE Appendices process expressed that they now think differently about theirs and others’ research after having gone through the process.
AD Appendices were mandatory for all submissions. AE Appendices were still optional, and both were submitted via a standard form in the conference submission system. Three new Technical Program tracks, with their respective committees and chairs, were introduced in support of the SC Reproducibility Initiative.
SC extended the option of submitting AD Appendices to Workshops and Posters. The CRA Appendix was renamed Artifact Evaluation (AE) Appendix, and limited to four pages. AD Appendices were limited to 2 pages and remained optional (but required for consideration as Best Paper/Best Student Paper, and also Best Poster/Best Student Poster).
SC made the AD Appendix a requirement to be considered for the Best Paper or Best Student Paper awards. SC17 also introduced the Computational Results Analysis (CRA) Appendix. 40% of submitted and 50% of accepted papers included an AD appendix; nine submitted papers (six accepted) included a CRA Appendix.
Authors submitting to the SC16 conference could optionally submit an AD Appendix: nine authors submitted one, three were finalists, and one was selected to become the source for the SC17 Student Cluster Competition Reproducibility Challenge.
The SC steering committee approved the reproducibility initiative. Authors of SC15 papers were invited to submit an AD Appendix after the conference: one paper did so, became the source for the SC16 Student Cluster Competition Reproducibility Challenge and the first SC paper to display an ACM badge.