{"id":3302,"date":"2023-12-07T20:02:24","date_gmt":"2023-12-07T20:02:24","guid":{"rendered":"https:\/\/www.wp.fccm.org\/?page_id=3302"},"modified":"2024-01-31T23:23:27","modified_gmt":"2024-01-31T23:23:27","slug":"artifact-evaluation-2024","status":"publish","type":"page","link":"https:\/\/www.wp.fccm.org\/artifact-evaluation-2024\/","title":{"rendered":"Artifact Evaluation 2024"},"content":{"rendered":"\n
FCCM will again offer authors the opportunity to optionally<\/em><\/strong> participate in an artifact evaluation process. Artifacts are digital objects that were created by the authors as part of the research or experiments performed with the submitted work. Examples of artifacts are:<\/p>\n\n\n\n The goal of submitting artifacts promotes the availability and reproducibility of the experimental results and data such that other researchers can repeat experiments and replicate results with less effort. <\/p>\n\n\n\n The TRETS journal track submissions will also be able to participate in artifact evaluation through TRETS. <\/strong><\/p>\n\n\n\n If the authors would like to participate in artifact evaluation and are willing to prepare and document artifacts, they need to fill out and submit a copy of the Artifact Form alongside their paper when submissions are due. <\/p>\n\n\n\n The Artifact Form is a form used to collect information necessary for artifact evaluation. The form allows the authors to describe the presence or absence of artifacts, and their type (software, hardware, or data) that supports the research presented in the paper.<\/p>\n\n\n\n No. You are not asked to make any changes to your computing environment or design process in order to complete the form. The form is meant to describe the computing environment in which you produced your results and any artifacts you wish to share. Any author-created software does not need to be open source unless you wish to be eligible for an artifact review badge.<\/p>\n\n\n\n If your submission is accepted as a full paper for publication at FCCM 2024, then the Artifact form will be reviewed by the Artifact Evaluation Chairs. The Artifact Evaluation Committee (AEC) will review the information provided and will verify artifacts are indeed available at the URLs provided. They will also help authors improve their forms, in a double-open arrangement. If authors select this option, their paper may be evaluated for artifact review badges.<\/p>\n\n\n\n Artifact review will not take place until after decisions on papers have been made. Reviewers will not have access to the Artifact Form. Authors should not include links to their artifacts\/repositories in their submitted paper. The paper review process is double-blind. The artifact review process is not.<\/p>\n\n\n\n An artifact-evaluation effort can increase the trustworthiness of computational results. It can be particularly effective in the case of results obtained using specialized computing platforms, not available to other researchers. Leadership computing platforms, novel testbeds, and experimental computing environments are of keen interest to the FPGA community. Access to these systems is typically limited, however. Thus, most reviewers cannot independently check results, and the authors themselves may be unable to recompute their own results in the future, given the impact of irreversible changes in the environment (compilers, libraries, components, etc.). The various forms of Artifact Evaluation improve confidence that computational results from these special platforms are correct.<\/p>\n\n\n\n There are many good reasons for formalizing the artifact description and evaluation process. Standard practice varies across disciplines. Labeling the evaluation as such improves our ability to review the paper and improves reader confidence in the veracity of the results.<\/p>\n\n\n\n Author-created artifacts are the hardware, software, or data created by the paper\u2019s authors. Only these artifacts need to be made available to facilitate evaluation. Proprietary, closed-source artifacts (e.g. commercial software and CPUs) will necessarily be part of many research studies. These proprietary artifacts should be described to the best of the author\u2019s ability but do not need to be provided.<\/p>\n\n\n\n The ideal case for reproducibility is to have all author-created artifacts publicly available with a stable identifier. Papers involving proprietary, closed-source author-created artifacts should indicate the availability of the artifacts and describe them as much as possible. Note that results dependent on closed-source artifacts are not reproducible and are therefore ineligible for some artifact review badges.<\/p>\n\n\n\n Not necessarily. Data artifacts are the data (input or output) required to reproduce the results, not necessarily the results themselves. For example, if your paper presents a system that generates charts from datasets then providing an input dataset would facilitate reproducibility. However, if the paper merely uses charts to elucidate results, the input data to whatever tool you used to draw those charts aren\u2019t required to reproduce the paper\u2019s results. The tool which drew the chart isn\u2019t part of the study, so the input data to that tool is not a data artifact of this work.<\/p>\n\n\n\n\n
SUBMISSION REQUIREMENTS<\/strong><\/h2>\n\n\n\n
What is the Artifact Form?<\/strong><\/h3>\n\n\n\n
Do I need to open-source my software in order to complete the Artifact Form?<\/strong><\/h3>\n\n\n\n
REVIEW PROCESS<\/strong><\/h2>\n\n\n\n
Who will review my artifact form?<\/strong><\/h3>\n\n\n\n
How will the review of artifacts interact with the double-blind review process?<\/strong><\/h3>\n\n\n\n
IMPACT OF ARTIFACT EVALUATION<\/strong><\/h2>\n\n\n\n
What\u2019s the impact of an Artifact Form on scientific reproducibility?<\/strong><\/h3>\n\n\n\n
The paper text explains why I believe my answers are right and shows all my work. Why do I need an Artifact Evaluation?<\/strong><\/h3>\n\n\n\n
ARTIFACTS<\/strong><\/h2>\n\n\n\n
What are \u201cauthor-created\u201d artifacts and why make the distinction?<\/strong><\/h3>\n\n\n\n
What about proprietary author-created artifacts?<\/strong><\/h3>\n\n\n\n
Are the numbers used to draw our charts a data artifact?<\/strong><\/h3>\n\n\n\n
Help! My data is HUGE! How do I make it publicly available with a stable identifier?<\/strong><\/h3>\n\n\n\n