{"id":3302,"date":"2023-12-07T20:02:24","date_gmt":"2023-12-07T20:02:24","guid":{"rendered":"https:\/\/www.wp.fccm.org\/?page_id=3302"},"modified":"2024-01-31T23:23:27","modified_gmt":"2024-01-31T23:23:27","slug":"artifact-evaluation-2024","status":"publish","type":"page","link":"https:\/\/www.wp.fccm.org\/artifact-evaluation-2024\/","title":{"rendered":"Artifact Evaluation 2024"},"content":{"rendered":"\n<h1 class=\"wp-block-heading\"><strong>ARTIFACT EVALUATION<\/strong><\/h1>\n\n\n\n<p>FCCM will again offer authors the opportunity to <strong><em>optionally<\/em><\/strong> participate in an artifact evaluation process.&nbsp; Artifacts are digital objects that were created by the authors as part of the research or experiments performed with the submitted work.&nbsp; Examples of artifacts are:<\/p>\n\n\n\n<ul>\n<li><strong>Software<\/strong>: Source code, scripts, Makefiles, container images (like Docker files), etc.<\/li>\n\n\n\n<li><strong>Hardware<\/strong>: Verilog, VHDL, schematics, CAD tools, flows, etc.<\/li>\n\n\n\n<li><strong>Data<\/strong>: Spreadsheets, databases, binary files, design sets, etc.<\/li>\n<\/ul>\n\n\n\n<p>The goal of submitting artifacts promotes the availability and reproducibility of the experimental results and data such that other researchers can repeat experiments and replicate results with less effort.&nbsp;<\/p>\n\n\n\n<p><strong>The TRETS journal track submissions will also be able to participate in artifact evaluation through TRETS.&nbsp;<\/strong><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>SUBMISSION REQUIREMENTS<\/strong><\/h2>\n\n\n\n<p>If the authors would like to participate in artifact evaluation and are willing to prepare and document artifacts, they need to fill out and submit a copy of the Artifact Form alongside their paper when submissions are due.&nbsp;<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>What is the Artifact Form?<\/strong><\/h3>\n\n\n\n<p>The Artifact Form is a form used to collect information necessary for artifact evaluation.&nbsp; The form allows the authors to describe the presence or absence of artifacts, and their type (software, hardware, or data) that supports the research presented in the paper.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Do I need to open-source my software in order to complete the Artifact Form?<\/strong><\/h3>\n\n\n\n<p>No. You are not asked to make any changes to your computing environment or design process in order to complete the form. The form is meant to describe the computing environment in which you produced your results and any artifacts you wish to share. Any author-created software does not need to be open source unless you wish to be eligible for an artifact review badge.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>REVIEW PROCESS<\/strong><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Who will review my artifact form?<\/strong><\/h3>\n\n\n\n<p>If your submission is accepted as a full paper for publication at FCCM 2024, then the Artifact form will be reviewed by the Artifact Evaluation Chairs. The Artifact Evaluation Committee (AEC) will review the information provided and will verify artifacts are indeed available at the URLs provided. They will also help authors improve their forms, in a double-open arrangement. If authors select this option, their paper may be evaluated for artifact review badges.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>How will the review of artifacts interact with the double-blind review process?<\/strong><\/h3>\n\n\n\n<p>Artifact review will not take place until after decisions on papers have been made. Reviewers will not have access to the Artifact Form. Authors should not include links to their artifacts\/repositories in their submitted paper. The paper review process is double-blind. The artifact review process is not.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>IMPACT OF ARTIFACT EVALUATION<\/strong><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>What\u2019s the impact of an Artifact Form on scientific reproducibility?<\/strong><\/h3>\n\n\n\n<p>An artifact-evaluation effort can increase the trustworthiness of computational results. It can be particularly effective in the case of results obtained using specialized computing platforms, not available to other researchers. Leadership computing platforms, novel testbeds, and experimental computing environments are of keen interest to the FPGA community. Access to these systems is typically limited, however. Thus, most reviewers cannot independently check results, and the authors themselves may be unable to recompute their own results in the future, given the impact of irreversible changes in the environment (compilers, libraries, components, etc.). The various forms of Artifact Evaluation improve confidence that computational results from these special platforms are correct.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>The paper text explains why I believe my answers are right and shows all my work. Why do I need an Artifact Evaluation?<\/strong><\/h3>\n\n\n\n<p>There are many good reasons for formalizing the artifact description and evaluation process. Standard practice varies across disciplines. Labeling the evaluation as such improves our ability to review the paper and improves reader confidence in the veracity of the results.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>ARTIFACTS<\/strong><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>What are \u201cauthor-created\u201d artifacts and why make the distinction?<\/strong><\/h3>\n\n\n\n<p>Author-created artifacts are the hardware, software, or data created by the paper\u2019s authors. Only these artifacts need to be made available to facilitate evaluation. Proprietary, closed-source artifacts (e.g. commercial software and CPUs) will necessarily be part of many research studies. These proprietary artifacts should be described to the best of the author\u2019s ability but do not need to be provided.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>What about proprietary author-created artifacts?<\/strong><\/h3>\n\n\n\n<p>The ideal case for reproducibility is to have all author-created artifacts publicly available with a stable identifier. Papers involving proprietary, closed-source author-created artifacts should indicate the availability of the artifacts and describe them as much as possible. Note that results dependent on closed-source artifacts are not reproducible and are therefore ineligible for some artifact review badges.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Are the numbers used to draw our charts a data artifact?<\/strong><\/h3>\n\n\n\n<p>Not necessarily. Data artifacts are the data (input or output) required to reproduce the results, not necessarily the results themselves. For example, if your paper presents a system that generates charts from datasets then providing an input dataset would facilitate reproducibility. However, if the paper merely uses charts to elucidate results, the input data to whatever tool you used to draw those charts aren\u2019t required to reproduce the paper\u2019s results. The tool which drew the chart isn\u2019t part of the study, so the input data to that tool is not a data artifact of this work.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Help! My data is HUGE! How do I make it publicly available with a stable identifier?<\/strong><\/h3>\n\n\n\n<p>Use Zenodo (<a href=\"https:\/\/help.zenodo.org\/\">https:\/\/help.zenodo.org<\/a>). Contact them for information on how to upload extremely large datasets. You can easily upload datasets of 50GB or less, have multiple datasets, and there is no size limit on communities.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>ARTIFACT REVIEW BADGES<\/strong><\/h2>\n\n\n\n<p>The artifact evaluation process for FCCM 2024 will consider awarding the following IEEE reproducibility badges in two different categories:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh7-us.googleusercontent.com\/7fSNXJ2YY6f-RZko6l_5iFwq_8NHGUpL24kly2pSezgfqt_m6HARS9mJuVrff3MSrEIriC--V95Qjv1IS6McrxuYVJ8GZE6DQL7DM5oCiCBAcBHfxazJkgsj1fRyWLFuy_gnIetlK3VXPEqT-FcHmgg\" alt=\"\"\/><\/figure>\n\n\n\n<p>Each paper will be considered for only one of the badging categories (Code or Dataset) and can earn up to 3 badges (Available, Reviewed, and Reproducible).&nbsp;<\/p>\n\n\n\n<p>A notable difference from last year\u2019s offering is that Code Badges will require a published DOI from <a href=\"https:\/\/codeocean.com\/\">IEEE\u2019s Code Ocean<\/a>, which necessitates that <a href=\"https:\/\/help.codeocean.com\/en\/articles\/1120151-code-ocean-s-verification-process-for-computational-reproducibility-quality\">the code runs in a \u2018capsule\u2019 (container) and produces expected results<\/a>.&nbsp; Authors publishing with code artifacts that are <a href=\"https:\/\/help.codeocean.com\/en\/articles\/3294415-what-is-and-is-not-a-good-fit-for-publishing-and-sharing-on-code-ocean\">incompatible<\/a> with the <a href=\"https:\/\/help.codeocean.com\/en\/articles\/1120508-what-machine-will-my-code-run-on\">Code Ocean environment<\/a>, will be evaluated for the Dataset Badges despite their artifacts containing code.&nbsp; Below is a simple decision diagram to help authors understand if they should choose Code Ocean:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh7-us.googleusercontent.com\/gESY0ZNEGqYCHnvYZJzAfskogs9XwPwFku01-Le0c7XcPnd34ug_fv9q26sRQcbpeLFlmx0MGaPKbk6DpLSpWDBNCKWC02zJ7KP6N3MiGe91AO4itqu6R5HbGUNxSsWx5jzO5iwTWC80yrB5-ufCR08\" alt=\"\"\/><\/figure>\n\n\n\n<p>Please see IEEE\u2019s requirements for <a href=\"https:\/\/conferences.ieeeauthorcenter.ieee.org\/write-your-paper\/research-reproducibility\/#sharingdata\">Code and Dataset publishing<\/a> for more details.<\/p>\n\n\n\n<p>The badging definitions are based on <a href=\"https:\/\/ieeexplore.ieee.org\/Xplorehelp\/overview-of-ieee-xplore\/about-content#reproducibility-badges\">Reproducibility Badging at IEEE<\/a> and are synonymous with the 3 badges offered last year:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Code\/Dataset Available<\/strong><\/h3>\n\n\n\n<p>This badge signals that author-created digital objects used in the research (including data and\/or code) are permanently archived in a public repository that assigns a global identifier (DOI) and guarantees persistence, and are made available via standard open licenses that maximize artifact availability.<\/p>\n\n\n\n<p>Notes:<\/p>\n\n\n\n<p>1. This is akin to author-supplied supplemental materials, shared under a standard public license such as an Open Science Initiative (OSI)\u2013approved license for software or a Creative Commons license or public domain dedication for data and other materials.<\/p>\n\n\n\n<p>2. This definition corresponds to the Association for Computing Machinery (ACM) \u201cArtifacts Available\u201d badge, and to the combined Center for Open Science (COS) \u201cOpen Data\u201d and \u201cOpen Materials\u201d (pertaining to digital objects) badges.&nbsp;<\/p>\n\n\n\n<p>3. The determination of what objects are \u201crelevant\u201d to a research publication is in the hands of the editorial board or leadership members of the community, in addition to the authors themselves.<\/p>\n\n\n\n<p>4. For physical objects relevant to the research, the metadata about the object should be made available.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Code\/Dataset Reviewed<\/strong><\/h3>\n\n\n\n<p>This badge signals that all relevant author-created digital objects used in the research (including data and code) were reviewed according to the criteria provided by the badge issuer.&nbsp;<\/p>\n\n\n\n<p>Notes:<\/p>\n\n\n\n<p>1. This badge corresponds to the ACM \u201cArtifacts Evaluated\u201d badge, while the Institute of Electrical and Electronics Engineers (IEEE) has used a \u201cCode Reviewed\u201d badge.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Code\/Dataset Reproducible<\/strong><\/h3>\n\n\n\n<p>This badge signals that an additional step was taken or facilitated by the badge issuer (e.g., publisher, trusted third-party certifier) to certify that an independent party has regenerated computational results using the author\u2011created research objects, methods, code, and conditions of analysis.<\/p>\n\n\n\n<p>Results Reproduced assumes that the research objects were also reviewed.&nbsp;<\/p>\n\n\n\n<p>This Recommended Practice has adopted the National Academies of Sciences Engineering Medicine, definition of Reproducibility: \u201cWe define reproducibility to mean computational reproducibility\u2014obtaining consistent results using the same input data, computational steps, methods, code, and conditions of analysis.\u201d<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>FAQ<\/strong><\/h2>\n\n\n\n<p><strong>Q: I only want to make my artifacts public if the paper is accepted. Is that okay?<\/strong><\/p>\n\n\n\n<p>A: Yes. Artifacts will only be examined after papers are accepted. You can wait until you hear the decision on your paper before making your artifacts public, but you must complete the form by the deadline set; information in the form will not be used until after paper acceptance.<\/p>\n\n\n\n<p><strong>Q: Does the artifact submission link need to be anonymous?<\/strong><\/p>\n\n\n\n<p>A: No, it should not be anonymous and thus should not be included in the submitted paper.&nbsp; Any information in the paper must be anonymous to preserve the double-blind review.&nbsp; After a paper has been accepted, links to the appropriate artifacts, code repositories, etc., can be added to the camera-ready paper.&nbsp; The submitted Artifact Form is not anonymous and is reviewed independently and will not be examined until after paper decisions are made.<\/p>\n\n\n\n<p><strong>Q: What is the deadline for the submission of artifacts?<\/strong><\/p>\n\n\n\n<p>A: No one will look at the artifacts until your paper is accepted. We require the information available to start the evaluation process as soon as papers are accepted, hence the requirement to submit the form.<\/p>\n\n\n\n<p><strong>Q: Can I update the artifact evaluation form after paper acceptance?<\/strong><\/p>\n\n\n\n<p>A: You are free to make changes to the details of the artifacts up to the start of the evaluation and in consultation with your evaluator, but the initial submission should include as many details as possible to allow the evaluation process to be organized.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>ARTIFACT EVALUATION FCCM will again offer authors the opportunity to optionally participate in an artifact evaluation process.&nbsp; Artifacts are digital objects that were created by the authors as part of the research or experiments performed with the submitted work.&nbsp; Examples of artifacts are: The goal of submitting artifacts promotes the availability and reproducibility of the &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/www.wp.fccm.org\/artifact-evaluation-2024\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Artifact Evaluation 2024&#8221;<\/span><\/a><\/p>\n","protected":false},"author":581,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"_links":{"self":[{"href":"https:\/\/www.wp.fccm.org\/wp-json\/wp\/v2\/pages\/3302"}],"collection":[{"href":"https:\/\/www.wp.fccm.org\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.wp.fccm.org\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.wp.fccm.org\/wp-json\/wp\/v2\/users\/581"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wp.fccm.org\/wp-json\/wp\/v2\/comments?post=3302"}],"version-history":[{"count":5,"href":"https:\/\/www.wp.fccm.org\/wp-json\/wp\/v2\/pages\/3302\/revisions"}],"predecessor-version":[{"id":3346,"href":"https:\/\/www.wp.fccm.org\/wp-json\/wp\/v2\/pages\/3302\/revisions\/3346"}],"wp:attachment":[{"href":"https:\/\/www.wp.fccm.org\/wp-json\/wp\/v2\/media?parent=3302"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}