Review before publication

[rougier]

Overall this look good but there are some minor things to be done before the actual publication.
I first post things that need to be done globally before posting for each individual entry.

For all entries:

• Remove under review
• Assign volume 6, issue 2
• Assign a unique number for each entry
  (Meta Issue for format conversion #1 to nayakReComprehensiveStudy #10 for entry and Format singhReMixupTraining #11 doe editorial such that editorial appears on top)

Submit a PR for the website (https://github.com/ReScience/rescience.github.io/blob/sources/01-read.md) such as to add an entry for the special issue. You can try to build locally the website if you want to see the result, but don't try to upload it (it's easy to mess it)

[rougier]

Editorial Looks good. Not sure the link to the source (bottom of first page is necessary) but it is up to you. There are now only 2 names that take two lines in the reviewer table, it would be nice to fix it (but it's fine if you leave it as it is).

Comments:

Replication of a computational study typically means running the same code, using the same input data, and then checking if the results are the same or at least “close enough” by some degree of numerical approximations.

This is what we call reproduction, while replication means writing a new code and trying to get the same result (see https://arxiv.org/abs/1802.03311 by @labarba and https://www.frontiersin.org/articles/10.3389/fninf.2017.00069/full by @rougier)

[rougier]

For all the entries, I think there are problems with fonts in almost all of them and link colors in some of them. Maybe a problem in finding the font while for the link color (in green), I think it is because of the use of the openreview/package.tex that messes up with the rescience template. Best would be not to use it.

[koustuvsinha]

Hi @rougier! Regarding the comment on the definition of replication, my understanding was from the definitions adopted by ACM (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5778115/). Not sure on which definitions should be the de-facto, maybe I can add the source?

Regarding the other changes, I will work on them this weekend.

[labarba]

For your information, the ACM will be reversing its definitions. They are implementing a terminology swap on about 300 already published papers!

See:
"Toward a compatible reproducibility taxonomy for computational and computing sciences", Michael A. Heroux, Lorena A. Barba, Manish Parashar, Victoria Stodden, Michela Taufer. (October 2018). 10.2172/1481626
OSTI Record

[labarba]

Also see"
"Reproducibility and Replicability in Science", National Academies of Sciences, Engineering, and Medicine. (May 2019). 10.17226/25303

[koustuvsinha]

Hi @rougier ! Sorry for the delayed response, was a bit busy this week! Changes I did according to the review:

• Added DOI's for all articles, updated the issue number
• Added a footnote regarding the updated definition of Replicability.
• Fixed the two names in Editorial
• Fixed the link color issue of the paper at Format ferlesReZeroShotKnowledge #3

I'll check the website now. I have kept the "Published date" empty for now, let me know which date I should use.

[koustuvsinha]

@rougier I have been able to compile the website locally. As per my understanding, after I publish the articles then I'll get the appropriate bib files to submit the PR in the website repository. Let me know if things look okay and I'll publish them. (I added the DOI's by only running process.py which I think only reserves the DOI's?)

[koustuvsinha]

Hi @rougier, any updates on this?

[rougier]

Sorry for the delay. We're unsynchronized :)

For the DOI it is a two steps process. First you pre-reserve the DOI and use it in the PDF and then you can do the actual upload to the server. Note There's a sandbox server that is incredibly useful to tested beforehands because once you upload something on Zenodo, you cannot remove it.

After the actual uploadong to Zenodo, you'll need to update the the bibtex file for the website and things should appear on it. Just tell me once done such that I can regenerate the website.

[koustuvsinha]

@rougier , I have added the publications to the website in this PR

[rougier]

Ok. I've curated (accepted) the entries on Zenodo. I'll rebuild the website once I've merged your PR. Then we can make the annoument of the special issue on twitter. What's your twitter handle ? Or do you want to make the annoucement (with cc @ReScienceEds such that I can retweet). Also, it would be good to make an annoucemnt on connectionist and comp neuro list. I can do it if you're not subscribed, just tell me.

[rougier]

Update: it's online.