-
Notifications
You must be signed in to change notification settings - Fork 418
Create two separate actions for R2 and S3 upload #170
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
cameel
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some comments need to be updated, but other than that the code seems fine, so feel free to merge whenever you're ready.
| - name: Wait for other instances of this workflow to finish | ||
| # It's not safe to run two S3 sync operations concurrently with different files | ||
| uses: softprops/turnstyle@v1 | ||
| with: | ||
| same-branch-only: no | ||
| poll-interval-seconds: 120 # in seconds |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For a moment I was worried that this might prevent the R2 sync from running concurrently with the S3 one. But looking at the source, the action only waits if the workflow name matches, so it should be fine. I originally expected that you were going to put both jobs in the same workflow, but if you did that, it would block, so good that you did not.
Still, after we merge, we should check to see if both are indeed running at the same time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be good to also update the comment I asked you to add in #169 (comment) to say that the actions are not in fact idempotent and that each script will only work on a fresh checkout.
|
As a side note, we should also pin those actions. Back when I wrote this we weren't doing that consistently, but now that's what we normally do. Looks like github even recently added an option to enforce that in PRs (Enforce SHA pinning) and we should enable it. But that's just FYI. It's been like that before and it's not strictly related to the PR, just something I noticed while reviewing it. I'll add this as an org TODO. |
With the previous PR (#169), the r2 mirror script can no longer run because the .git files are removed. In general, both scripts run in the same environment, causing the second script to repeat the timestamp updates, removal of hidden files, and creation of symlinks that the S3 version has already performed.
With this PR, we introduce two separate GitHub actions: one for R2 and one for S3.