Skip to content

SBP-487: workflow input staging - #131

Closed
vtnphan wants to merge 13 commits into
devfrom
sbp-487
Closed

SBP-487: workflow input staging#131
vtnphan wants to merge 13 commits into
devfrom
sbp-487

Conversation

@vtnphan

@vtnphan vtnphan commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Pull Request

Summary

SBP-487 Stages all workflow input files from S3 to Gadi via Globus before a run executes, instead of assuming Nextflow can read S3 URIs directly on Gadi.

Changes

  • New Globus service (globus_client.py, globus_transfer.py, globus_errors.py, DataTransfer model): submits/polls Globus transfers per run and gates execution — a queued job stays "staging" until every input DataTransfer reports "completed".
  • Per-workflow staging (routes/workflows.py): the samplesheet is always staged; bindcraft's starting_pdb, proteinfold's fasta, and WISPS's aggregated FASTA each get their own additional transfer, with samplesheet columns rewritten from S3 URIs to local Gadi paths where needed.
  • Executors (bindflow_, proteindj_, proteinfold_, wisps_executor.py): consume the already-staged local path instead of building an S3 URI; prerun scripts no longer carry AWS credentials — WISPS's now receives its staged FASTA path/split dir as inline shell vars instead.
  • Scheduler: new sync_data_transfers job (every 2 min) drives submission/polling.
  • DataTransfer.provider_metadata (JSON) replaced with reusing transfer_id for both the pre-submission Globus submission id and the post-submission task id (removes a redundant column).
  • New migration for the above; .env.example documents the new Globus env vars.

How to Test

.venv/bin/python -m pytest -q
.venv/bin/ruff check app tests
.venv/bin/python -m mypy app --ignore-missing-imports

Manually: launch a run, confirm its DataTransfer row(s) reach Globus SUCCEEDED, the job flips "staging""pending", and staged files land on Gadi as local paths (not s3://).

Type of change

  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Checklist

  • I have added tests that prove my fix is effective or that my feature works
  • I have added or updated documentation where necessary
  • I have run linting and unit tests locally
  • The code follows the project's style guidelines

@vtnphan vtnphan changed the title Sbp 487 SBP-487: workflow input staging Aug 14, 2026
@vtnphan
vtnphan marked this pull request as ready for review August 14, 2026 01:10
@marius-mather

Copy link
Copy Markdown
Collaborator

Closing, will be merged via #132

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants