GCSFileSystem requires gcp extra at lookup time while S3FileSystem does not

[hjtran]

There's an inconsistency in how FileSystems.get_filesystem() handles missing optional dependencies between GCS and S3.

Current Behavior

S3 (without aws extra):

>>> from apache_beam.io import filesystems
>>> filesystems.FileSystems.get_filesystem("s3://blah")
<apache_beam.io.aws.s3filesystem.S3FileSystem at 0x11a0af750>

Returns the filesystem object; validation happens later when the filesystem is actually used.

GCS (without gcp extra):

>>> from apache_beam.io import filesystems
>>> filesystems.FileSystems.get_filesystem("gcs://blah")
ValueError: Unable to get filesystem from specified path, please use the correct path or ensure the required dependency is installed, e.g., pip install apache-beam[gcp]. Path specified: gcs://blah

Raises immediately because GCSFileSystem isn't registered as a subclass.

Proposed Behavior

Both should behave consistently. GCSFileSystem should be returned from get_filesystem() like S3FileSystem, allowing callers to validate dependencies when the filesystem is actually used rather than at lookup time.

Why This Matters

• Inconsistent API behavior is confusing
• Code that handles multiple filesystem types can't catch/handle GCS gracefully
• Dependency validation at usage time (not lookup time) allows for better error handling and lazy loading patterns

Environment

• Apache Beam version: 2.70.0
• Python version: 3.11

---

Generated by Claude Code, confirmed by @hjtran

[hjtran]

@damccorm do you have an opinion on this? I'd like a second opinion on the proposed behavior before someone works on it. The fix will likely just be lazy loading gcsio in gcsfilesystem.py

[damccorm]

SGTM - I agree with the proposed behavior change and with the fix you're suggesting

[RushabhRatnaparkhi]

Hi @hjtran , I want to work on this issue if it is ok.

[hjtran]

Feel free to!

…

On Thu, Jan 29, 2026, 10:56 PM Rushabh Ratnaparkhi ***@***.***> wrote: *RushabhRatnaparkhi* left a comment (apache/beam#37445) <#37445 (comment)> Hi @hjtran <https://github.com/hjtran> , I want to work on this issue if it is ok. — Reply to this email directly, view it on GitHub <#37445 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACQJSFEMWP5KE3QBEE2CCL34JLI6HAVCNFSM6AAAAACTJYFQJSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTQMRRGY3DSMRQGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[hjtran]

@RushabhRatnaparkhi , you can comment .take_issue to assign the issue to yourself

[Hridayyy1]

Thanks for confirming! I went through the issue and the filesystem lookup logic to understand the GCS vs S3 inconsistency. Since someone else is already working on this, I’ll step back, but I’m happy to help review or test the fix if needed. I’ll also look out for related issues. Thanks!

[RushabhRatnaparkhi]

.take_issue

[Simarjeet112]

Hi, I’d like to work on this issue as a new contributor.
.take-issue

[hjtran]

Hi @Simarjeet112 , it looks like a PR has already been put up for this, see #37488

[MansiSingh17]

Hi, I’m Mansi Singh, prospective GSoC 2026 contributor interested in Kafka Streams Runner.
I’d like to work on this issue if no one else is assigned. Thanks!

[shunping]

Hi @hjtran. I just came across this issue and saw a lot of interests here (which is great!).

I would like to understand the rationale behind the proposed behavior a bit more. To me, I'm more inclined toward the 'fail-fast' approach of GCSFileSystem, which provides immediate feedback if a required dependency is missing.

Could you elaborate on your specific use case? Particularly, why deferring validation to usage time is preferable for your scenario?

Proposed Behavior
Both should behave consistently. GCSFileSystem should be returned from get_filesystem() like S3FileSystem, allowing callers to validate dependencies when the filesystem is actually used rather than at lookup time.

[hjtran]

Hi @hjtran. I just came across this issue and saw a lot of interests here (which is great!).

I would like to understand the rationale behind the proposed behavior a bit more. To me, I'm more inclined toward the 'fail-fast' approach of GCSFileSystem, which provides immediate feedback if a required dependency is missing.

Could you elaborate on your specific use case? Particularly, why deferring validation to usage time is preferable for your scenario?

Proposed Behavior
Both should behave consistently. GCSFileSystem should be returned from get_filesystem() like S3FileSystem, allowing callers to validate dependencies when the filesystem is actually used rather than at lookup time.

Hey @shunping
For my use case, it's optional whether or not the GCS dependencies are installed. I want to be able to do something like if Filesystems.get_filesystem(user_provided_path) is GCSFileSystem: raise RuntimeError("You haven't installed GCS dependencies!").

I can do this today with S3Filesystem, but not with GCSFilesystem. I can do a manual if path.startswith("gcs"), but this is a bit awkward especially since S3 doesn't require it