Add web_identity chain type for IRSA support

[mst]

Adds a web_identity credential chain that uses
STSAssumeRoleWebIdentityCredentialsProvider to support IAM Roles for Service Accounts (IRSA) in EKS.

The no-arg constructor reads AWS_ROLE_ARN,
AWS_WEB_IDENTITY_TOKEN_FILE, and AWS_ROLE_SESSION_NAME from environment variables. Auto-refresh is enabled by default since these are temporary STS credentials.

Fixes #119, fixes #93, fixes #31

note: automatic refresh only works on un-globbed s3 access duckdb/duckdb-httpfs#165
note2: auto-refresh doesn't work with Ducklake or Iceberg at all: duckdb/duckdb-httpfs#165 (comment)

[mohdikramsteeleye]

The PR is open for weeks now
Any ETA on it being merged??

[isarns]

Hey @samansmink, hope you’re doing well.

We’d really appreciate it if you could merge this PR when you have a chance.

[HaoXuAI]

@samansmink, @Tmonster @ccfelius appreciates as well

[valenzuelaomar]

@atolopko-czi and @prathapsridharan appreciates as well

[arnoldwakim]

cc @zmarouf 🥹

[fredbrowne]

Any updates on when this is going to be merged ? Appreciate this.

[Tmonster]

Is there a way to test this? I see a test file has been added, but no AWS_WEB_IDENTITY_TOKEN_FILE has been shared to validate credentials are created.
Some steps to get such a file within an AWS console would be appreciated, thanks

[mst]

duckdb-oidc-test.sh

@Tmonster , you'll need an OIDC provider and and setup a trust policy for that provider, add a role with this trust policy and assume it. The attached script sets up AWS Cognito as OIDC provider, and creates the role. You can use this to run local, one-off tests with this.

./duckdb-oidc-test.sh setup

eval "$(./duckdb-oidc-test.sh env)"

./build/release/test/unittest "test/sql/env/aws_secret_web_identity_env.test"

./duckdb-oidc-test.sh cleanup

Instead of Cognito, you could also run this with GitHub as OIDC provider within GitHub Workflow. If this is what you're aiming at, I can help with that; but it'd be a test-time dependency to AWS.

edit: GitHub workflow poc: mst#1

[Tmonster]

Hi @mst,

I took a look. It looks good to me. Maybe you can cherry-pick the commit I made here?
Tmonster@8841e9a

I thought it might be helpful for users to be able to manually specify the web_identity_tokenfile,assume_role_arn, and session_name parameters in case they want to create multiple secrets in one DuckDB session based on different web providers.

[mst]

@Tmonster - thanks, makes sense. Cherry pick applied

[Tmonster]

Thanks! I'll put this in CI in a follow up PR

[mst]

Thanks! I'll put this in CI in a follow up PR

Awesome, thanks for merging! Regarding CI, not sure if you've seen my comment -- there's a slightly simpler approach using GitHib as OICD provider instead of Cognito (saves some infra for this test): see mst#1.

[Tmonster]

Thanks! I'll put this in CI in a follow up PR

Awesome, thanks for merging! Regarding CI, not sure if you've seen my comment -- there's a slightly simpler approach using GitHib as OICD provider instead of Cognito (saves some infra for this test): see mst#1.

I did see this, but did not take close enough look. I will fork/cherry-pick and see if I can get it working. Might tag you as a reviewer if that's alright

[Tmonster]

CC @carlopi wondering if this might have any implications for duckdb-wasm

[HaoXuAI]

awesome work!

[Charmatzis]

the docs should also be updated, right?