OBO tokens: load the encryption key from a keystore and harden the crypto

[beanuwave]

Description

Category: Bug fix, Enhancement

The OBO key path was audited end to end. The encryption rewrite is the FIPS
trigger; the rest are weaknesses found alongside it.

Key changes

• Encryption rewrite. EncryptionDecryptionUtil built its cipher with a bare
  Cipher.getInstance("AES") and a key forced to 16 bytes via Arrays.copyOf
  (silently truncating or zero-padding). Now AES-256-GCM (random 12-byte IV,
  128-bit tag) with the key derived via HKDF-SHA256. A bare "AES" transformation
  resolves to the provider's default mode — ECB — which for data confidentiality
  violates NIST SP 800-38A^[2] (ECB is approved only for key wrapping,
  SP 800-38F^[3]). The mode was never spelled out or configurable, and
  BC FIPS permits the ECB primitive at runtime, so this had to be caught by review
  rather than by the provider.
• Entropy floor (SP 800-133r2^[5]). HKDF cannot create entropy, so a
  short encryption_key would yield a nominal AES-256 key with sub-112-bit
  strength. A FIPS guard rejects input keying material < 32 bytes and zeroes the
  IKM after derivation.
• Signing-key length check. Validates the decoded key bytes; measuring the
  Base64 string would over-count by ~4/3 and let a 384-bit key pass a 512-bit gate.
• Lazy, fail-closed init. OnBehalfOfAuthenticator initialises lazily and
  atomically: the constructor is inert, the first token request triggers init, and
  a failure logs once and declines. A bad key can no longer throw out of the
  constructor, propagate through config reload and retry forever. Mirrors
  ApiTokenAuthenticator.
• Keystore support. Signing and encryption keys can be loaded from a keystore
  (KeyUtils.loadKeyFromKeystore → PemKeyReader.loadSecretKeyFromKeystore)
  instead of inline config, keeping secrets out of cluster state. Relative
  *_keystore_path values resolve against the node config dir, consistent with
  other security file settings. This is SP 800-57^[6] key-at-rest
  hardening, not a FIPS 140-3^[1] requirement.
• Secret redaction (CWE-532). OnBehalfOfSettings.toString() redacts
  signing_key / encryption_key so they no longer leak into logs.

Reviewer call-outs

1. OBO token wire-format break. GCM-encrypted tokens are not interchangeable
   with old AES-ECB tokens across the upgrade boundary. Benign in practice (OBO TTL
   < 10 min, self-heals shortly after rollout), but worth a release note for
   hot/rolling deployments.
2. JKS cannot hold SecretKey entries (engineSetKeyEntry requires a
   PrivateKey), which is why the JWT-signing-key path needs BCFKS — the only
   FIPS-approved store that holds secret keys.

Testing

The suite runs in non-FIPS mode by default. To exercise the FIPS code paths, set the environment variable before invoking Gradle:

OPENSEARCH_FIPS_MODE=true ./gradlew test integrationTest

When set, the build swaps in the FIPS java.security policy (BCFIPS-only providers), enables -Dorg.bouncycastle.fips.approved_only=true, and points the JVM at the BCFKS truststore. FIPS-incompatible tests (BCrypt, Argon2, SAML, SSLv3, JKS/PKCS12, weak/short passwords) are auto-skipped via JUnit assumptions. Static bcrypt fixtures and their short demo passwords are rewritten to PBKDF2 and padded past the 14-char floor by FipsHashAdapter (a no-op outside FIPS), and a few timing-sensitive integ tests scale down under FIPS, where PBKDF2 logins and BCTLS handshakes are markedly slower.

For a running cluster, select the FIPS-approved password hasher in opensearch.yml (BCrypt/Argon2 are not available in approved-only mode):

plugins.security.password.hashing.algorithm: pbkdf2

The demo hashes in config/opensearch-security/internal_users.yml are BCrypt, which won't verify under PBKDF2 - regenerate the hash for each test account (e.g. with tools/hash.sh) and replace it before applying the security config.

Test OnBehalfOf (OBO) token

Exercises OBO token issuance and verification against an already-running cluster, in three modes: keys inline in the dynamic config (Scenario A), held in a BCFKS keystore out of cluster state (Scenario B), or in a PKCS#12 keystore for non-FIPS builds (Scenario C). No restart needed - -t config pushes only the dynamic on_behalf_of block, picked up live. Pick ONE scenario, edit config.yml, then run the apply / issue / use steps. All paths are relative to $OPENSEARCH_HOME.

cd $OPENSEARCH_HOME
export ADMIN_AUTH="admin:<admin-password>"

# === Apply / issue / use (the verify loop - run after editing config.yml) ====
# apply: push the dynamic config (-t config = on_behalf_of block only, live reload).
sh ./plugins/opensearch-security/tools/securityadmin.sh \
  -f ./config/opensearch-security/config.yml \
  -t config \
  -icl \
  -nhnv \
  -cacert config/root-ca.pem \
  -cert config/kirk.pem \
  -key config/kirk-key.pem \
  -h localhost \
  -p 9200

# issue: generate a token.
export OBO_TOKEN=$(curl -sk \
  -u "$ADMIN_AUTH" \
  -X POST \
  -H 'Content-Type: application/json' \
  https://localhost:9200/_plugins/_security/api/generateonbehalfoftoken \
  -d '{
        "description": "obo test",
        "service": "test-service",
        "durationSeconds": "300"
      }' | jq -r '.authenticationToken')
echo "$OBO_TOKEN"

# use: a populated user_name / roles proves the verify side loaded the key,
#      checked the signature, and decrypted the roles.
curl -sk \
  -H "Authorization: Bearer $OBO_TOKEN" \
  https://localhost:9200/_plugins/_security/authinfo?pretty

# === Scenario A - inline keys ================================================
# signing_key >= 512 bits (64 bytes) for HS512; encryption_key >= 256 bits
# (32 bytes) for the FIPS IKM floor.
export OBO_SIGNING_KEY=$(openssl rand 64 | base64 -w0)
export OBO_ENCRYPTION_KEY=$(openssl rand 32 | base64 -w0)

# Put under config.dynamic.on_behalf_of in config/opensearch-security/config.yml:
#   on_behalf_of:
#     enabled: true
#     signing_key: "<value of $OBO_SIGNING_KEY>"
#     encryption_key: "<value of $OBO_ENCRYPTION_KEY>"
#
# Negative checks: a too-short encryption_key (e.g. ZW5jcnlwdGlvbktleQ==, 13 bytes)
# is declined in FIPS mode ("encryption_key is not strong enough for FIPS mode");
# tampering with the token's last segment makes the `use` step return no credentials.

# === Scenario B - BCFKS keystore =========
keytool \
  -genseckey \
  -alias obo-signing \
  -keyalg HmacSHA512 \
  -keysize 512 \
  -storetype BCFKS \
  -providername BCFIPS \
  -keystore config/obo.bcfks \
  -storepass kspass \
  -keypass keypass \
  -providerClass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider \
  -providerPath lib/bc-fips-2.1.2.jar

keytool \
  -genseckey \
  -alias obo-enc \
  -keyalg AES \
  -keysize 256 \
  -storetype BCFKS \
  -providername BCFIPS \
  -keystore config/obo.bcfks \
  -storepass kspass \
  -keypass keypass \
  -providerClass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider \
  -providerPath lib/bc-fips-2.1.2.jar

# Reference the keystore in config.yml (no inline keys). <key>_keystore_path is
# config-dir-relative.
#   on_behalf_of:
#     enabled: true
#     signing_key_keystore_path: "obo.bcfks"
#     signing_key_keystore_type: "BCFKS"
#     signing_key_keystore_alias: "obo-signing"
#     signing_key_keystore_password: "kspass"
#     signing_key_keystore_key_password: "keypass"
#     encryption_key_keystore_path: "obo.bcfks"
#     encryption_key_keystore_type: "BCFKS"
#     encryption_key_keystore_alias: "obo-enc"
#     encryption_key_keystore_password: "kspass"
#     encryption_key_keystore_key_password: "keypass"
#
# Same result as Scenario A, but no key material in the config index. GET
# /_plugins/_security/api/securityconfig shows only keystore references.

# === Scenario C - PKCS#12 keystore (NON-FIPS builds only) ====================
# A FIPS build rejects PKCS#12 (no FIPS-approved PKCS#12 in BC FIPS). PKCS#12 has
# no per-entry passwords, so use one value for -storepass and -keypass.
keytool \
  -genseckey \
  -alias obo-signing \
  -keyalg HmacSHA512 \
  -keysize 512 \
  -storetype PKCS12 \
  -keystore config/obo.p12 \
  -storepass kspass \
  -keypass kspass

keytool \
  -genseckey \
  -alias obo-enc \
  -keyalg AES \
  -keysize 256 \
  -storetype PKCS12 \
  -keystore config/obo.p12 \
  -storepass kspass \
  -keypass kspass

# Reference it in config.yml as in Scenario B with the PKCS#12 path/type (the
# loader falls back to the keystore password when _keystore_key_password is absent).
#   on_behalf_of:
#     enabled: true
#     signing_key_keystore_path: "obo.p12"
#     signing_key_keystore_type: "PKCS12"
#     signing_key_keystore_alias: "obo-signing"
#     signing_key_keystore_password: "kspass"
#     encryption_key_keystore_path: "obo.p12"
#     encryption_key_keystore_type: "PKCS12"
#     encryption_key_keystore_alias: "obo-enc"
#     encryption_key_keystore_password: "kspass"

Check List

• New functionality includes testing
• New functionality has been documented
• New Roles/Permissions have a corresponding security dashboards plugin PR
• API changes companion pull request created
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit 5bd14ff.

⛔ Hard block: Issues at High severity or above will block this PR from merging.

Path	Line	Severity	Description
build.gradle	77	high	New build script 'gradle/fips.gradle' applied via 'apply from' — injects JVM arguments and modifies test task scope across all subprojects. Mandatory flag for new build plugin/script additions.
build.gradle	601	high	New dependency 'org.bouncycastle:bctls-fips' added to integrationTestImplementation. Mandatory flag for all dependency additions — verify artifact authenticity.
build.gradle	812	high	New dependency 'org.bouncycastle:bctls-fips' added to testImplementation. Mandatory flag for all dependency additions.
build.gradle	711	high	New 'org.bouncycastle:bctls-fips' dependency added to compileOnly scope and bouncycastle dependency scope restructured from conditional (FipsBuildParams) to unconditional. Mandatory flag for all dependency additions.
build.gradle	854	high	New 'configureSecurityAdminBcFips' function adds 'org.bouncycastle:bctls-fips' to distribution bundles via detached configuration. Mandatory flag for all dependency additions.
src/test/resources/fips-jvm-truststore.bcfks	1	medium	Binary BCFKS trust store added to the repository. Its trusted certificate contents cannot be reviewed in this diff — a malicious CA certificate embedded here would be silently trusted by all FIPS test workers.
src/main/java/org/opensearch/security/support/FipsMode.java	20	medium	FipsMode.envSupplier is a public static mutable field, allowing any code at runtime to replace the FIPS mode detector (e.g., to force isEnabled() to return false), bypassing FIPS enforcement checks downstream.
gradle/fips.gradle	73	medium	javax.net.ssl.trustStorePassword=changeit is hardcoded in plaintext as a JVM argument applied to all test workers. While this is a well-known default, embedding credentials in build scripts sets a risky precedent and could affect production-adjacent configurations.
src/main/java/org/opensearch/security/auth/ldap2/SNISettingTLSSocketFactory.java	47	low	ThreadLocal hostname storage is package-accessible via static getHostname() and clearContext() methods. If thread-pool reuse occurs before clearContext() is called (e.g., on exception paths), a stale hostname could be used for SNI in a subsequent unrelated connection.
src/test/resources/fips_java_test.security	1	low	New JVM security policy file that overrides the full provider list with == (complete replacement). Removes SunJCE entirely and installs BouncyCastle FIPS as the primary crypto provider. Changes to security policy files warrant review to confirm no trusted provider is silently removed or replaced.

The table above displays the top 10 most important findings.

Total: 10 | Critical: 0 | High: 5 | Medium: 3 | Low: 2

---

Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.

---

⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.