Enforce FIPS-approved cryptography across hashing, TLS, tokens and the CLI

[beanuwave]

Description

Category: Enhancement, Bug fix

Completes FIPS support on top of the mode added in PR 1. This is where the plugin
actually becomes FIPS-capable.

Key changes

Keystores / TLS

• PemKeyReader rewritten onto BouncyCastle (PEMParser / JcaPEMKeyConverter /
  PKCS8 decryptor) instead of raw JCE; adds BCFKS and store-type auto-detection.
• SSLConfigConstants: both defaults become FIPS-conditional — default store type
  is forced to BCFKS in FIPS, and ALLOWED_SSL_PROTOCOLS drops TLSv1.1 in FIPS.
• The BC FIPS provider is declared, not instantiated. main self-registered it
  at plugin load (OpenSearchSecuritySSLPlugin.tryAddSecurityProvider() →
  Security.addProvider(new BouncyCastleFipsProvider())); that method is removed.
  Providers now come solely from the active java.security file (JCA lazy-loads
  them), so FIPS vs non-FIPS is a launch-time provider swap (BCJSSE vs SunJSSE)
  with no code branch — the security files are a core/distribution concern.

Auth hardening (found mid-audit)

• HTTPSpnegoAuthenticator: no longer mutates global System.setProperty debug
  flags; stops logging the acceptor principal; proper LoginContext.logout() and
  decoded-header zeroing in finally.
• InternalAuthenticationBackend + PasswordHasher.getDummyHash(): the not-found
  timing path now uses the configured hasher, closing a user-enumeration
  side-channel under PBKDF2.
• Password-length floor. PBKDF2 keys are derived from the password itself, and
  BC FIPS rejects key material under 112 bits (< 14 ASCII chars) at hashing time.
  PasswordValidator.FIPS_MIN_PASSWORD_LENGTH (14) anchors both ends: FIPS raises
  an unset restapi.password_min_length to 14, and startup rejects a lower
  explicit value — otherwise the REST API accepts passwords the hasher then
  refuses. validateFipsMode collects every violation and reports them together,
  and additionally verifies the BC provider is genuinely in approved-only mode.
• Randomness from core. Randomness.createSecure() replaces new SecureRandom()
  for OBO encryption, api-tokens and user passwords — it resolves to the approved
  SP 800-90A^[4] DRBG in FIPS. UserService generates 20–27 chars in
  FIPS (≥119 bits over the 62-char alphabet), 8–15 otherwise, with char[] zeroed
  in finally.

CLI

• SecurityAdmin accepts BCFKS/PKCS11, with a PKCS#11 PIN prompt.
  buildPkcs11SslContext routes PKCS#11 keys through SunJSSE client-side, the same
  reasoning as the server TLS layer in PR 2.
• Launcher scripts delegate to core's shared opensearch-cli.

Test support

• The 28 *FipsTests / *FipsIT variants that gradle/fips.gradle selects.
• FipsHashAdapter rewrites static BCrypt fixtures and their short demo passwords
  to PBKDF2, padded past the 14-char floor (a no-op outside FIPS).
• A few timing-sensitive integration tests scale down under FIPS, where PBKDF2
  logins and BCTLS handshakes are markedly slower.
• HTTP/3 is refused in FIPS mode because the bundled BoringSSL is not built from
  the FIPS-validated branch — a build-level constraint, not a categorical ban. A
  FIPS-certified BoringSSL substituted at the OS level re-enables it.

Reviewer call-outs

1. securityadmin now launches via core's opensearch-cli — this targets the
   in-distribution path; the standalone bundle is no longer self-launching and is
   effectively deprecated.
2. SAML is scoped out. Both SAML stacks in use (OneLogin java-saml and
   OpenSAML/Shibboleth) are not FIPS-compliant, so those tests do not run in FIPS
   mode.
3. Operational: keystore passwords must be ≥14 chars (BCFIPS), as must user
   passwords (the 112-bit PBKDF2 floor, now enforced).
4. The provider list is a complete override; SunJGSS is deliberately retained for
   Kerberos/SPNEGO.

Testing

The suite runs in non-FIPS mode by default. To exercise the FIPS code paths, set the environment variable before invoking Gradle:

OPENSEARCH_FIPS_MODE=true ./gradlew test integrationTest

When set, the build swaps in the FIPS java.security policy (BCFIPS-only providers), enables -Dorg.bouncycastle.fips.approved_only=true, and points the JVM at the BCFKS truststore. FIPS-incompatible tests (BCrypt, Argon2, SAML, SSLv3, JKS/PKCS12, weak/short passwords) are auto-skipped via JUnit assumptions. Static bcrypt fixtures and their short demo passwords are rewritten to PBKDF2 and padded past the 14-char floor by FipsHashAdapter (a no-op outside FIPS), and a few timing-sensitive integ tests scale down under FIPS, where PBKDF2 logins and BCTLS handshakes are markedly slower.

For a running cluster, select the FIPS-approved password hasher in opensearch.yml (BCrypt/Argon2 are not available in approved-only mode):

plugins.security.password.hashing.algorithm: pbkdf2

The demo hashes in config/opensearch-security/internal_users.yml are BCrypt, which won't verify under PBKDF2 - regenerate the hash for each test account (e.g. with tools/hash.sh) and replace it before applying the security config.

Test securityadmin.sh with BCFKS + PKCS#11 keystores (SoftHSM)

Exercises a token-resident node TLS key (signed via SunJSSE) and securityadmin authenticating with a PKCS#11 client key. This example is FIPS-specific, but adjusts easily to non-FIPS by registering a SunPKCS11 provider via OPENSEARCH_JAVA_OPTS="-Djava.security.properties=$OPENSEARCH_HOME/config/java.security" instead of OPENSEARCH_FIPS_MODE=true. All paths below are relative to $OPENSEARCH_HOME.

# 1. Set up a test cluster.
...
cd $OPENSEARCH_HOME
sh bin/opensearch-keystore create --password
sh plugins/opensearch-security/tools/install_demo_configuration.sh -y -i -s

# 2. Init token - the --pin becomes the keystore/truststore password.
softhsm2-util --init-token \
  --free \
  --label opensearch \
  --so-pin 4321 \
  --pin 1234

# 3. Register the provider in config/fips_java.security:
      security.provider.<n>=SunPKCS11 /path/to/config/softhsm-pkcs11.cfg
#    with softhsm-pkcs11.cfg in the same dir:
      name = SoftHSM
      library = /usr/lib/softhsm/libsofthsm2.so
      slotListIndex = 0

# 4. Import node + admin keys WITH chains (PKCS#12 -> token).
#    Repeat the keytool step for kirk (swap -name / -srcalias / -destalias).
openssl pkcs12 -export \
  -inkey config/esnode-key.pem \
  -in config/esnode.pem \
  -certfile config/root-ca.pem \
  -name esnode-cert \
  -out /tmp/esnode.p12 \
  -passout pass:1234

jdk/bin/keytool \
  -importkeystore \
  -srckeystore /tmp/esnode.p12 \
  -srcstoretype PKCS12 \
  -srcstorepass 1234 \
  -srcalias esnode-cert \
  -destkeystore NONE \
  -deststoretype PKCS11 \
  -deststorepass 1234 \
  -destalias esnode-cert \
  -addprovider SunPKCS11 \
  -providerarg config/softhsm-pkcs11.cfg

# 5. Trust anchor -> BCFKS file.
jdk/bin/keytool -importcert -noprompt \
  -alias root-ca \
  -file config/root-ca.pem \
  -keystore config/root-ca.bcfks \
  -storetype BCFKS \
  -storepass changeit \
  -providerClass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider \
  -providerPath lib/bc-fips-2.1.2.jar

# 6. In opensearch.yml: comment the demo *.pem*_filepath lines, keep admin_dn: CN=kirk,...,
#    and add for both transport and http:

# --- Node identity from the PKCS#11 token ---
plugins.security.ssl.transport.keystore_type: PKCS11
plugins.security.ssl.transport.keystore_alias: esnode-cert
plugins.security.ssl.transport.keystore_password: "1234"      # SoftHSM PIN
plugins.security.ssl.http.keystore_type: PKCS11
plugins.security.ssl.http.keystore_alias: esnode-cert
plugins.security.ssl.http.keystore_password: "1234"

# --- Trust anchor from a BCFKS file ---
plugins.security.ssl.transport.truststore_type: BCFKS
plugins.security.ssl.transport.truststore_filepath: root-ca.bcfks
plugins.security.ssl.transport.truststore_password: "changeit"
plugins.security.ssl.http.truststore_type: BCFKS
plugins.security.ssl.http.truststore_filepath: root-ca.bcfks
plugins.security.ssl.http.truststore_password: "changeit"

# 7. Run the cluster and apply security config with the PKCS#11 admin key.
#    Pass = 'Connected as "CN=kirk,..."' followed by 'Done with success'.
OPENSEARCH_FIPS_MODE=true sh plugins/opensearch-security/tools/securityadmin.sh \
  -cd config/opensearch-security/ \
  -icl \
  -nhnv \
  -cacert config/root-ca.pem \
  -kst PKCS11 \
  -kspass 1234 \
  -ksalias kirk

Check List

• New functionality includes testing
• New functionality has been documented
• New Roles/Permissions have a corresponding security dashboards plugin PR
• API changes companion pull request created
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit f7781cd.

⛔ Hard block: Issues at High severity or above will block this PR from merging.

'Diff too large, requires skip by maintainers after manual review'

---

Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.

---

⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.