Skip to content

SSL error when having https enabled with manual certificates in jupyterhub side.  #319

Open

Description

@MarineChap

Dear,
We have a daskhub server on a supercomputer in our lab. We are based on the latest helm chart ( daskhub-2022.6.0 version 2022.6.1)
https on jupyterhub is enabled through manual certificate and it seems dask cannot success to communicate together.

config :

jupyterhub:
  hub:
    services:
      dask-gateway:
        apiToken: secret1
  proxy:
    https:
      enabled: true
      hosts:
         - 10.86.1.95
      type: secret
      secret:
         name: example-tls
    traefik: 
      networkPolicy: 
        enabled: False


dask-gateway:
  gateway:
    auth:
      jupyterhub:
        apiToken: secret1

Now when I try to create a dask cluster in my jupyterhub session, I am having a ssl error.

from dask_gateway import Gateway
gateway = Gateway()
cluster = gateway.new_cluster() 

Error:

ClientConnectorCertificateError: Cannot connect to host proxy-public:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')]

Note that if I deactivate https in jupyterhub and work only with http everything is running correctly. So, I think there is a bug somewhere when using manual authentification.

Any idea for me ? Thanks

Activity

  1. consideRatio commented on Aug 26, 2022

    @consideRatio
    Collaborator

    I'm always a bit unsure about things related to HTTPS certificates, but here are some guesses.

    If you have a HTTPS certificate that you have created yourself (as compared to getting it from commonly trusted Let's Encrypt for example), it is often causing issues like this. Typcially you would need to acquire a HTTPS certificate by a trusted issuers of certificates (Let's Encrypt for example), or, declare that all computers accessing a server (jupyterhub) using HTTPS should trust the self-generated certificate.

    I'm not sure there isn't a bug to be fixed or similar, but it seems likeley to me that this failure is expected as the dask-gateway software isn't configured to accept what I understand is a HTTPS certificate - provided by a not commonly trusted source.


    This is related configuration btw:

    if "PROXY_HTTP_SERVICE_HOST" in os.environ:
    # https is enabled, we want to use the internal http service.
    gateway_address = "http://{}:{}/services/dask-gateway/".format(
    os.environ["PROXY_HTTP_SERVICE_HOST"],
    os.environ["PROXY_HTTP_SERVICE_PORT"],
    )
    print("Setting DASK_GATEWAY__ADDRESS {} from HTTP service".format(gateway_address))
    else:
    gateway_address = "http://proxy-public/services/dask-gateway"
    print("Setting DASK_GATEWAY__ADDRESS {}".format(gateway_address))


    To conclude, I'm overall confused about the situation etc, and can't say that there isn't a bug or is a bug etc. Logs from various pods and clarifications about what kind of certificate you have provided for use by JupyterHub etc could be relevant.

  2. MarineChap commented on Aug 26, 2022

    @MarineChap
    Author

    Thanks for your quick answer.
    I also not very good with this concept of certificates but from what I understood the certificates are issued by a commonly trusted orga (just without using let's encrypt because of a problem with the name of the computer - space were not accepted something like that) but at the end, it should be the same.

    Good you remember this part of the code for me ! I checked before and forget to add this to the issue but it is indeed super important. From the log in my hub, it seems that it does not detect that https is enabled.

    Loading /usr/local/etc/jupyterhub/secret/values.yaml
    No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
    Loading extra config: 00-add-dask-gateway-values
    Setting DASK_GATEWAY__ADDRESS http://proxy-public/services/dask-gateway
    Adding dask-gateway service URL
    

    Maybe I could force it by adding this two variables as extra-config in jupyterhub side but I don't know what value should be expected here. The one from the service proxy-api or another service ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions