beegfs commands error: "transport: authentication handshake failed: EOF"

[edpuig97]

I have set up a 3 node BeeGFS cluster with InfiniBand network, the IP for the IPoIB interfaces are 192.168.30.1, 192.168.30.2, 192.168.30.3. The managment and metadata services are running only in the first node (192.168.30.1). When installed client packages in a different client node, beegfs commands didn't work, ie:

# beegfs health check
###################################################
Running Health Check for beegfs://192.168.30.1:8008
###################################################
###################################
>>>>> Checking for Busy Nodes <<<<<
###################################
Error: error getting node list from management: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: EOF"

Hoping it was some TLS error about using self-signed certificates, I disabled authentication and TLS in all nodes and services, the error changed to:

 beegfs health check --tls-disable --auth-disable
###################################################
Running Health Check for beegfs://192.168.30.1:8008
###################################################
###################################
>>>>> Checking for Busy Nodes <<<<<
###################################
Error: error getting node list from management: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF"

I have enabled monitoring and I can see in the grafana dashboards the correct status of the services. Also I mounted the filesystem and wrote a 10GB file to it without problems:

 dd if=/dev/urandom of=filename.img bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 39.2898 s, 273 MB/s

Some logs from beegfs-mgmtd service that seem interesting to me (because of the early oef part):

Nov 27 17:55:26 hive01 beegfs-mgmtd[14961]: Updated target consistency states for Storage nodes
Nov 27 17:55:26 hive01 beegfs-mgmtd[14961]: Updated target consistency states for Meta nodes
Nov 27 17:55:26 hive01 beegfs-mgmtd[14961]: Updated Meta target info
Nov 27 17:55:34 hive01 beegfs-mgmtd[14961]: Accepted incoming stream from [::ffff:192.168.30.2]:59450
Nov 27 17:55:34 hive01 beegfs-mgmtd[14961]: Closed stream from [::ffff:192.168.30.2]:59450: early eof
Nov 27 17:55:37 hive01 beegfs-mgmtd[14961]: Updated target consistency states for Storage nodes
Nov 27 17:55:40 hive01 beegfs-mgmtd[14961]: Accepted incoming stream from [::ffff:192.168.30.3]:48054
Nov 27 17:55:40 hive01 beegfs-mgmtd[14961]: Closed stream from [::ffff:192.168.30.3]:48054: early eof
Nov 27 17:55:41 hive01 beegfs-mgmtd[14961]: Accepted incoming stream from [::ffff:192.168.30.1]:56684
Nov 27 17:55:41 hive01 beegfs-mgmtd[14961]: Closed stream from [::ffff:192.168.30.1]:56684: early eof
Nov 27 17:55:44 hive01 beegfs-mgmtd[14961]: Updated target consistency states for Storage nodes
Nov 27 17:55:53 hive01 beegfs-mgmtd[14961]: Running switchover check

I'm using Ubuntu 24.04 as base OS for all nodes, and BeeGFS 8.2
Please, help me out with this.

[iamjoemccormick]

@edpuig97 ,

I would expect the error error reading server preface: unexpected EOF if the management service has TLS enabled but you disable TLS on CTL with --tls-disable. Did you set tls-disable = true (and auth-disable = true if desired) in your /etc/beegfs/beegfs-mgmtd.toml file and restart the service?

Alternatively you could copy the TLS certificate used by your management (by default at /etc/beegfs/cert.pem) to /etc/beegfs/cert.pem on the machine(s) where you are running CTL and it should "just work".

Reference: https://doc.beegfs.io/latest/advanced_topics/tls.html

[edpuig97]

@iamjoemccormick
Thanks for your reply!
At first we intended to use TLS and auth as suggested in the documentation. As you can see in the original question, that gave us the first error running client commands. After that we change to no auth nor TLS and restarted all services and the second error message came.
The thing that's so unexpected for me is that all the services are running ok, even monitoring services and I can see in the Grafana dashboards all the I/O operations and network usage when we write data to the client mounted path.
The issues here are only with client commands.

[iamjoemccormick]

Currently TLS is only (optionally) used to encrypt control plane traffic between the BeeGFS command line tool (CTL), Management service, and optional Remote/Sync services. For now the rest of the core file system (including the monitoring service) doesn't use TLS in the data plane, which is why the client I/O path will work regardless if TLS is configured "correctly".