Race conditions with client mount on management node reboot

[frsc-ku]

This is a follow-up on #79 (see also #80).

After fixing problem #79, we observe two race conditions with client mounts, one on the management node and another one on a different node. I'm not sure if this is expected or not. The startup log for the client on the management node is (it seems to be able to talk to the mgmtd at this point in time):

Oct  7 13:46:10 rubusnas01fl beegfs-client[5499]: Starting BeeGFS Client:
Oct  7 13:46:10 rubusnas01fl beegfs-client[5499]: - Loading BeeGFS modules
Oct  7 13:46:10 rubusnas01fl kernel: beegfs: modprobe(5504): File system registered. Type: beegfs. Version: 8.1.0
Oct  7 13:46:10 rubusnas01fl beegfs-client[5499]: - Mounting directories from /etc/beegfs/beegfs-mounts.conf
Oct  7 13:46:10 rubusnas01fl beegfs-client[5532]: + mount --internal -t beegfs --source beegfs_nodev --target /mnt/beegfs '-orw,nosuid,cfgFile=/etc/beegfs/beegfs-client.conf,fscontext="system_u:object_r:tmp_t:s0",_netdev,sysMgmtdHost=169.245.12.144'
Oct  7 13:46:10 rubusnas01fl kernel: beegfs: mount(5552): Built without NVFS RDMA support.
Oct  7 13:46:10 rubusnas01fl kernel: beegfs: mount(5552): DatagramListener (init sock): Listening for UDP datagrams: Port 8004
Oct  7 13:46:10 rubusnas01fl kernel: beegfs: mount(5552): App_logInfos: BeeGFS Client Version: 8.1.0
Oct  7 13:46:10 rubusnas01fl kernel: beegfs: mount(5552): App_logInfos: ClientID: c15B0-68E4FD82-rubusnas01fl.unic
Oct  7 13:46:10 rubusnas01fl kernel: beegfs: mount(5552): App_logInfos: Usable NICs: ibs3(RDMA) vlan939(RDMA) vlan855(RDMA) ibs3(TCP) vlan939(TCP) vlan855(TCP) 
Oct  7 13:46:10 rubusnas01fl kernel: beegfs: beegfs_XNodeSyn(5554): Init: Waiting for beegfs-mgmtd@169.245.12.144:8008...
Oct  7 13:46:10 rubusnas01fl kernel: beegfs: beegfs_DGramLis(5553): Heartbeat incoming: New node: beegfs-mgmtd nas01_mgmt [ID: 1]; 
Oct  7 13:46:10 rubusnas01fl kernel: beegfs: beegfs_XNodeSyn(5554): Init: Management node found. Downloading node groups...
Oct  7 13:46:10 rubusnas01fl kernel: beegfs: beegfs_XNodeSyn(5554): NodeConn (acquire stream): Connected: beegfs-mgmtd@169.245.12.144:8008 (protocol: TCP)
Oct  7 13:46:10 rubusnas01fl kernel: beegfs: beegfs_XNodeSyn(5554): Sync: Nodes added (sync results): 4 (Type: beegfs-meta)
Oct  7 13:46:10 rubusnas01fl kernel: beegfs: beegfs_XNodeSyn(5554): Sync: Nodes added (sync results): 4 (Type: beegfs-storage)
Oct  7 13:46:10 rubusnas01fl kernel: beegfs: beegfs_XNodeSyn(5554): Init: Node registration...
Oct  7 13:46:10 rubusnas01fl kernel: beegfs: beegfs_XNodeSyn(5554): Registration: Node registration successful.
Oct  7 13:46:10 rubusnas01fl kernel: beegfs: beegfs_XNodeSyn(5554): Init: Init complete.
Oct  7 13:46:10 rubusnas01fl kernel: beegfs: mount(5552): Mount sanity check: Retrieval of root directory entry failed. Are all metadata servers running and registered at the management daemon? (Error: Communication error)
Oct  7 13:46:10 rubusnas01fl kernel: beegfs: mount(5552): Mount sanity check failed. Canceling mount. (Log file may provide additional information. Check can be disabled with sysMountSanityCheckMS=0 in the config file.)
Oct  7 13:46:10 rubusnas01fl kernel: beegfs: mount(5552): App (stop components): Stopping components...
Oct  7 13:46:10 rubusnas01fl kernel: beegfs: beegfs_XNodeSyn(5554): Deregistration: Node deregistration successful.
Oct  7 13:46:12 rubusnas01fl kernel: beegfs: mount(5552): App (wait for component termination): Still waiting for this component to stop: beegfs_AckMgr
Oct  7 13:46:12 rubusnas01fl kernel: beegfs: mount(5552): App (wait for component termination): Component stopped: beegfs_AckMgr
Oct  7 13:46:12 rubusnas01fl kernel: beegfs: mount(5552): App (stop): All components stopped.
Oct  7 13:46:12 rubusnas01fl beegfs-client[5552]: mount: /mnt/beegfs: mount(2) system call failed: Operation canceled.
Oct  7 13:46:12 rubusnas01fl systemd[1]: beegfs-client.service: Main process exited, code=exited, status=32/n/a
Oct  7 13:46:12 rubusnas01fl systemd[1]: beegfs-client.service: Failed with result 'exit-code'.

Is that something as simple as adding the beegfs-meta@.service units to the After list in the client unit file?

A second race occurs on remote client nodes. During reboot of the management node clients on other nodes hang as expected. However, when the mgmtd is up, the client operation (an ls) fails ungracefully. An ls right after succeeds. Is this expected or do I need to increase some timeout? Here the session:

$ ls /mnt/beegfs/
permanent  scratch
$ ls /mnt/beegfs/                      <-- This hangs as expected but fails unexpectedly.
ls: cannot access '/mnt/beegfs/': Communication error on send
$ ls /mnt/beegfs/
permanent  scratch

Are we looking at an unlucky timeout that triggered just before the mgmtd came on-line again? Here the log minus trivial message repeats:

Oct  7 13:42:01 rubuscmpn05fl kernel: beegfs: beegfs_XNodeSyn(7028): NodeConn (acquire stream): Connect failed on all available routes: nas01_mgmt [beegfs-mgmtd:1]
Oct  7 13:42:20 rubuscmpn05fl kernel: beegfs: beegfs_XNodeSyn(7028): Update states and mirror groups: Metadata node state sync failed. All nodes set to probably-offline.
Oct  7 13:42:26 rubuscmpn05fl kernel: beegfs: beegfs_XNodeSyn(7028): Update states and mirror groups: Storage target state sync failed. All targets set to probably-offline.
Oct  7 13:46:12 rubuscmpn05fl kernel: beegfs: beegfs_XNodeSyn(7028): NodeConn (acquire stream): Connected: beegfs-mgmtd@169.245.12.144:8008 (protocol: TCP)
Oct  7 13:47:15 rubuscmpn05fl kernel: beegfs: ls(30902): NodeConn (acquire stream): Connected: beegfs-meta@169.245.12.144:8110 (protocol: RDMA)
Oct  7 13:47:15 rubuscmpn05fl kernel: beegfs: ls(30902): Messaging (RPC): Receive failed from nas01_g5_a2 [beegfs-meta:102] @ 169.245.12.144:8120 (recv result: -70)
Oct  7 13:47:15 rubuscmpn05fl kernel: beegfs: ls(30902): Messaging (RPC node): Retrying communication with node: nas01_g5_a2 [beegfs-meta:102]

Thanks for any pointers!

[frsc-ku]

I should mention that we have a multi-mode set-up and it might be possible that the issue is related to that. The logs suggest that one meta data service is up and responding while another is not. The client doesn't wait for the second to become active and errs out prematurely.

Here some extra information about the set-up:

$ systemctl list-units beegfs-*
  UNIT                              LOAD   ACTIVE SUB     DESCRIPTION                       
  beegfs-client.service             loaded active exited  Start BeeGFS Client
  beegfs-meta@meta_g5_a1.service    loaded active running BeeGFS Metadata Server (multimode)
  beegfs-meta@meta_g5_a2.service    loaded active running BeeGFS Metadata Server (multimode)
  beegfs-meta@meta_g5_a3.service    loaded active running BeeGFS Metadata Server (multimode)
  beegfs-meta@meta_g5_a4.service    loaded active running BeeGFS Metadata Server (multimode)
  beegfs-mgmtd.service              loaded active running BeeGFS Management Server
  beegfs-storage@storage_g1.service loaded active running BeeGFS Storage Server (multimode)
  beegfs-storage@storage_g2.service loaded active running BeeGFS Storage Server (multimode)
  beegfs-storage@storage_g3.service loaded active running BeeGFS Storage Server (multimode)
  beegfs-storage@storage_g4.service loaded active running BeeGFS Storage Server (multimode)

The services beegfs-meta and beegfs-storage are both disabled.

$ sudo beegfs node list
ID     TYPE        ALIAS                             
c:17   client      c1B70-68E3A886-rubuscmpn05fl.uni  
c:22   client      c1C37-68E5096B-rubusnas01fl.unic  
m:101  meta        nas01_g5_a1                       
m:102  meta        nas01_g5_a2                       
m:103  meta        nas01_g5_a3                       
m:104  meta        nas01_g5_a4                       
s:101  storage     nas01_g1                          
s:102  storage     nas01_g2                          
s:103  storage     nas01_g3                          
s:104  storage     nas01_g4                          
mg:1   management  nas01_mgmt

[iamjoemccormick]

Hi @frsc-ku ,

When the client starts it will wait up to sysMountSanityCheckMS to download node lists from the management, but once it does that it expects the metadata and storage nodes are already online. After it has the node lists it will attempt to fetch the root directory from whichever metadata server owns it, and also fetch free space for all storage targets. If either of those sanity checks fail, the mount syscall returns an error.

Adding beegfs-meta@.service to the After list in the client unit file only helps the client(s) that are on the same machine as your root metadata server. If possible I would recommend to wait and start the clients until after all servers are started. If that isn't possible you could disable the sanity checks by setting sysMountSanityCheckMS=0 in the client config file. The main drawback of disabling the mount sanity check is an unusable mount point until at least the root meta node is available. If you did try to interact with the mount (e.g., ls /mnt/beegfs) most likely the process will end up in an uninterruptible sleep (i.e., D state) so even kill -9 <pid> is likely ineffective.

[frsc-ku]

Dear Joe,

sorry for the slow reply, was on holidays. I think I need to report issues in a better prioritized order. The first one for the co-located beegfs client (please also read about the second issue further down) I could solve with a sudo systemctl edit beegfs-client.service and adding the somewhat silly workaround

[Service]
ExecStartPre=/bin/sleep 30

The second issue, however, is not that easy. Here we talk about a remote client crashing on storage server reboot or service restart. To be more precise, lets call the host running the BeeGFS file system srv and the (different) host running a client clnt. If maintenance including a restart of BeeGFS storage- and meta-services happens on srv with an active client on clnt, the following scenario is expected:

• The client on clnt hangs on any access to a service on srv until connection to this service is restored. Processes accessing BeeGFS on the client node clnt are in D-state for the duration of service outage and resume operation after connectivity is reestablished.

A scenario somewhat consistent with error handling but not preferred at all is that all attempts to access BeeGFS on clnt fail immediately during a service outage. This, however, would be considered a very poor implementation. The D-state during + resume after service is standard by now.

With multi-mode meta-services this becomes a bit more involved and here is what I observe as an actual bug. We have a client on clnt and do service on srv. My observation is in conflict with both of the above scenarios, especially with the highlighted preferred one. Expectation is "D-state hang + resume". Observed is:

• On service shutdown on srv the client on clnt hangs in D-state as expected.
• The services on srv resume in random order (whatever systemd decides to shoot up first). As a consequence, a meta-service that does not hold the root-inode comes up first.
• The client connects to this meta service and finds out that the meta-service for the root inode is down. The client service decides to disconnect without waiting for the other meta-services to come up.

This behavior is the worst of all possibilities and is likely not observed unless multi-mode is used. My absolute preference would be:

• Clients always hang on down services and processes are in D-state until the required service comes up again.
• To allow graceful maintenance on client nodes with D-state processes (in case BeeGFS cannot be recovered fast), please consider something like an out-of-band operation triggered with echo 1 > /proc/fs/beegfs/*/force_disconnect that causes any operation (hanging or new) to terminate with an IO error to allow a clean umount of the file system.

Mainly I would appreciate a consistent handling of restart of multiple meta-services on a single srv node. The possibility to get rid of D-state processes without reboot is less important. We got used to this with NFS already.

In case sysMountSanityCheckMS=0 also works for clients that are already mounted, please let me know. I would appreciate disabling this sanity check altogether, I don't really see its value if it makes service restarts impossible without crashing all clients.

Thanks and best regards,
Frank

[iamjoemccormick]

Hi @frsc-ku ,

My turn to apologize for the delayed response.

Were the metadata services immediately restarted or where they down for some period of time? Do you know if the management had marked them as offline before the services were restarted (these states are actually recorded as part of the target information beegfs target list --state)?

The management will mark metadata and storage services as offline based on the node-offline-timeout parameter in beegfs-mgmtd.toml. I would expect client processes to hang in D-state if a service they are trying to access is down, and if services don't come back before the management marks them as "offline", then an error (typically EREMOTEIO) will be returned to the process. If the management service is not available then clients will wait up to sysTargetOfflineTimeoutSecs in beegfs-client.conf.

In my experience nodes being marked as offline usually explains behavior like this, but please let me know if that is not what is happening here. If that is/might be the case you could try raising the timeout. The main reason you wouldn't want to increase the timeout is if you are using buddy mirroring where you actually want a switchover to happen once that timeout is exceeded. Otherwise the main drawback is processes getting stuck in D-state for a long time and preventing BeeGFS from being unmounted.

If that is not what is happening here, what operation(s) are you are attempting when you get an error (mount, read/write, open, mv, etc.) and what error do you see? In my experience it doesn't matter if the root metadata service is up before any other services, but there might be a corner case for some specific operation(s) that involves meta services to communicate directly.

I do like your /proc/fs/beegfs/*/force_disconnect suggestion. One argument against this is that we should just rely on the timeouts to elapse naturally and processes to be unblocked. But I think it is worth adding to the wishlist and if there is enough interest we can look at adding it.

Lastly sysMountSanityCheckMS=0 only applies when a mount is happening, this won't help after a client is mounted. Increasing timeouts would be the solution there.

[frsc-ku]

Dear Joe,

I have a bit the feeling we are talking about different scenarios here. Therefore, a quick recap of our set-up and my observations:

We have a single-node BeeGFS server running the management-, 4 storage- and 4 meta services. In other words, everything is on a single machine and we observe inconsistent behavior of remote clients on reboot of this single server. We are talking here about a "restart" of all BeeGFS services simultaneously with a reboot command (clean reboot).

So all services were down for the reboot time of about 4-5 minutes and came back up in systemd-startup order. Probably the management service first and everything else in random order.

The experiment is as follows:

1. Execute reboot on the storage server (all services disappear pretty much simultaneously).
2. On every client execute an ls -lR on the BeeGFS root, which will now hang.
3. After services came back, some clients completed the ls-command correctly while others failed.

Expected is that all clients hang until all accessed services are back and then complete the hanging command, but at least that all clients do the same. The inconsistent behavior indicates a race condition.

I only brought up a somewhat more general multi-storage-node set-up, because this inconsistency seems to be caused by an attempt of the clients to be smart about handling unavailable (meta-)services. My preferred method is not to handle that at all and always let clients hang until any queried service is back up. It seems the easiest way to prevent this inconsistency with an option to set a timeout to "wait forever" and have the force_disconnect option to allow a clean umount of all BeeGFS file systems if absolutely necessary (clean up D-state processes on a compute node without reboot).

I have played with sysTargetOfflineTimeoutSecs=0 already to no avail. What I am asking for is that instead of:

ls: cannot access '/mnt/beegfs/': Communication error on send

the ls command keeps hanging until the missing (meta-)service is responding again instead of erring out.

As far as I can see, after a reboot of our single BeeGFS node there is no timeout and the client errs out on the first failed communication attempt. It should - after the management service is back up - at least give a little bit of time for the remaining services to come up again. It looks like there is a race between client requests and the management service querying the status of all services. That the client command error always happens at exactly around the moment the BeeGFS services come back up indicates that a simultaneous restart of all BeeGFS services (or management + some others) is not handled consistently.

I have not seen in the documentation that the management service should not be co-located with storage- or meta-services, so it should be possible to have a clean consistent resume behavior in case everything is rebooted at the same time.

Best regards,
Frank

[iamjoemccormick]

Hi Frank,

That helps narrow things down so I can better explain what is likely happening.

First, running multiple services on the same machine (including all of them) is supported. Given a 4–5 minute server reboot, the mgmtd’s node-offline-timeout (default 180s) is likely the culprit. When management comes back, it will initially consider services offline until they check in again.

As metadata and storage nodes start to reconnect and report their state, the management notifies clients to refresh node/target states. Clients also poll on their own schedule, so refresh timing varies slightly between clients. Depending on when the root metadata node reports back relative to those refreshes, some clients may still see the root meta as Offline while others already see it Online. This explains why ls fails on some clients and succeeds on others.

As a workaround (and to verify this theory) you could raise the management’s node-offline-timeout to >2× the boot time. You would also want to increase sysTargetOfflineTimeoutSecs in your metadata and storage configuration files to match, but in this case it is the management configuration that matters most.

That said, I believe we should add a grace period when the management starts up, to allow time for nodes to reconnect. We have a similar mechanism already to avoid unnecessary switchovers when buddy mirroring is used. I opened ThinkParQ/beegfs-rust#46 to track this, and will discuss options with the team.

If you do get a chance to test with an increased timeout if that does not fully solve the issue please let me know.

Thank you,

~Joe

[frsc-ku]

Dear Joe, thanks a lot for your answer. I will execute the experiment you suggest and found an inconsistency in the documentation of the parameters you ask me to change. Could you please clarify what is correct? Thanks!

In beegfs-meta.conf, beegfs-storage.conf and beegfs-client.conf:

In the help text below, the name of the mgmtd config file is incorrect, should be *.toml. Also, this setting does not exist in beegfs-mgmtd.toml, maybe it should match the value of node-offline-timeout in beegfs-mgmtd.toml? Note the difference in the description
and default value in the client conf (which has an explanation and seems intentional).

beegfs-meta.conf, beegfs-storage.conf

# [sysTargetOfflineTimeoutSecs]
# Timeout until the metadata nodes and storage targets are considered offline
# when no target state updates can be fetched from that node.
# Note: This must be the same value as in the /etc/beegfs/beegfs-mgmtd.conf on
#    the management node.
# Values: time in seconds
# Default: 180

beegfs-client.conf

# [sysTargetOfflineTimeoutSecs]
# Timeout until all storage targets and metadata nodes are considered offline
# when no target state updates can be fetched from the management server.
# If this value is 0, targets will never be set to offline due to an
# unreachable management node and will stay in state probably-offline.
# Note: This must be at least twice as large as the value of
#    sysTargetOfflineTimoutSecs in the server config files.
# Values: time in seconds
# Default: 900

In beegfs-mgmtd.toml:

The setting node-offline-timeout exists only for the mgmtd service and there is only one such service (if I understood correctly). Therefore, the statement below seems not to make sense. Should some other setting be identical across all nodes and match the value of node-offline-timeout or is the mgmtd config file used by more services than just the mgmtd? Or is this an anachronism that was forgotten to clean up after a change?

Since sysTargetOfflineTimeoutSecs has different default values between clients and other services, it cannot be this value that should be the same across all nodes (I mention here this one, because you talk about this value together with node-offline-timeout).

# Defines after which time without contact a node/target is considered offline.
# IMPORTANT: This setting must be the same on all nodes in the system, especially when using
# mirroring.
# node-offline-timeout = "180s"

[iamjoemccormick]

Hi Frank,

With BeeGFS 8 the management configuration file format was switched to TOML and the configuration parameters overhauled to have more modernized parameter names so sysTargetOfflineTimoutSecs -> node-offline-timeout. As I was looking into this issue I also realized updating the meta/storage/client configuration files to reflect this was overlooked. I'll update those references so going forward they will look like:

Meta/Storage:

# [sysTargetOfflineTimeoutSecs]
# Timeout until the metadata nodes and storage targets are considered offline
# when no target state updates can be fetched from that node.
# Note: This must be the same value as the management's node-offline-timeout.
# Values: time in seconds
# Default: 180

Client:

# [sysTargetOfflineTimeoutSecs]
# Timeout until all storage targets and metadata nodes are considered offline
# when no target state updates can be fetched from the management server.
# If this value is 0, targets will never be set to offline due to an
# unreachable management node and will stay in state probably-offline.
# Note: This must be at least twice as large as the management's node-offline-timeout.
# Values: time in seconds
# Default: 900

Thanks,

~Joe

[frsc-ku]

Dear Joe,

please also refer in beegfs-mgmtd.toml to the values of sysTargetOfflineTimeoutSecs, for example, with something like:

# Defines after which time without contact a node/target is considered offline.
# IMPORTANT: This setting must be adjusted together with sysTargetOfflineTimeoutSecs
# in all meta-/storage- and client configuration files, especially when using mirroring.
# Please see the respective sections of documentation for what rules apply.
# node-offline-timeout = "180s"

I changed the configuration values to appropriate larger timeouts and repeated the experiment 3 times. This time the ls command executed successfully on all 3 clients every time. They were hanging different periods of time after the mgmtd came back, but eventually all resumed operation without hiccups. I guess the value of sysUpdateTargetStatesSecs (30s) determines the longest time a client will hang after the management service comes back up.

This is a good workaround. Now it is much easier to do standard maintenance that requires a reboot without having to schedule a service window.

I think your hypothesis is correct, there is a race between clients and the mgmtd querying service statuses and there should be some grace window relative to the startup time of the mgmtd service. I guess one should still set reasonable offline-timeout values to have normal node reboot handled without issues.

I have one last question. When changing certain config values, especially timeouts, is a simple systemctl daemon-reload enough? Specifically for client services it would be great if one could update such values without the full umount+mount implied by a systemctl restart beegfs-client (would allow update of certain config parameters on busy clients).

Thanks for your help!
Frank

[iamjoemccormick]

Hi Frank,

Thanks for confirming! I'll update the help text in beegfs-mgmtd.toml.

The systemctl daemon-reload is mainly for re-reading unit files and rebuilding the internal systemd dependency graph. AFAIK it doesn't start/stop/restart/reload units.

You may be thinking of systemctl reload <unit> which can execute different functionality if the unit defines ExecReload. The BeeGFS client service doesn't define this directive and there is currently no way to update all configuration without a full unmount+mount. This is mainly because some configuration is impossible/challenging to update without a remount (e.g., buffer or cache configuration).

Some configuration is possible to update without a remount, notably log_levels can be set by executing:

sudo bash -c "echo -e 'general=5\nconn=5\ncommkit=5' > /proc/fs/beegfs/<client_id>/log_levels"

But for the most part other configuration cannot be safely updated without a remount anyway.

~Joe