OVS-DPDK startup race: PCI devices are bound after ovs-vswitchd starts

[fpolanco-one]

There is a startup-ordering issue between openvswitch.service and the generated opennebula-ovs.service.

The generated service contains:

After=openvswitch.service network-pre.target
Requires=openvswitch.service

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/opennebula-ovs.sh

However, openvswitch.service starts ovs-vswitchd.service, while the PCI driver binding required by the DPDK interfaces is performed inside opennebula-ovs.sh.

As a result, ovs-vswitchd starts before the physical interfaces have been prepared for DPDK.

Relevant one-deploy files:

roles/openvswitch/templates/opennebula-ovs.service.jinja
roles/openvswitch/templates/opennebula-ovs.sh.jinja

Observed behavior

The affected host uses two physical interfaces in an OVS-DPDK bond:

0000:17:00.0 → vmnic0
0000:ca:00.0 → vmnic6

During boot, ovs-vswitchd repeatedly attempted to attach the interfaces before they were bound to vfio-pci:

EAL: Driver cannot attach the device (0000:17:00.0)
EAL: Driver cannot attach the device (0000:ca:00.0)
EAL: Failed to attach device on primary process
netdev_dpdk|WARN|Error attaching device

The one-deploy networking script subsequently executed the PCI binding operations, after which both interfaces eventually attached successfully.

Expected behavior

The PCI devices required by OVS-DPDK should be prepared before ovs-vswitchd attempts to initialize them.

Actual behavior

OVS starts first and attempts to initialize PCI devices that have not yet been bound to the required driver.

The interfaces eventually recovered in our validation, but this introduces avoidable startup errors and a dependency on DPDK retry behavior.

Progress Status

• Code committed
• Testing - QA
• Documentation (Release notes - resolved issues, compatibility, known issues)

[sk4zuzu]

@fpolanco-one Hi,we want something like this e78b208 merged anyway, will it work for you? 🤔

[fpolanco-one]

@sk4zuzu I checked the commit. It improves the service ordering, but I don't think it fixes the specific DPDK race we have.

openvswitch.service still starts ovs-vswitchd before opennebula-ovs.sh binds the PCI devices to vfio-pci. The commit actually makes opennebula-ovs.service run later, after network-online.target. We need that binding to happen before openvswitch.service runs.

I think we need to move the PCI binding into an earlier initialization step, before ovs-vswitchd starts, while keeping the bridge and IP configuration in opennebula-ovs.sh.

One thing worth checking with the new ordering: our management IP and default gateway are configured on the OVS bridge, so waiting for network-online.target could potentially introduce a boot dependency issue.

[sk4zuzu]

But I don't think that's possible what you want with single opennebula-ovs.service this is direct change to original research / specs. 🤔

[fpolanco-one]

sorry, I'm not aware of the original request, I'm raising this since we saw this yesterday with a customer during a maintenance window, vfio-pci binding seems to be a prerequisite of openvswitch service for dpdk.

[dann1]

This issue looks like cosmetic only. The output of ovs-vsctl show does show the dpdk devices with error if they are not available. Once the binding is done, that error disappears because the device becomes available again.

Since we need the demon to be up and running to run ovs-vsctl commands in opennebula-ovs it is a strict dependency, we have to run it afterwards. The binding bit could be done earlier, even with the pci role logic, but then we don't cover the case of IP migration to use the management interfaces as OVS bridge uplinks. So to avoid this issue we could then have two systemd units, but that is overcomplicating.

Maybe we can driverctl to perma bind instead of using dpdk-devbind.py to do the binding.