Add ChainBuilder to support chaining work requests

[aryeyur]

No description provided.

[aryeyur]

Hey 👋

Would it be possible to look into this PR?

It adds support for chaining RDMA operations.

[xmakro]

Thank you for the PR, please see the comment for discussion

[xmakro]

The ChainBuilder only needs to know the QueuePair during post, so we could disentangle them until that point. This avoids the QP lifetime and allows for example creating two ChainBuilders at the same time, and posting them later with the same QueuePair. So ChainBuilder just becomes a helper around &mut [ibv_send_wr].

If we go this route, then that raises the question if we need a ChainBuilder at all, or if we can have the helper only around ibv_send_wr, with an API looking like:

#[repr(transparent)]
pub struct WorkRequest<'a> {
    wr: ffi::ibv_send_wr,
    _local: PhantomData<&'a [LocalMemorySlice]>,   // keeps the sg_list borrow alive
}

impl<'a> WorkRequest<'a> {
    pub fn send (local: &'a [LocalMemorySlice], wr_id: u64, imm: Option<u32>) -> Self;
    pub fn write(local: &'a [LocalMemorySlice], remote: RemoteMemorySlice, wr_id: u64) -> Self;
    pub fn read (local: &'a [LocalMemorySlice], remote: RemoteMemorySlice, wr_id: u64) -> Self;
}

on the QueuePair:

pub unsafe fn post(&mut self, mut wrs: impl AsMut<[WorkRequest<'_>]>) -> io::Result<()>;

usage:

qp.post([
    WorkRequest::write(&locals[0], remotes[0].clone(), 1, None),
    WorkRequest::write(&locals[1], remotes[1].clone(), 2, None),
    WorkRequest::send(&notify, 3, None).signaled(),
])?;

// dynamic / reused buffer,
qp.post(&mut wrs)?;

Then we can also do things like:

WorkRequest::write(..).signaled(), .fenced()

What do you think? This API should have zero overhead, so we can redirect the existing single post_send/post_recv API to it.