This document is a work-in-progress that will change over time. I am still pretty sketchy w.r.t. the ofed/rdma stuff. To set up NFS-over-RDMA, you first need to have RDMA enabled hardware (there is no rxe or swi driver in FreeBSD). I have only tested using Mellanox, so I am even sketchier for the other NICs, although an Intel 8nn might work? At this time, you will need an up-to-date main kernel. (__FreeBSD_version 1600029 or higher. Post-Oct. 31 for the client.) (You'll need the sources to build the modules and a kernel built from the sources running on the system.) For Mellanox, the ConnectX/4->6 drivers should be in the GENERIC kernel. You will need to load mlx5ib.ko. Put a line like: mlx5ib_load="YES" in /boot/loader.conf to do this. (I think the module to load for Intel is irdma.ko?) With the above kernel module loaded.. # ibv_devinfo -v should give you a bunch of stuff about the NICs configuration. If this doesn't work (such as crash or ??), you can try updating the firmware on the NIC. For Mellanox, I did this using a utility called mlxflint, which I downloaded from github. Once built and working, I did.. # mlxflint -d pci0:2:0:0 -i /root/fw-Connect...bin burn (fw-Connect...bin is the file I found on Mellanox's firmware download site for the correct NIC model.) - Reboot and the firmware update takes effect. If you are not yet using RDMA, it is recommended that you test the RDMA functionality with one of the simple RDMA demos out there. (For example: /usr/src/contrib/ofed/librdmacm/examples/rdma_server.c and rdma_client.c. I'm not sure why, but some find this example only works for IPv6 with a link-local address.) Note that, when built, they need to be linked to 3 libraries: -lrdmacm -libverbs (plus the one for your NIC) For Mellanox, this is -lmlx5, for Intel I think it is -lirdma? ** You might only be able to get this to work if you use an IPv6 link local address to get to the server. Once you have these two little programs talking to each other between the client and server, your rdma should be working. For the NFS setup, there is a module called nfsrdma.ko for the server and a module called nfsclrdma.ko for the client. Both are currently available as experimental ports, which you will need to download from the sites and then build and install the module. For the NFS server.. You can download Vinicius Ferrao's server module from here: (The code was developed with generative AI and FreeBSD does not currently have a policy w.r.t. generative AI submissions.) This must either be done on the system that will be used as the NFS server or /boot/modules/nfsrdma.ko will need to be copied to the system and followed by.. # kldxref /boot/modules on the NFS server system. # git clone --depth=1 https://github.com/VersatusHPC/freebsd-nfsrdma-kmod # cd freebsd-nfsrdma-kmod/port # make SRC_BASE= stage check-plist package # make SRC_BASE= install Assuming you already have the NFS server set up. Add a line like: nfs_server_rdma_enable="YES" to your /etc/rc.conf. Reboot your NFS server and it should be working. If you have verbose booting set, you should see a line like: nfsrdma: listening on port 20049 (IPv4+IPv6) If you wish to use a different port#, just add a line like: nfs_server_rdma_listen="port#" to your /etc/rc.conf. For the client, you will need the nfsclrdma.ko module. The client code was written by me without the use of generative AI. It will find it's way into FreeBSD's main, once further testing has been completed. Download it via.. # fetch https://people.freebsd.org/~rmacklem/nfsclrdma-kmod.tar.gz Unroll the tarball and then.. # cd nfsclrdma-kmod # make SRC_BASE= # cd work/stage/boot/modules # cp nfsclrdma.ko /boot/modules # kldxref /boot/modules ("make install" will try and put it in the wrong place. I'm not a ports guy, so I don't know how to fix that?) Add these two lines to your /etc/rc.conf: nfs_client_enable="YES" nfs_client_rdma_enable="YES" and reboot. You should now be able to mount the NFS server, using the additional two options: rdma=20049,nowritereduce For example: # mount -t nfs -o nfsv4,rdma=20049,nowritereduce nfs-server:/ /mnt For now, you may want the "nowritereduce" option, since few NFS over RDMA servers implement write reduction at this time. (If you have a post-Nov. 4 kernel and nfsrdma.ko server module, write reduction should work ok between the FreeBSD client and FreeBSD server and you will not need the "nowritereduce" mount option.) Also, it does not support "sec=krb5[ip]" or the "tls" options. With the default vfs.maxbcachebuf setting of 65536, the I/O is limited to 64K. The current code with use up to 56 concurrent RPCs, with a limit of max_qp_rd_atom (16 for Mellanox) concurrent write RPCs (which do RDMA Reads). For ConnectX4 NICs, the limit is set at 12 RPCs and a single segment/MR per RPC. This seems to avoid the "mlx5_1: WARN: dump_cqe.."" for my testing. (See below w.r.t. these.) If you are using a newer NIC and encounter these messages in your /var/log/messages, then you can try setting: kern.rpc.rdma.maxrpc to a smaller # and see if you can find a setting that makes it go away. (A setting for kern.rpc.rdma.maxrpc is applied during mount, so a umount/mount is needed after changing the value.) --> Please email me (rick.macklem@gmain.com) if you experience this. If you with to try larger I/O sizes, you can put vfs.maxbcachebuf="131072" in your /boot/loader.conf. and reboot. (It can be any power of 2 up to 1048576.) The current code does not, as yet, use kib@'s helper function that allows it to RDMA data to/from the pages associated with the file's vnode instead of copying the data to/from those pages to/from anonymous pages in the server. It does RDMA data in/out of the buffer cache pages in the client. It will not work (or at least not well) on ConnectX3 NICs. Don't be surprised if you see the following on the client's console (or in /var/log/messages). It doesn't happen very frequently and the client should recover and continue on after a few seconds. I have no idea when this will get resolved, but it appears that the WC gets corrupted after ib_post_send() sets it up. I cannot see any way that my code would strash it. --> If you run the code a bunch and never see this on the client's console log, please let me know, along with your NIC type and whether or not the firmware is up to date. (For me, the test systems have ConnectX4 NICs and limiting to 12 RPCs and 1 segment has alleviated it.) Sep 12 03:28:24 mercat1 kernel: mlx5_1: WARN: dump_cqe:273:(pid 100228): dump error cqe Sep 12 03:28:24 mercat1 kernel: 00000000 00000000 00000000 00000000 Sep 12 03:28:24 mercat1 syslogd: last message repeated 2 times Sep 12 03:28:24 mercat1 kernel: 00000000 08007806 25000907 1b0405d3 Sep 12 03:28:24 mercat1 kernel: rpcrdma_send_done: failed opcode=0 status=6 Sep 12 03:28:24 mercat1 kernel: rpcrdma_send_done: pg=0x400003163 sgeaddr=0x0 len=0 lkey=0x0 num_sge=0 send_flags=0x0 If the client misbehaves in other ways, you can: # sysctl kern.rpc.rdma.debuglevel=4 and then it will be very chatty, but hopefully the messages in /var/log/messages will help isolate the problem. Thanks go to Vinícius Ferrão for his work on the on the server (Sponsored by: VersatusHPC), to the Mellanox/NVidia folk such as Wafa Hamzah for the testing they are doing and to the FreeBSD Netperf folk. Without the Netperf lab, I would have never gotten this tested. I anticipate that the client module will be pushed into FreeBSD's main in a couple of months, once it has seen more testing and appears stable. The server code done by Vinícius Ferrão could take longer, since generative AI was used and FreeBSD is working on a policy w.r.t. handling of these submissions. Good luck with it and let me know how it goes, rick rick.macklem@gmail.com