This article describes how to configure Scalable Function (referred to below as "SF") on Debian.
SF is a technology implemented using the Sub Function of the Linux Kernel, and a similar technology is Intel Scalable IOV.
This article will describe how to set up inter-VM communication using SF instead of SR-IOV VF in VMs using vhost vDPA module.
Requirement:
Hardware: MLX 6
Software:
kernel: 5.7+
qemu 5.1+
libvirt 6.9.0+
Setup:
1. Kernel Config Dependence
MLX5_ESWITCH MLX5_SF VDPA QEMU Config
../configure --enable-vhost-vdpa
2. MLX Setup
Changing the operating mode
devlink dev eswitch set pci/0000:31:00.0 mode switchdev &&¥ devlink dev eswitch set pci/0000:31:00.1 mode switchdev
Checking the operation mode
devlink dev eswitch show pci/0000:31:00.0 devlink dev eswitch show pci/0000:31:00.1 devlink port show
lyan@s2r5node80:~$ devlink dev eswitch show pci/0000:31:00.0 kernel answers: Operation not permitted lyan@s2r5node80:~$ sudo devlink dev eswitch show pci/0000:31:00.0 pci/0000:31:00.0: mode switchdev inline-mode none encap-mode basic lyan@s2r5node80:~$ sudo devlink dev eswitch show pci/0000:31:00.1 pci/0000:31:00.1: mode switchdev inline-mode none encap-mode basic lyan@s2r5node80:~$ devlink port show pci/0000:31:00.0/32774: type eth netdev tapext9867445 flavour pcisf controller 0 pfnum 0 sfnum 6 splittable false function: hw_addr f2:c4:a2:17:67:95 state active opstate attached pci/0000:31:00.0/32775: type eth netdev tapint9867445 flavour pcisf controller 0 pfnum 0 sfnum 7 splittable false function: hw_addr 36:7c:8a:1b:57:d7 state active opstate attached pci/0000:31:00.0/32776: type eth netdev tapext9870923 flavour pcisf controller 0 pfnum 0 sfnum 8 splittable false function: hw_addr c6:ed:f1:cc:3f:54 state active opstate attached pci/0000:31:00.0/32777: type eth netdev tapint9870923 flavour pcisf controller 0 pfnum 0 sfnum 9 splittable false function: hw_addr ba:88:5a:f8:1e:43 state active opstate attached pci/0000:31:00.1/131071: type eth netdev eth1 flavour physical port 1 splittable false
Configure the Scalable Function
devlink port add pci/0000:31:00.0 flavour pcisf pfnum 0 sfnum 11 devlink port function set pci/0000:31:00.0/32768 hw_addr CA:FE:C0:FF:EE:11 devlink port function set pci/0000:31:00.0/32768 state active devlink port add pci/0000:31:00.1 flavour pcisf pfnum 1 sfnum 12 devlink port function set pci/0000:31:00.1/98304 hw_addr CA:FE:C0:FF:EE:12 devlink port function set pci/0000:31:00.1/98304 state active
lshw -businfo -c network lyan@s2r5node80:~$ sudo lshw -businfo -c network Bus info Device Class Description ============================================================= pci@0000:31:00.0 eth0 network MT2892 Famil pci@0000:31:00.1 eth1 network MT2892 Famil pci@0000:31:00.0 tapext9862715 network Ethernet int pci@0000:31:00.0 tapint9862715 network Ethernet int pci@0000:31:00.0 tapext9866222 network Ethernet int pci@0000:31:00.0 tapint9866222 network Ethernet int devlink port show en7f0pf0sf11 -jp devlink port show en7f1pf1sf12 -jp
3. vdpa setup
modprobe vdpa modprobe vhost_vdpa echo mlx5_core.rdma.2 > /sys/bus/auxiliary/devices/mlx5_core.sf.2/mlx5_core.rdma.2/driver/unbind echo mlx5_core.eth.2 > /sys/bus/auxiliary/devices/mlx5_core.sf.2/mlx5_core.eth.2/driver/unbind echo mlx5_core.rdma.3 > /sys/bus/auxiliary/devices/mlx5_core.sf.3/mlx5_core.rdma.3/driver/unbind echo mlx5_core.eth.3 > /sys/bus/auxiliary/devices/mlx5_core.sf.3/mlx5_core.eth.3/driver/unbind vdpa dev add name vdpa0 mgmtdev auxiliary/mlx5_core.sf.2 vdpa dev add name vdpa1 mgmtdev auxiliary/mlx5_core.sf.3 4. ovs setup
systemctl start openvswitch ovs-vsctl set Open_vSwitch . other_config:hw-offload=true other_config:tc-policy=none systemctl restart openvswitch
ovs-vsctl add-br br30-ovs ovs-vsctl add-port br30-ovs ens2f0 ovs-vsctl add-port br30-ovs en7f0pf0sf11
5. VM setup
<interface type='vdpa'>
<source dev='/dev/vhost-vdpa-0'/>
</interface>
Ref: 1. https://docs.nvidia.com/networking/display/BlueFieldDPUOSv390/Scalable%20Functions 2. https://docs.nvidia.com/networking/display/MLNXOFEDv53100143/Introduction