vDPA Scalable Function setup for VMs

lyan 2023-03-13 15:22

This article describes how to configure Scalable Function (SF) on Debian. SF is implemented using the Sub Function capability in the Linux kernel. A similar technology is Intel Scalable IOV.

In this guide, we demonstrate how to enable inter-VM communication using SF instead of SR-IOV VFs, leveraging the vhost-vDPA module.

Requirements

Hardware

MLX6 (ConnectX-6 family)

Software

Kernel 5.7+
QEMU 5.1+
libvirt 6.9.0+

1. Kernel and QEMU Configuration

Kernel Config Dependencies

CONFIG_MLX5_ESWITCH
CONFIG_MLX5_SF
CONFIG_VDPA

QEMU Configuration

../configure --enable-vhost-vdpa

2. MLX Setup

Switch eSwitch to switchdev Mode

devlink dev eswitch set pci/0000:31:00.0 mode switchdev
devlink dev eswitch set pci/0000:31:00.1 mode switchdev

Verify Operation Mode

sudo devlink dev eswitch show pci/0000:31:00.0
sudo devlink dev eswitch show pci/0000:31:00.1

Example output:

pci/0000:31:00.0: mode switchdev inline-mode none encap-mode basic
pci/0000:31:00.1: mode switchdev inline-mode none encap-mode basic

Check devlink Ports

devlink port show

Configure Scalable Functions

devlink port add pci/0000:31:00.0 flavour pcisf pfnum 0 sfnum 11
devlink port function set pci/0000:31:00.0/32768 hw_addr CA:FE:C0:FF:EE:11
devlink port function set pci/0000:31:00.0/32768 state active

devlink port add pci/0000:31:00.1 flavour pcisf pfnum 1 sfnum 12
devlink port function set pci/0000:31:00.1/98304 hw_addr CA:FE:C0:FF:EE:12
devlink port function set pci/0000:31:00.1/98304 state active

Verify Network Devices

sudo lshw -businfo -c network

Show Specific SF Devices

devlink port show en7f0pf0sf11 -jp
devlink port show en7f1pf1sf12 -jp

3. vDPA Setup

Load Required Modules

modprobe vdpa
modprobe vhost_vdpa

Unbind SF Auxiliary Devices

echo mlx5_core.rdma.2 > /sys/bus/auxiliary/devices/mlx5_core.sf.2/mlx5_core.rdma.2/driver/unbind
echo mlx5_core.eth.2 > /sys/bus/auxiliary/devices/mlx5_core.sf.2/mlx5_core.eth.2/driver/unbind
echo mlx5_core.rdma.3 > /sys/bus/auxiliary/devices/mlx5_core.sf.3/mlx5_core.rdma.3/driver/unbind
echo mlx5_core.eth.3 > /sys/bus/auxiliary/devices/mlx5_core.sf.3/mlx5_core.eth.3/driver/unbind

Create vDPA Devices

vdpa dev add name vdpa0 mgmtdev auxiliary/mlx5_core.sf.2
vdpa dev add name vdpa1 mgmtdev auxiliary/mlx5_core.sf.3

4. Open vSwitch Setup

systemctl start openvswitch
ovs-vsctl set Open_vSwitch . other_config:hw-offload=true other_config:tc-policy=none
systemctl restart openvswitch

Create Bridge and Add Ports

ovs-vsctl add-br br30-ovs
ovs-vsctl add-port br30-ovs ens2f0
ovs-vsctl add-port br30-ovs en7f0pf0sf11

5. VM Configuration (libvirt)

Use a vDPA interface in your VM XML definition:

<interface type='vdpa'>
  <source dev='/dev/vhost-vdpa-0'/>
</interface>

References

NVIDIA BlueField DPU OS Documentation – Scalable Functions
MLNX_OFED Documentation – Introduction

Conclusion

By leveraging Scalable Functions with vhost-vDPA, we avoid traditional SR-IOV VF assignment while still achieving hardware-accelerated datapath and inter-VM communication. This approach provides better flexibility in modern switchdev-based environments.