UFM vs OpenSM: Understanding InfiniBand Fabric Management

Quick Answer

Both UFM and OpenSM are Subnet Managers for InfiniBand fabrics, but they serve very different purposes. UFM is NVIDIA's comprehensive enterprise fabric management platform, while OpenSM is a lightweight, open-source Subnet Manager implementation.

What Is a Subnet Manager (SM)?

Before comparing UFM and OpenSM, it helps to understand what a Subnet Manager actually does. The Subnet Manager is the "brain" of an InfiniBand fabric. It is responsible for:

1. Fabric Discovery — Discovers all switches, adapters, and connections in the fabric
2. LID Assignment — Assigns Local IDs (LIDs) to every port
3. Routing Table Configuration — Programs forwarding tables into switches
4. Port State Management — Drives ports through: Down → Initialize → Armed → Active
5. Partition Management — Creates VLANs (partitions) for network isolation

OpenSM (Open Subnet Manager)

What Is OpenSM?

OpenSM is an open-source, lightweight Subnet Manager that implements the InfiniBand specification for fabric management. It runs as a daemon on any Linux node, embedded switch, or dedicated management server.

Key Characteristics

Where OpenSM Runs

• On a compute node — Any server with an InfiniBand adapter
• Embedded in a switch — Many IB switches ship with OpenSM built-in
• On a dedicated management server — A separate server solely for SM duties

Feature Set

What OpenSM does:

• ✅ Fabric discovery and initialization
• ✅ LID assignment and routing
• ✅ Partition (VLAN) management
• ✅ Basic high availability (master/standby)
• ✅ Multiple routing algorithms (MinHop, LASH, Fat-Tree, etc.)

What OpenSM does NOT do:

• ❌ No graphical user interface
• ❌ No performance monitoring or analytics
• ❌ No centralized logging or alerting
• ❌ No topology visualization
• ❌ No advanced diagnostics
• ❌ No job/workload scheduler integration

Configuration Files

/etc/rdma/opensm.conf       # Main configuration file
/etc/sysconfig/opensm       # Startup parameters (GUID, priority)
/etc/rdma/partitions.conf   # Partition definitions

Best Suited For

• Small to medium InfiniBand clusters (< 100 nodes)
• Development and test environments
• Simple point-to-point or small switched fabrics
• Environments where minimal overhead and full control are priorities
• Budget-conscious deployments

UFM (Unified Fabric Manager)

What Is UFM?

UFM (Unified Fabric Manager) is NVIDIA's enterprise-grade fabric management platform. It is a comprehensive software suite that includes a full Subnet Manager plus extensive monitoring, analytics, diagnostics, and management capabilities — all accessible through a web interface and REST API.

Key Characteristics

UFM Architecture Components

1. UFM Server — Core management engine with embedded SM
2. UFM Web UI — Browser-based management interface
3. UFM Agents (optional) — Installed on compute nodes for detailed telemetry
4. Database — Stores historical data, events, and topology snapshots

Additional Capabilities (Beyond OpenSM)

Monitoring & Analytics

• ✅ Real-time performance monitoring (bandwidth, latency, errors)
• ✅ Historical data collection and trend analysis
• ✅ Per-port counters and statistics
• ✅ Cable health and temperature monitoring

Visualization & Topology

• ✅ Interactive fabric topology maps
• ✅ Color-coded port status visualization
• ✅ Rack and chassis views

Advanced Management

• ✅ Firmware update management
• ✅ Job-aware routing (Slurm, PBS, and other scheduler integration)
• ✅ Quality of Service (QoS) configuration
• ✅ Multi-fabric management from a single console
• ✅ Prometheus / Grafana exporters

Best Suited For

• Large-scale HPC clusters (hundreds to thousands of nodes)
• Production data centers and AI/ML training clusters
• Environments requiring detailed monitoring and analytics
• Organizations with compliance or audit requirements (historical data retention)

Side-by-Side Feature Comparison

How to Identify Which One You Have

Check if UFM is installed locally

rpm -qa | grep -i ufm
systemctl list-units | grep -i ufm
ps aux | grep ufm

Check if OpenSM is installed / running locally

rpm -qa | grep -i opensm
systemctl status opensm
ps aux | grep opensm

Identify the active SM on the fabric

# Query the SM (requires at least one Active port)
sminfo

# Discover the fabric and look for SM entries
ibnetdiscover | grep -i "sm"

Common Deployment Patterns

Recommendations for Your Current Situation

If you have a switch-embedded SM

1. Log in to the switch management interface
2. Check SM status: show sm
3. Restart the SM service (NVIDIA/Mellanox switches):

sm stop
sm start

If you have a UFM server

1. Contact the UFM administrator, or log in to the UFM server directly
2. Restart the UFM service:

sudo systemctl restart ufmd

If you are unsure which SM you have

• Ask your cluster administrator or network team
• They should know what SM infrastructure is in place
• Run rpm -qa | grep -E "ufm|opensm" to check what is installed locally

Summary

Both tools can manage your InfiniBand fabric, but UFM provides far greater visibility and control at scale. Your current problem is that the primary SM — wherever it runs — has stopped working correctly. The path to recovery is to either fix the remote SM (switch or UFM) or bring up a new SM instance once the root cause of the conflict has been resolved.