LFT and Topology Exports in InfiniBand: Diagnose the Fabric Without Expanding the Attack Surface
LFT and Topology Exports in InfiniBand: Diagnose the Fabric Without Expanding the Attack Surface
Summary. InfiniBand routing and topology data are indispensable for troubleshooting, capacity analysis, and incident response. They are also operationally sensitive. This article explains what LFTs, topology exports, UFM snapshots, SA keys, and umad actually do—and offers a practical framework for deciding when, where, and how such data may be collected and stored.
In a large AI or HPC cluster, a troubleshooting request often sounds harmless: “Can we take a snapshot of the routing tables and analyze it on a compute node?” The correct answer is not an automatic yes or no. The answer depends on what artifact is being collected, which privilege is used, which node receives it, and what controls remain in force after collection.
Why this data deserves protection
An InfiniBand switch’s Linear Forwarding Table (LFT) maps destination LIDs to egress ports. In other words, it helps explain how unicast traffic is forwarded through the fabric. NVIDIA documents that the Subnet Administrator (SA), which runs with the Subnet Manager (SM), can supply topology and switch-forwarding-table information in response to information or registration requests.[1] NVIDIA’s fabric utilities also document that ibroute uses Subnet Management Packets (SMPs) to display unicast LFT or multicast forwarding-table data for a switch, while ibnetdiscover can output a full topology including GUIDs, node types, port numbers, port LIDs, and node descriptions.[3]
None of those fields is necessarily a secret in the cryptographic sense. Together, however, they can form a high-value operational map: which systems are connected, where they sit, and which paths carry traffic. That is valuable to operators during an outage; it is also useful reconnaissance material if exposed to an unauthorized party. Treat the artifact according to its combined operational sensitivity, not merely according to whether any one field looks confidential.
| Artifact | Typical contents | Recommended sensitivity | Preferred storage location |
|---|---|---|---|
| Derived health report | Aggregated counts, redacted findings, selected KPIs | Internal operational data | Approved analytics or reporting service |
| LFT / topology export | Paths, GUIDs, LIDs, ports, links, descriptions | Sensitive infrastructure metadata | Protected management or security-analytics environment |
| Full UFM snapshot | UFM database and configuration; optionally logs and system troubleshooting data | Highly sensitive administrative backup | Restricted backup repository or management plane |
Do not confuse an LFT export with a UFM snapshot
This distinction matters. A targeted LFT or topology export is a diagnostic artifact with a relatively narrow purpose. A UFM snapshot is broader: NVIDIA describes it as an export of UFM database information and configuration files, with optional system snapshots and UFM logs for troubleshooting.[2] Its exact contents depend on the UFM release and the options selected. Therefore, a full snapshot should be handled as a privileged backup by default, even if the immediate analysis only needs routing information.
Design rule: collect the smallest artifact that answers the operational question. Do not move a full management backup merely because an LFT comparison is needed.
umad, SA_Key, and permissions: three different things
Linux exposes a user-space Management Datagram interface through per-port devices such as /dev/infiniband/umad0.[4] This is what engineers commonly mean by umad: a path that lets user-space programs exchange InfiniBand management datagrams. It is not an Ethernet-style ICMP equivalent, and it is not, by itself, permission to change the fabric.
Likewise, an SA_Key should not be described casually as a universal “network-admin key.” NVIDIA characterizes InfiniBand keys as access tokens rather than cryptographic encryption keys, and documents SA_Key as the mechanism for identifying trusted requests for sensitive SA operations such as setting or deleting records.[1] Whether a particular query, tool, or table access requires a key depends on the management class, the SM/UFM configuration, device security policy, and the software release. The safe engineering principle is straightforward: never distribute a management credential to a compute node merely because a diagnostic program can use it.
What can go wrong on a compute node?
Compute nodes are often multi-tenant or run rapidly changing user workloads. Their attack surface is normally broader than that of a hardened UFM/SM or backup host. Moving a sensitive export there can create three distinct classes of risk.
| Security objective | Failure mode on a compute node | Operational consequence | Primary mitigation |
|---|---|---|---|
| Confidentiality | A job, local privilege escalation, shared filesystem, or backup agent reads the artifact | Fabric topology and routing intelligence is disclosed | Keep raw exports off compute nodes; encrypt and minimize when transfer is unavoidable |
| Integrity | Artifact is altered before analysis or automation consumes it | False diagnosis, unreliable comparisons, or unsafe operator decisions | Read-only handling, signed hashes, provenance records, and immutable object storage |
| Availability | Many nodes launch broad management queries at once | Management-plane load, slow diagnostics, or degraded incident response | Single controlled collector, scheduling, rate limits, and job admission controls |
| Credential safety | SA/UFM credentials are copied to a node to enable collection | Potential expansion from data exposure to management-plane compromise | Keep credentials in a protected management domain; issue narrowly scoped, short-lived access where supported |
The last row is usually the decisive one. A topology file can be sensitive; a leaked management credential can be far worse. Separating data export from credential possession is therefore more important than setting a restrictive mode bit after the fact.
A safer operating model
Use a protected management-plane collector or a dedicated security-analytics host to run approved collection jobs. That host should have controlled access to UFM/SM services, central logging, restricted administrator access, and a documented retention policy. It should create an LFT/topology export only when a real diagnostic need exists, derive the result required by the requester, and return the smallest useful output.
For example, a scheduler or capacity-planning workflow may only need “path diversity per rack,” “non-optimal-link count,” or a signed diff between two topology states. It does not need a permanent copy of every GUID, LID, link, and forwarding entry on every worker node. Returning a derived result also makes authorization more legible: the service can permit a team to see a health conclusion without giving that team a reusable fabric blueprint.
If a compute node must receive data
There are exceptional cases—isolated forensic analysis, a tightly controlled validation appliance, or a dedicated analysis node—in which a compute-class host may receive a dataset. Such an exception should be explicit, time-bounded, and reviewed. The host should not be a general-purpose user node.
At minimum, transfer only the reduced artifact needed for the analysis; encrypt it in transit and at rest; keep decryption keys outside the node’s ordinary filesystem; enforce service-account-only access; record every retrieval; verify integrity with a signed manifest or cryptographic hash; and delete the object automatically at the end of its retention window. A read-only mounted repository is often preferable to copying the source artifact. These measures reduce risk, but they do not make a broadly shared compute node equivalent to a management enclave.
A practical decision test
Before approving collection or transfer, answer four questions in order. First, is the request asking for a derived finding, an LFT/topology export, or a full UFM snapshot? Second, can the question be answered from the management plane without copying raw data? Third, does the requested destination host run untrusted workloads or have broad user access? Finally, does the workflow require a persistent credential on that host?
If the answer to the third or fourth question is yes, the default should be to keep the raw artifact and all credentials off the compute node. If only a derived report is required, produce that report centrally. If an export is truly necessary, use a controlled collector and a short-lived, encrypted transfer path. If the request is for a full UFM snapshot, use a restricted backup repository rather than a compute environment.
Conclusion
Collecting LFT and topology information is a normal and valuable part of InfiniBand operations. The risk begins when convenience erases the boundary between the management plane and the compute plane. Use umad and diagnostic utilities as controlled management mechanisms, not as reasons to distribute management access. Keep full UFM snapshots in protected backup locations, centralize raw LFT/topology collection, and expose only the smallest verified result that the consumer needs. This preserves diagnostic capability while preventing routine observability from becoming an unintentional control-plane exposure.