TransferBench v1.70.00

v1.70.00

Added

• Adding support for Tensor Data Mover (TDM)-based executor [T] on supported hardware (gfx1250, NVIDIA sm_90+ via TMA).
  This provides an alternative data movement mechanism which utilizes async loads to shared memory / from shared memory
• Added support for SWEEP_MIN_POW2 and SWEEP_MAX_POW2 to set sweep bounds when bytes to transfer is 0
• Added new "tdmsweep" preset that sweeps TDM executor options (block size / LDS / block order / subExecs)
• Added TB_SEND_USLEEP to insert a configurable microsecond delay after each socket SendData call (default: 0); useful for diagnosing small-message timing issues on sensitive clusters
• Adding ppodId / vpodId printing to verbose mode
• Added VALIDATE_ON_DEVICE to validate GPU destination (and source) memory via an on-device kernel instead of
  copying back to the host. Expected values are pre-uploaded during preparation; only a mismatch count and the
  first mismatch offset are returned. Takes precedence over VALIDATE_DIRECT for GPU destinations.

Modified

• a2a, p2p, rings, poda2a, and podp2p presets now support the TDM executor
• CPU NUMA nodes with 0 cores will now be hidden. To re-enable, set TB_SHOW_ALL_NUMA=1
• Switching to use of persistent threadpools to cut-down on thread creation overheads
• Updating default GFX unroll on GFX1250 to 32
• Improved socket communicator robustness (TCP_NODELAY, partial send/recv handling, MSG_NOSIGNAL)
• Improved mismatch logging and smoketest fail reporting
• Destination memory is now cleared after each iteration when ALWAYS_VALIDATE is enabled, so each iteration starts from a known-zero state

Fixed

• Guard before ibv_free_device_list to avoid invalid free
• Fix NIC to GPU proximity detection on systems with multiple PCIe domains

TransferBench v1.69.01

v1.69.01

Added

• Added support for ABI change introduced in amd-smi 27.0.0 (ROCm 10.0)

TransferBench v1.69.00

v1.69.00

Added

• Added support for ALWAYS_VALIDATE=-1 to disable validation

Fixed

• Fix for non-zero byte offsets used with DMA executor
• Explicit copy back to host for GPU subExecParam on platforms without large BAR
• Add missing device synchronization after hipMemcpy in RunTransfers()
• Fix for out of bound dstReference array when there are no transfers
• Disable and warn a2a and poda2a presets when running on single GPU without A2A_LOCAL

Modified

• TransferBench now dynamically loads IB verbs, and its dependency and support for NIC executor is checked in runtime
  • Created a top level third-party/ folder for ibverbs related files. Will also harbor future external source which TransferBench depends
  • Created a separate minimal header IbvHeader.hpp for ib verbs structs and IbvDynLoad.hpp for dynamic loading and status report for ib verbs functionality.
  • Dynamic loading is a singleton and done once per process, and TransferBench header will probe in runtime if basic ibverbs function as well as dmabuf export is supported.
  • Also got rid of HAVE_DMABUF_SUPPORT macro. Got rid of redundant dependency check on hsa header and rocr binaries (they are mandatory for AMD platform) in build process. Similar to ibv, it now dynamically checks for hsa_amd_portable_export_dmabuf symbol as part of check kernel support, and returns dmabuf support in runtime.

TransferBench v1.68.00

v1.68.00

Fixed

• Improper draining of writes that could artificially inflate transfer timing (bug introduced in v1.67.00)
• Potential timing bug when running GFX Executor in warp-subexecutor mode with small data sizes
• Keeping subiteration threads in sync per subiteration

TransferBench v1.67.00

v1.67.00

Highlights

This release of TransferBench enables scale-up support, and introduces a number of new presets.
To get more information about available presets, run TransferBench presets. The helper script LaunchTransferBench.sh simplifies using TransferBench across multiple nodes without mpi.

Added

• Added NIC_TRAFFIC_CLASS to set the DSCP/traffic class byte in the RoCE GRH for QPs (RoCE only)
• Added NIC_SERVICE_LEVEL to set the IB service level (sl) for QPs (IB and RoCE)
• Initial support for pod communication. Requires compatible hardware / ROCm version and subject to further testing
  • This potentially enables GFX/DMA executors to access SRC/DST memory locations on GPUs within the same pod
  • Pod membership requires amd-smi however can be skipped by setting TB_FORCE_SINGLE_POD=1
• Support for dumping executed Transfers to a config file specified by TB_DUMP_CFG_FILE
  • This will write Transfers that are executed (for example via a preset) to a config file that can then be executed
• Reporting number of iterations run when running in timed mode (NUM_ITERATIONS < 0)
• Adding NIC_CQ_POLL_BATCH to control CQ poll batch size for NIC transfers
• New "hbm" preset which sweeps and tests local HBM read performance
• Added a new TB_WALLCLOCK_RATE that will override GPU GFX wallclock rate if it returns 0 (debug)
• Adding new batched-DMA executor "B", which utilizes the hipMemcpyBatchAsync API introduced in HIP 7.1 / CUDA 12.8
• Added new "bmasweep" preset that compares DMA to batched DMA execution for parallel transfers to other GPUs
• Added new "wallclock" preset that compares wallclock counters across XCCs within a GPU
• Added new "smoketest" preset that runs a variety of DMA/GFX tests for simple correctness tests
• Added new "help" preset to show config file examples
• Added new "presets" preset to show available presets and their descriptions
• Added new "rings" preset that runs parallel rings of transfers (pod-capable)
• Added new "envvars" preset to show environment variables that can change TransferBench behavior
• Adding information on how to run multi-rank with TransferBench, when run with no args
• Added new "nica2a" preset (NIC all-to-all over GPUs via NIC executors, multi-node)
• Added new GFX_KERNEL to allow experimenting with copy-only GFX kernel. Currently this is opt-in only
• Added SHOW_PERCENTILES (e.g. 50,75,90,95,99) to show empirical percentiles of per-iteration duration
• Adding new LaunchTransferBench.sh script to simplify launching TransferBench across multiple nodes (via socket)
• New empty preset (EmptyKernel) to measure empty-kernel launch latency with BATCHSIZES/GRIDSIZES/BLOCKSIZES sweeps

Modified

• DMA-BUF support enablement in CMake changed to ENABLE_DMA_BUF to be more similar to other compile-time options
• Adding extra information to CMake and make build methods to indicate enabled / disabled features
• a2asweep preset changes from USE_FINE_GRAIN to MEM_TYPE to reflect various memory types
• a2asweep preset changes from NUM_CUS to NUM_SUB_EXECS to match with a2a preset naming convention
• scaling preset changes from using USE_FINE_GRAIN to CPU_MEM_TYPE and GPU_MEM_TYPE
• NIC_FILTER renamed to TB_NIC_FILTER for consistency
• DUMP_LINES renamed to TB_DUMP_LINES for consistency
• Dynamically size CQs for NIC transfers in high QPs case
• Switch to using hipMemcpyDeviceToDeviceNoCU instead of hipMemcpyDefault for DMA Executor if available (requires HIP >= 6.0)
• Allow for multiple destination memory locations for DMA/Batched-DMA Transfers
• Removed env vars printing and preset print when running TransferBench with no args
• Modification to simplify socket comm usage - first rank only needs to set TB_NUM_RANKS=X to see connection info

rocm-7.2.4

ROCm release v7.2.4

rocm-7.2.3

ROCm release v7.2.3

rocm-7.2.2

ROCm release v7.2.2

rocm-7.2.1

ROCm release v7.2.1

TransferBench v1.66.02

v1.66.02

Added

• Adding DMA-BUF support
  • TransferBench now supports GPU memory registration via DMA-BUF handle
    • Disabled by default in build time
    • To enable, DISABLE_DMABUF must be explicitly turned off for both CMake and Make process
  • DMA-BUF registration relies on both IB verbs registration and HSA dmabuf export functionality
    • Support for both are checked in compile time as well as runtime
• New preset: nicp2p
  • A point to point performance sweep for all paris of NIC on all nodes
  • The preset requires all participating nodes have identical topology
  • All NICs on every node will bind to closest device of user specified memtype, and communicate with all
    other NICs in the potentially multi-node world
  • Execution of node pairs, as well as NIC pairs under the same node pair, are parallelized by default
• Adding GPU affinity check for NIC devices

Fixed

• Fixing some typo in result table and NIC verbose output