Benchmarking ALCF systems with torch-hammer

Nine GPU benchmarks on Sunspot (PVC Max 1550) and Sirius (A100). Aurora and Polaris are queued. PVC Max 1550 measured 8.4× A100 on Sparse MM.

torch-hammer is an HPE-maintained GPU benchmark suite. It runs 9 workloads, from batched GEMM and stencil PDEs to sparse matmul. Telemetry uses NVML, ROCm-SMI, and xpu-smi. It writes CSV and HTML fleet reports and aggregates multi-GPU results with multiprocessing.spawn. One codebase supports CUDA, ROCm, XPU, and MPS.

Results

Both completed runs used the same 9 benchmarks, --stress-test, and HBM-scaled sizing. Aurora and Polaris were queued in debug behind reservations when this snapshot was recorded.

1-GPU stress (HBM-scaled, float32):

All-GPUs fleet (stress, per-node):

PVC Max 1550 measured 2,028 GFLOP/s on Sparse MM; A100 measured 243 GFLOP/s, an 8.4× difference on this unstructured sparse workload. A100 led the other reported benchmarks or tied Batched GEMM.

Neither system completed 3D FFT. XPU returned an MKL DFT “unimplemented” error for the configured shape. Sirius ran out of memory at 11.75 GiB under stress fill on its 40 GB GPU; earlier benchmarks already held about 30 GiB.

What was needed per machine

Sirius defaults to Python 3.6, which raises a SyntaxError on from __future__ import annotations. It has no mpirun, srun, or pbsdsh, and the report tool also fails under Python 3.6. The run used miniconda3/envs/cu122-pt221 with Python 3.11 and torch 2.2. A single-node --all-gpus fallback uses torch.multiprocessing.spawn to run on all 4 A100s.

The same PBS template works on the larger systems with per-system account, queue, and filesystem values. Each host uses one rsync staging step for torch-hammer.py, reports/, and config-examples/ in ~/thammer-staging.

How the run is structured

Every job does the same four phases:

1. Single-GPU quick: all 9 benchmarks at small sizes with --device-index 0. This checks the run and exercises the CSV, JSON, and HTML report paths.
2. Single-GPU stress: all 9 benchmarks with --stress-test, sized to fill about 50% of free HBM. The memory benchmark measured 0.3 GB/s before the sizing fix and 27.7 GB/s after. The earlier run used a 4 KiB buffer and measured randint overhead.
3. Per-node all-GPUs: the gemm, conv, einsum, and memory-traffic subset under stress. One mpiexec rank runs per node and starts the node’s GPUs.
4. Reports: hammer_report.py reads each CSV and JSON artifact. A fleet rollup reads all-benchmarks/.

Phase 3 is what produces the fleet verdict. Mixing quick + stress CSVs in one directory gives a spurious 0/N OUTLIER report (different problem sizes) — isolated per-phase outputs are the correct inputs to the fleet report.

Detailed results

Sunspot — Intel Data Center GPU Max 1550

2 nodes, 12 PVC tiles each, 24 GPUs total. Job 12483256, walltime 00:12:18, via ezpz_setup_env → frameworks/2026.1.0 (torch 2.13 + XPU).

Single-GPU stress (head tile):

Fleet (4 benches, stress, both nodes):

Batched GEMM       18.0 TFLOP/s  CV 0.7%
Convolution         1.49M img/s  CV 1.0%
Einsum             13.9 TFLOP/s  CV 5.7%
Memory Traffic     27.0 GB/s    CV 4.6%
FLEET VERDICT: 2/2 PASS

No power/temp columns — xpu-smi / Level Zero telemetry is not yet wired on the compute image; IntelTelemetry currently returns N/A for those fields.

Sirius — NVIDIA A100-SXM4-40GB

1 node, 4 visible A100s (2 allocated; head-node fallback). Job 33426 (retry after pbsdsh fix), walltime 00:02:08 / 00:01:14, via miniconda3/envs/cu122-pt221 (torch 2.2, CUDA 12.1, driver 535).

Single-GPU stress (head GPU):

Fleet (4 benches, stress, 4 GPUs):

Batched GEMM       18.9 TFLOP/s  CV 0.2%  313 W  48 °C
Convolution         9.39M img/s  CV 0.2%  386 W  52 °C
Einsum             17.1 TFLOP/s  CV 0.2%  339 W  53 °C
Memory Traffic     29.6 GB/s    CV 0.0%  191 W  52 °C
FLEET VERDICT: 1/1 PASS

Full NVML telemetry — power, temp, SM util, clocks — in every row.

Aurora & Polaris — queued

Both jobs were queued in debug behind unrelated reservations and a 58-deep prod routing queue. Their results are not included here.

Reproducing the runs

• Stage torch-hammer.py, reports/, and config-examples/ in ~/thammer-staging on each host.
• Set the system account, queue, filesystem, environment, and MPI launcher in the PBS template.
• Keep report inputs separated by phase. Mixing quick and stress CSVs produces a 0/N OUTLIER result because the problem sizes differ.
• Run hammer_report.py on each single-GPU, stress, and fleet artifact.

Artifacts

• Code: saforem2/torch-hammer, branch feature/intel-xpu-support (XPU backend and --stress-test HBM fix)
• HTML report snapshots: combined 1-GPU comparison and stress comparison; Sunspot compact 1-GPU, full 1-GPU, 24-GPU fleet, and fleet rollup; Sirius compact 1-GPU, full 1-GPU, and 4-GPU fleet.
• Jobs: Sunspot 12483256 (F 00:12:18), Sirius 33426 (E 00:02:08 → R 00:01:14 after fix)