Optimization of a 2D Jacobi heat solver on SuperMUC-NG at the Leibniz Supercomputing Centre, from compiler flags on one core to 48 threads on a node and hybrid MPI/OpenMP over several nodes. A node has two 24-core Intel Xeon Platinum 8174. Each speedup below is explained by a measured mechanism, mostly the bytes moved per memory level, which LIKWID reads from the hardware counters.
| Step | Result | Mechanism |
|---|---|---|
| Compiler flags, 1 core | 16.4x over -O0 at n = 4500 |
with -xCORE-AVX512 the compiler interchanges and vectorizes the loop nest, which cuts L1–L2 traffic against -O3 from 1120 to 175 B per point update |
| Hand optimization, 1 core | 3.83x over the compiler-optimized build at n = 6000 | residual fusion, pointer swap and temporal blocking, each of which removes DRAM traffic |
| OpenMP, 48 cores | 28.2x over the best single-core version at n = 2000, 18.7x at n = 6000 | at n = 2000 the grid share of each core moves into cache; at n = 6000 plain threading saturates at 81% of the node's DRAM bandwidth, and temporal blocking lowers the traffic |
| NUMA-aware initialization | 7.5x at n = 2000 | one hidden write in std::vector::resize removed |
Both final single-core versions against the compiler-optimized build V1, each metric as percent of V1 at the same grid size, lower is better.
Miss ratios and traffic. At n = 4500 the fastest build has the highest L2 miss ratio, 0.229 against 0.141, while it moves 6.4 times less L1–L2 traffic. A miss ratio is a quotient of two quantities which an optimization changes by different factors. Every comparison here therefore uses absolute bytes per point update and names the memory level.
Contribution of each transformation. At n = 6000, swapping pointers instead of copying gives 2.17x, fusing the residual into the stencil 1.33x and temporal blocking 1.38x. Hand-written AVX-512 intrinsics give 1.00x. They issue 72% more instructions than the same algorithm in plain C, while cycles and traffic per point stay identical, since instruction issue is not the binding resource of a memory-bound kernel.
Control version. M0 is the original kernel inside the new driver. It ran 6% faster than the original program, with identical instructions, cycles and traffic per point. The original program allocates with calloc and writes the array first inside the timed region, so its kernel also pays for zeroing the pages. Against this control, the kernel-only speedups at n = 6000 are 2.61x for the intrinsics kernel and 3.60x with temporal blocking.
DRAM traffic accounting. For the versions with a copy loop, the measured DRAM reads sat exactly one 8 B unit per point below the hand count, while the writes matched it. A read which disappears at the memory controller and belongs to a store is a skipped read-for-ownership. The disassembly shows one call to _intel_fast_memcpy, which switches to non-temporal stores above a size threshold. The pointer swap is therefore 2.17x faster than a copy which already uses non-temporal stores. The same split of reads and writes corrected the temporal blocking target from 24/T to 32/T B per point, which the measurement matches within 1 to 5%.
First touch. Linux places a page on the socket of the thread which writes it first. std::vector::resize value-initializes, so the master thread wrote every page before any worker thread existed, and the parallel fill afterwards changed nothing. The per-socket memory counters showed a traffic imbalance of 591x. An allocator whose construct() default-initializes removes this write, after which both sockets carry the same traffic within 1% and the plain OpenMP variant runs 7.5x faster at n = 2000.
Blocking depth on the full node. On one core, performance peaks at the largest depth T whose wavefront still fits the 1 MiB L2. With 48 threads every thread recomputes T ghost rows at both slab edges, and the single-core depth made 55% of the work redundant at n = 2000. Tuned on the full node, the depth moved in opposite directions, from 22 to 8 at n = 2000 and from 6 to 16 at n = 6000. At n = 2000 the redundant work dominates, at n = 6000 the DRAM bandwidth.
Thread placement. KMP_AFFINITY=compact packed all 48 threads onto the 24 cores of one socket and ran 2.08x slower than OMP_PROC_BIND=close. SLURM's default CPU binding overrides OMP_PLACES without a message. Every run therefore prints its affinity mask, which the analysis parses and checks.
Speedup against thread count on one node, physical cores only, each variant scaled against the sequential kernel it threads, without and with NUMA-aware initialization. The star marks temporal blocking with the depth tuned at 48 threads.
src/hybrid/ distributes the grid over a two-dimensional Cartesian grid of MPI processes, and each process splits its subdomain over its OpenMP threads. The last process in each direction takes the remainder, and rows are padded to a multiple of 8 doubles. Each iteration starts a non-blocking halo exchange, updates the interior of the subdomain which needs no halo, then waits and updates the outer ring. The west and east columns travel through contiguous buffers, which at the physical boundary keep the boundary values, since a receive from MPI_PROC_NULL leaves them unchanged. Only the master thread calls MPI (MPI_THREAD_FUNNELED). The grid is zeroed with the same static schedule as the interior update, so first touch places the rows of every thread in the memory of its own socket. Built without OpenMP, the same sources give the pure MPI version.
The scalar column updates sum the neighbours in the same order as the SIMD stencil. Every process grid and thread count therefore produces the grid of the single-node SIMD kernels, bit for bit in the portable build. The tests check this for seven process grids without threads and for four with 2 or 3 threads per process. The image is assembled on rank 0 from the points each rank owns. Two job scripts run the pure MPI scaling from 1 to 192 processes and seven hybrid configurations on four nodes. The hybrid configurations range from 192 processes with one thread to 4 processes with 48 threads, and each of them uses all 192 cores. The MPI and hybrid versions were developed together with Hinrich Thiele.
tests/compares every variant against an independent NumPy solver with the same boundary and the same floating-point operation order. Grids are compared bit for bit, residuals to the printed precision and images pixel for pixel.- The 85 cases cover full and masked vector widths, several resolutions in one run, convergence before the iteration limit, 1 to 4 OpenMP threads, blocking depths of 1, 3 and 7, and seven MPI process grids.
- The intrinsics map to a scalar header in
portable/, so the suite runs on any CPU. That build turns off FMA contraction, which makes a bitwise comparison possible. CI runs the suite on every push, also with the real AVX-512 instructions when the runner supports them. - The suite also passes under AddressSanitizer and UndefinedBehaviorSanitizer. Reintroducing four defects into the MPI code, one at a time, fails between 2 and 14 cases each, which shows the suite can detect them.
- The temporal blocking does not reorder any arithmetic, so its grid is bitwise identical to the plain kernel. Before any measurement, the wavefront schedule was verified on a scalar model over 8063 (n, T) cases, and a mutated schedule with the time levels in descending order fails 6120 of them.
- The allocator change has its own control. A poison test shows that
construct()leaves every double untouched, whilestd::allocatoroverwrites all of them.
- Hardware. SuperMUC-NG batch nodes, 2 sockets of Intel Xeon Platinum 8174 (Skylake-SP) with 24 cores each. The nodes run at 1.9 GHz under a userspace governor, which the counters confirm within 0.24%, so the double-precision peak is 60.8 GFlop/s per core. No AVX-512 frequency drop occurs at any thread count.
- Tools. Intel icpx 2025.3.0 and LIKWID 5.3.0 with marker regions around the iteration loop, plus
objdumpand the optimization reports of the compiler, checked against each other. - Ceilings. Single-core bandwidth measured with
likwid-bench -t triad_avx512_fmaand corrected by 40/32 for the write-allocate fill which the benchmark does not count, 86.05 GB/s for L2 and 15.59 GB/s for DRAM. - Protocol. Exclusive nodes, one change per comparison, medians of 10 repetitions on one core and of 5 on the full node. Time per point update is derived from the printed rate, since the printed seconds carry only three decimals.
- Units. Every byte figure is per point update and names its level, L1–L2, L2–L3 or DRAM.
.
├── src/
│ ├── single_core/ M0 to M5, one optimization per step
│ ├── openmp/ plain OpenMP and temporal blocking with ghost slabs
│ └── hybrid/ MPI across nodes and OpenMP within each process, also built as pure MPI
├── portable/ scalar stand-in for the AVX-512 intrinsics, for tests on any CPU
├── tests/ NumPy reference solver and the comparison suite
├── studies/ job scripts, raw data, analysis and figures of the three studies
└── inputs/ input files of the measurements
studies/README.md maps the studies to the course repository in which they ran, and lists what changed in the code since.
# SuperMUC-NG with Intel oneAPI: every variant into build/native/
make
build/native/M5_temporal_blocking inputs/test6000.dat
OMP_NUM_THREADS=48 OMP_PLACES=cores OMP_PROC_BIND=spread build/native/openmp_plain inputs/test2000.dat
mpiexec -n 16 build/native/mpi inputs/test6000.dat 4 4 # 4 x 4 process grid
sbatch src/hybrid/job_hybrid.scp # seven hybrid configurations on four nodes
# any machine with a C++20 compiler, MPI and NumPy: portable build and the test suite
make PORTABLE=1 test
# every table and figure of a study, from its raw data
bash studies/single_core/regen_all.shThe SLURM sessions in studies/ carry the exact settings of every measurement. On macOS, Apple clang has no OpenMP, so the portable build needs CXX=clang++ from Homebrew LLVM.
- G. Wellein, G. Hager, T. Zeiser, M. Wittmann and H. Fehske, "Efficient temporal blocking for stencil computations by multicore-aware wavefront parallelization," COMPSAC, 2009, pp. 579–586.
- M. E. Wolf and M. S. Lam, "A data locality optimizing algorithm," PLDI, 1991, pp. 30–44.
- D. Wonnacott, "Using time skewing to eliminate idle time due to memory bandwidth and network limitations," IPDPS, 2000, pp. 171–180.
- K. Datta, M. Murphy, V. Volkov, S. Williams, J. Carter, L. Oliker, D. Patterson, J. Shalf and K. Yelick, "Stencil computation optimization and auto-tuning on state-of-the-art multicore architectures," SC, 2008.
MIT, see LICENSE.

