A head-to-head CUDA vs Mojo GPU kernel shootout on NVIDIA Ampere (RTX 3090,
sm_86). Three kernels (reduction, softmax, matmul), each written twice, optimized
as far as is reasonable by hand, timed with the same methodology, and plotted
against the hardware roofline.
The question this answers: how close does hand-written Mojo get to hand-written CUDA (and to the vendor libraries, cuBLAS / CUB / torch) on the same GPU, for the same algorithm?
Clocks pinned at SM 1695 MHz for the run; each number is the median of 30
timed samples and ± is the 25/75 inter-quartile range (IQR). "Indistinguishable"
means the CUDA and Mojo IQRs overlap, so the difference is inside the measurement
noise; where they do not overlap the gap is real and quoted. Raw rows and the
run-env.txt clock record are in results/.
| Kernel | Metric | Best CUDA | Best Mojo | Mojo / CUDA | Vendor ref | Verdict |
|---|---|---|---|---|---|---|
| reduction (256M f32) | GB/s | 889 (warp_shfl) | 895 (warp_shfl) | 100.7% | 895 (CUB) | indistinguishable |
| softmax (16384×1024) | GB/s | 722 (online) | 728 (online) | 100.8% | 824 (torch) | indistinguishable |
| softmax (1024×16384) | GB/s | 544 (online) | 504 (online) | 92.6% | 643 (torch) | CUDA +8% |
| matmul (1024³ f32) | GFLOP/s | 9 279 (regblock) | 8 497 (regblock) | 91.6% | cuBLAS 18 118 | CUDA +9% |
| matmul (4096³ f32) | GFLOP/s | 17 370 (regblock) | 13 780 (regblock) | 79.3% | cuBLAS 23 385 | CUDA +26% |
The picture splits cleanly by what limits each kernel:
- Memory-bound (reduction, softmax): both languages sit near the 936 GB/s
GDDR6X ceiling and are statistically indistinguishable at the sizes that
saturate the bus. The hand-written Mojo reduction matches both the hand-written
CUDA kernel and
cub::DeviceReduceto within ~6 GB/s at 256M (all 95-96% of the bus). Softmax is a dead heat on square/wide-block shapes; only the very wide 1024×16384 shape opens a real ~8% gap, where the per-block reduction overhead is a larger share of a short kernel. - Compute-bound (matmul): once both sides stage A/B through
float4-vectorized loads with an identical 8×8 register-tile inner loop, Mojo reaches 92% of the CUDA kernel at 1024³ and settles at ~80% at 2048³/4096³. The residual gap is double buffering (overlapping the next tile's global loads with the current tile's FMAs) and instruction scheduling, which cuBLAS and the tuned CUDA kernel still do better. It is not memory movement and not the arithmetic.
On methodology: the previous version of this table reported single-run figures and, worse, compared a CUDA minimum against a Mojo mean. Both sides now report the median over 30 locked-clock samples with the IQR, which is why some gaps moved: the old "95% / 87%" memory-bound gaps were measurement artifacts (they are now dead heats), and the old "84% at 4096³" matmul was an optimistic single run (the honest locked-clock median is ~80%).
Benchmarks are only interesting if they translate into a bill. Take the compute-bound case, the only one where the languages actually differ, and price it. A 3090 rents for roughly $0.22/hr on community clouds (mid-2026 spot; the numbers below scale linearly with whatever price you pay).
Cost to push 1B tokens through one 4096-wide float32 projection (a single
(1e9 × 4096) · (4096 × 4096) SGEMM = 33.6 PFLOP), at the measured 4096³ rates:
| impl | GFLOP/s | wall time | cost / 1B tokens | vs CUDA |
|---|---|---|---|---|
| cuBLAS | 23 385 | 23.9 min | $0.088 | 0.74× |
| CUDA regblock | 17 370 | 32.2 min | $0.118 | 1.00× |
| Mojo regblock | 13 780 | 40.6 min | $0.149 | 1.26× |
So on this fp32 kernel the Mojo implementation costs ~26% more per token than the hand-written CUDA one, and ~70% more than cuBLAS, purely from the throughput gap. On the memory-bound kernels the implementations are within measurement noise, so they cost the same to run.
Caveats, stated plainly: this is a single fp32 GEMM with no tensor cores, so it is a language-vs-language cost of the same kernel, not a production inference cost (real serving uses fp16/bf16 tensor cores and is an order of magnitude cheaper per token). The point is the shape of the answer: for bandwidth-bound work Mojo is already free of penalty; for compute-bound work the ~20% throughput gap is also a ~25% cost gap, and closing it (double buffering) is worth real money at scale.
kernels/
reduction/ { cuda/reduction.cu mojo/reduction.mojo README.md }
softmax/ { cuda/softmax.cu mojo/softmax.mojo README.md }
matmul/ { cuda/matmul.cu mojo/matmul.mojo README.md }
bench/ Makefile, run.sh (harness -> CSV), plot.py, profile.sh (ncu),
vendor_softmax.py (optional torch ref)
results/ results.csv (raw), run-env.txt (clocks), *.png (charts), ncu/ (profiles)
docs/ methodology.md, roofline-3090.md, portability.md
Every kernel program is self-contained: it runs its own size sweep, checks
correctness against a reference, and prints rows in one shared CSV schema
(kernel,impl,variant,dtype,m,n,k,median_ms,p25_ms,p75_ms,n_runs,gflops,gbytes_s,correct).
See docs/methodology.md.
Mojo runs on Linux; this repo is developed under WSL2 (Ubuntu) with the NVIDIA
driver exposing the 3090. You need CUDA (nvcc) and a uv environment with
Mojo + MAX.
# 1. Python env with Mojo + MAX (the max package ships the GPU host runtime and
# the layout library; the bare mojo wheel alone cannot launch kernels).
uv sync
# 2. (optional but do it before step 4) the torch vendor reference for softmax
# (~2.5 GB CUDA build). Install it first so the harness can emit the vendor
# column; without it the torch rows are simply absent from results.csv.
uv pip install --python .venv/bin/python torch --index-url https://download.pytorch.org/whl/cu124
# 3. (recommended) pin GPU clocks so the medians are not chasing the boost
# governor. Under WSL this MUST be issued from an Administrator shell on the
# Windows host, not inside WSL (the guest does not own the GPU) -- see the
# "Locking GPU clocks under WSL" section of docs/methodology.md for why:
# nvidia-smi -lgc 1695,1695 && nvidia-smi -lmc 9751 (reset: -rgc / -rmc)
# 4. Build + run everything -> results/results.csv + results/run-env.txt
./bench/run.sh
# 5. Charts
uv run bench/plot.py
# 6. (optional) Nsight Compute profiles (needs GPU counter permissions)
sudo ./bench/profile.shRun a single side by hand (the bench/Makefile auto-detects the arch; the explicit
-arch=sm_86 below is just the 3090 override for a one-off nvcc invocation —
change it to your card, or use make to let it detect):
nvcc -O3 -std=c++20 -arch=sm_86 -lcublas -o /tmp/mm kernels/matmul/cuda/matmul.cu && /tmp/mm --header
.venv/bin/mojo run kernels/matmul/mojo/matmul.mojo- reduction: grid-stride
float4-vectorized load, warp-shuffle (__shfl_down/warp.sum) block reduction. Memory-bound; targets peak bandwidth. Referenced againstcub::DeviceReduce::Sum. The two sides finish the reduction slightly differently — CUDA does oneatomicAddper block into a single accumulator; Mojo is two-pass (each block writes a partial, then a single block sums the partials). At 256M both are limited by thefloat4load traffic, not the block-finalization step, so the regime stays memory-bound and the two land within ~6 GB/s of each other and of CUB (see docs/portability.md). - softmax: one block per row, single-pass online softmax (fused max+sum) with
a warp-shuffle reduction over
(max, sum)pairs, vs a naive three-pass baseline. Referenced againsttorch.softmax(optional). - matmul: 128×128 block tile,
BK=8, an 8×8 register tile per thread (256 threads/block), shared-memory staging,float4loads. These tile dimensions are the NVIDIA config; on the Mojo side they are a device-selected comptime config, not hardcoded (see docs/portability.md). Compared against a simple 32×32 tiled kernel and against cuBLAS.
Details and the per-kernel optimization story are in each kernels/*/README.md.


