Py vs R implementations of torch: speed benchmarking

[Artur-man]

Hello there,

I am new to torch and mostly program R. I am interested in the performance differences between this implementation and pytorch.

I did a quick benchmarking with the bench/nn-small.R here: https://github.com/Artur-man/py_r_torch_benchmarking. It seems local GPU accelerates quite well, but there is a small performance gap between languages. I would really love to get your opinion on this, is it expected ?

(py_r_torch_benchmarking) LANDTHALER57:py_r_torch_benchmarking amanuky$ python benchmark_device_simple/benchmark_device_simple.py 
{'load': 0.0013628005981445312, 'setup': 0.0018796920776367188, 'mps': 1.6316609382629395, 'cpu': 6.573911190032959}
{'load': 0.0011260509490966797, 'setup': 0.0017371177673339844, 'mps': 1.5132861137390137, 'cpu': 6.622015953063965}
{'load': 0.0011050701141357422, 'setup': 0.002045869827270508, 'mps': 1.5158348083496094, 'cpu': 6.62052583694458}
{'load': 0.0010769367218017578, 'setup': 0.0025539398193359375, 'mps': 1.518923044204712, 'cpu': 6.630447149276733}
{'load': 0.001125335693359375, 'setup': 0.002821207046508789, 'mps': 1.5086002349853516, 'cpu': 6.6927337646484375}
(py_r_torch_benchmarking) LANDTHALER57:py_r_torch_benchmarking amanuky$ Rscript benchmark_device_simple/benchmark_device_simple.R
  mps   cpu  load setup 
 1.77  9.83  0.00  0.07 
  mps   cpu  load setup 
 1.68  9.74  0.00  0.08 
  mps   cpu  load setup 
 1.78  9.73  0.00  0.07 
  mps   cpu  load setup 
 1.76  9.61  0.00  0.07 
  mps   cpu  load setup 
 1.77  9.80  0.00  0.07 
It seems there is native mps/cuda support in torch without python thats nice .... and it accelerates the model as much as the python version does. It is a bit small though but not surprising.

Note: I also left a short forum entry here, would love to know more about dev's opinion on mlverse/torch vs PyTorch:
https://forum.posit.co/t/torch-vs-pytorch/210730

[dfalbel]

There are a few issues discussing the performance of torch for R compared to PyTorch with some detailed explanations:

• Weird runtime behaviour #970
• Runtime #268
• Slow train loop compared to python #694

In summary, the overhead comes from:

• R↔C++ boundary crossing cost — every call to a torch function goes through R's .Call interface, which is heavier than Python's CPython/C bindings.
• R6 method dispatch — model$forward(), opt$zero_grad(), opt$step(), etc. go through R6 class dispatch, which is slower than Python's attribute lookup.
• Loop overhead — R's interpreted loop and dataloader iteration (especially with coro) adds per-iteration cost that Python's loops don't.
• The gap narrows with scale — on MPS the gap is smaller (~15%) because GPU compute dominates; on CPU the gap is larger (~47%) because the per-call overhead is a bigger fraction of total time.
• Frequency of R's GC calling

Do you have a more specific question?

[waynelapierre]

Thank you for the detailed explanation. It seems the performance gap stems from R's inherent architecture and may be unavoidable. Not sure if there is a plan to tackle them.

[t-kalinowski]

This is ultimatly the tradeoff: eager computation comes with interperter overhead. It's the same tradeoff in R and Python. Comparing microbenchmarks of primitive language operations like class attribute access or method dispatch is generally not very worthwhile, since that's not where the bulk of the time is spent in typical workflows anyway.

A more performance-oriented archectiture moves computation out of interperter and into a dedicated optimized runtime. E.g., this is the approach taken in https://github.com/r-xla.

[TroyHernandez]

I had a lot of these same questions. Going through the libtorch 2.8 bump with @dfalbel (disclaimer: Claude Code did most of the typing) gave me real appreciation for what he built. Doing all of that pre-LLM-agent is wild.

tinytorch is Linux-only so it won't help with your mps numbers, but if you're curious what R torch perf looks like when you skip the GPU/CPU × Linux/Mac/Windows matrix and just wrap libtorch with Rcpp:

https://github.com/cornball-ai/tinytorch/blob/main/dev-notes/benchmarks.md

More context here: https://cornball.ai/posts/torchandfriends/

[dfalbel]

Very nice @TroyHernandez !

In principle, torch operations on CUDA are asynchronous until you ask for tensor data (via printing, converting or converting to an R array). This means, that we should be able to chain operations and pay the interpreter overhead at the same time CUDA computations are happening. Since in most programs the CUDA computation time would dominate, we shouldn't be worried about the overhead.

It happens though that since R is not aware that torch tensors are using GPU memory, it doesn't garbage collect them enough. So it was easy to end up with thousands of tensors allocated that were no longer in use. And the R garbage collector wouldn't care about them because they don't cause memory pressure in the R side - just on CUDA side.

To fix that, we need to call GC from backward(), or at least allow that to happen. This was a long standing bug in torch, that would cause long training loops to OOM at some point after a long training run. We have a whole heuristic to control how to GC from backward but it's quite tricky. It will probably happen with tinytorch too, every torch operation returns a new copy of a Tensor and if GC is not called at some point this will fill the GPU memory.

[TroyHernandez]

Thanks @dfalbel !
Definitely need to dig into this.
I clearly have much to learn from you about libtorch and C++!

[dfalbel]

I ended up spending some time on this. And (of course) with the help of Claude, ended up with a caching allocator strategy that makes R even faster than Python in this specific example on CPU.

You can see in this PR: #1447

It's quite interesting how it works, and there are many pieces involved making it really hard to state exactly what causes the slowdown. This specific benchmark is slow in R because of how R allocates memory.

R's very lazy at garbage collecting. It will essentially only cleanup memory when its initial pool is full and it needs to allocate more. The problem is that since torch tensor memory is not managed by R (except from a very tiny external pointer) this pool takes time to fill and we keep accumulating tensors holding a lot of memory. In Python this is different because with their reference counting, tensors are quickly disposed even if gc was not executed.

Since Python frees tensor memory pretty quickly, it can use happy paths of system level memory allocators like glibc malloc or Apple's libmalloc. Those happy paths keep a process level pool of memory that is not immediately freed to the OS, so if a new tensor needs memory, they can quickly serve memory without doing sys calls. R ends up not using those optimizations, and keeps needing all the sys calls.

A fix for this is to use a caching memory allocator, which is what we implemented in #1447. The idea is similar to the CUDA caching allocator (that's also used in PyTorch). Instead of freeing up memory to the OS immediately, we keep a pool of cached memory, that we only periodically give back to the OS. This allows us to allocate Tensors much faster, specially if they are of the same size of previously allocated tensors - which is quite likely the case in training loops - where we re-run the same allocations over and over.

The tradeoff is that we may have a larger peak memory usage, compared with Python - because we keep a pool of memory in process. But that's already the case anwyay unless you gc before every torch operation.

[Artur-man]

Thanks @dfalbel. We started testing on a feature extraction model (https://github.com/rpautrat/SuperPoint). It looks like the R implementation is faster in some occasions (even on CPU), but we need to test more before reporting here (or somewhere else).

Do you have a more specific question?

I do not really have a say, "specific" question really, but I am still puzzled with your (or posit's ) intentions with this implementation. Going back to this post: https://forum.posit.co/t/torch-vs-pytorch/210730.

PyTorch is extremely powerful and popular, so many would find ... interesting ... why an R implementation exists. Perhaps you can clarify that. My understanding is that you want to either:

• Provide a simple implementation to build models in R (which I respect, we should motivate more multi-language implementations) or
• Use R and Python implementations together to build multi-language workflows. An example to this would be to pretrain models in Python but expose weights to the R implementation to generate embeddings/response_variables live in R workspaces, thus integrate NN models with other R implementations, e.g. tidyverse.