Inference Arena

View on GitHub

Inference Arena runs the same training and inference workload through every supported ML framework on every available platform, then publishes the results side by side. Each tab below covers one model — pick your model, filter to the frameworks you care about, and compare. Lower numbers are better; bold marks the best matching framework on each platform.

HuggingFaceTB/SmolLM2-135M — 134.5M parameter decoder-only language model.

Benchmark config: seq_len=128, input=[0,1,…,127]. The full forward is a 128-token prefill. The minimal measurement is a stateless one-token forward without a KV cache; it is not decode latency. The historical table below predates the current timing and precision metadata; regenerate it from one pinned revision before using it in a publication.

Platform Framework Compile (s) Prefill (ms) Stateless 1-token (ms) Training (ms) Loss
Intel Xeon @ 2.10GHz PyTorch 2.11.0+cu130 (CPU) 135.63 188 18 486 10.98
  ONNX Runtime 1.24.4 (CPU) 65.50 118 20 10.98
  JAX 0.9.2 (CPU) 6.79 194 31 2107 10.98
  Candle (CPU) 0.31 453 61 11.11
  Luminal (CPU) 3.37 17006 14459 10.81
  Burn (wgpu/Lavapipe) 0.00 2369 320 5700 11.73
  Meganeura (Vulkan/Lavapipe) 7.29 3933 852 3651 10.99
  llama.cpp (CPU) 0.10 221 24 10.98
AMD Radeon 890M Graphics PyTorch 2.10.0 (ROCm 7.2.53210) 51.07 64 27 119 8.35
  Burn (wgpu/vulkan) 0.00 182 31 206 11.55
  Inferi  
  Meganeura (Vulkan) 0.59 26 9.1 92 8.64
  ONNX Runtime  
Apple M3 PyTorch 2.11.0 (MPS) 0.00 356 71 699 8.35
  MLX (MLX) 0.00 97 253 8.64
  Candle (Metal) 0.02 22 2.8 10.80
  Burn (wgpu/metal) 0.00 873 39 905 11.59
  Inferi  
  Luminal  
  Meganeura (Metal) 1.50 201 9.1 464 8.65
  GGML (Metal) 0.38 49 11 8.69
  JAX (METAL) 3.13 47 21 253 5.79
NVIDIA GeForce RTX 5080 PyTorch 2.13.0+cu130 (CUDA 13.0) 20.53 4.4 3.0 6.4 6.01
  Candle (CUDA) 0.05 35 2.3 10.80
  Burn (wgpu) 0.00 105 19 112 11.75
  Meganeura (Vulkan) 0.62 8.5 2.2 22 5.79
  GGML (Vulkan) 0.18 29 1.5 8.69
  ONNX Runtime 1.27.0 (CUDAExecutionProvider) 20.62 5.3 3.2 6.01
  MAX (GPU) 0.92 4.5 0.0 10.80
  JAX 0.11.0 (GPU) 21.02 3.8 1.4 11 5.79
NVIDIA GeForce RTX 3050 (Windows) PyTorch 2.11.0+cu128 (CUDA 12.8) 0.00 11 5.1 51 8.35
  Burn (wgpu/vulkan) 0.00 125 28 138 11.76
  Inferi  
  Meganeura (Vulkan/DX12) 1.40 13 3.6 58 8.63
  GGML (CUDA) 0.31 132 5.9 8.69
  ONNX Runtime (CUDAExecutionProvider) 39.92 18 14 6.01
  JAX  
Intel(R) Graphics (RPL-U) PyTorch 2.11.0+xpu (CPU) 0.00 541 126 1130 8.35
  Candle (CPU) 0.41 524 76 12.16
  Burn (wgpu/vulkan) 0.00 604 83 1437 11.79
  Inferi (Vulkan) 1.06 25769 9.1 15.16
  Luminal (CPU) 3.57 15551 15473 10.81
  Meganeura (Vulkan) 1.74 172 52 700 8.64
  GGML (CPU) 0.11 433 33 8.69
  ONNX Runtime (CPUExecutionProvider) 66.86 381 64 6.01
  MAX  
  JAX (CPU) 9.97 589 176 1412 5.79
AMD Radeon RX 7900 XT PyTorch 2.10.0+rocm7.1 (ROCm 7.1.25424) 73.03 10 6.8 23 8.35
  Burn (wgpu/vulkan) 0.00 172 25 181 11.76
  Inferi  
  Meganeura (Vulkan) 0.99 7.3 2.1 21 8.64
  GGML (ROCm) 0.11 259 3.4 8.69
  MAX (GPU) 2.13 3.5 0.1 10.80

Historical correctness note: the old validator sampled the first 16 flattened logits, all from sequence position 0, and could accept a backend when only its loss was close. Those PASS/CLOSE labels are not publication-grade. The current harness samples across the complete output and requires both output and loss agreement, plus gradient-norm agreement when available.

Caveats: - PyTorch and Meganeura load real model weights and run the full architecture — their outputs match.

Run it yourself: git clone https://github.com/kvark/inferena && cd inferena && ./run.sh -m SmolLM2-135M

lerobot/smolvla_base — a matched 99,848,592-parameter SmolVLA-style action expert workload for robotics.

Benchmark config: batch=1, chunk_size=50, vlm_seq_len=16, float32, deterministic matched weights, MSE loss.

The table below predates the current timing and validation metadata. Its timings and historical CLOSE label are exploratory; regenerate the table from one revision before using it in a publication.

Platform Framework Compile (s) Inference (ms) Latency (ms) Training (ms) Loss
Intel Xeon @ 2.10GHz PyTorch 2.11.0+cu130 (CPU) 51.63 40 11 116 0.00
  Meganeura (Vulkan/Lavapipe) 2.75 696 3850 0.01
  ONNX Runtime (CPU)  
  JAX (CPU)  
  Candle (CPU)  
  Burn (wgpu)  
  Luminal (CPU)  
AMD Radeon 890M Graphics PyTorch 2.10.0 (ROCm 7.2.53210) 19.72 27 14 49 0.00
  Candle  
  Burn  
  Inferi  
  Luminal  
  Meganeura (Vulkan) 0.12 15 6.7 47 0.00
  GGML  
  ONNX Runtime (MIGraphXExecutionProvider) 19.38 10 0.00
Apple M3 PyTorch 2.11.0 (MPS) 0.00 173 9.1 117 0.00
  MLX (MLX) 0.00 13 24 0.00
  Candle  
  Burn  
  Inferi  
  Luminal  
  Meganeura (Metal) 0.12 34 6.4 170 0.00
  GGML  
  ONNX Runtime (CoreMLExecutionProvider) 7.96 86 0.00
  JAX (METAL) 1.17 15 147 0.00
NVIDIA GeForce RTX 5080 PyTorch 2.13.0+cu130 (CUDA 13.0) 9.30 2.0 1.1 2.6 0.00
  Candle  
  Burn  
  Meganeura (Vulkan) 0.24 3.3 1.5 8.2 0.00
  GGML  
  ONNX Runtime 1.27.0 (CUDAExecutionProvider) 2.64 1.5 0.00
  MAX (GPU) 1.12 25 0.00
  JAX 0.11.0 (GPU) 15.82 2.4 3.9 0.00
NVIDIA GeForce RTX 3050 (Windows) PyTorch 2.11.0+cu128 (CUDA 12.8) 0.00 4.5 3.5 22 0.00
  Candle  
  Burn  
  Inferi  
  Luminal  
  Meganeura (Vulkan/DX12) 0.70 4.9 2.9 23 0.00
  GGML  
  ONNX Runtime (CUDAExecutionProvider) 4.14 4.9 0.00
  JAX  
Intel(R) Graphics (RPL-U) PyTorch 2.11.0+xpu (CPU) 0.00 183 72 388 0.00
  Candle  
  Burn  
  Inferi  
  Luminal  
  Meganeura (Vulkan) 0.39 73 40 222 0.00
  GGML  
  ONNX Runtime (CPUExecutionProvider) 10.33 86 0.00
  MAX  
  JAX (CPU) 3.86 162 471 0.00
AMD Radeon RX 7900 XT PyTorch 2.10.0+rocm7.1 (ROCm 7.1.25424) 9.23 4.8 4.1 8.1 0.00
  Candle  
  Burn  
  Inferi  
  Luminal  
  Meganeura (Vulkan) 0.81 5.1 1.5 9.9 0.00
  GGML  
  MAX (GPU) 2.20 15 0.00

Historical correctness: PyTorch vs Meganeura was reported as CLOSE (loss diff 1e-5, max error 4.6e-3) under the old validator. The current harness rechecks output samples distributed across the tensor, loss, total gradient norm, and per-parameter gradient norms.

Caveats: - PyTorch and Meganeura implement the full action expert architecture and should produce matching outputs.

Run it yourself: git clone https://github.com/kvark/inferena && cd inferena && ./run.sh -m SmolVLA

The CLI retains the historical StableDiffusion workload name. The matched PyTorch/Meganeura workload is now a scaled, conditioned latent-diffusion U-Net, not the earlier generic convolutional U-Net and not the complete Stable Diffusion 1.5 pipeline.

It retains the denoiser features that materially change the compiler workload: timestep conditioning in every residual block, spatial self-attention, 77-token text cross-attention, transformer feed-forward blocks, convolutional down/up paths, GroupNorm, and U-Net skip connections.

The previous table measured the superseded 5.29M-parameter convolution-only workload and has been removed to prevent accidental comparison. This empty table preserves the validated results schema and insertion point for the next frozen run.

| Platform | Framework | Compile (s) | Inference (ms) | Latency (ms) | Training (ms) | Loss | |———-|———–|:———–:|:————–:|:————:|:————-:|:—-:|

Freeze one Meganeura revision and rerun both practical-default and strict configurations on every platform before publishing new results:

./run.sh -m StableDiffusion -f pytorch,meganeura
./run.sh -m StableDiffusion -f pytorch,meganeura --strict

During development, strict-f32 cross-engine validation on an NVIDIA RTX 5080 reached approximately 1.2e-5 forward relative L2 error and 8.1e-5 per-parameter gradient-vector relative L2 error. These are correctness smoke results from a dirty development revision, not frozen performance results.

The legacy Candle runner uses a much larger SD 1.5-like U-Net and is not comparable to this matched workload.

Run it yourself: git clone https://github.com/kvark/inferena && cd inferena && ./run.sh -m StableDiffusion

Classic convolutional neural network for image classification. The matched benchmark graph has 25,530,472 trainable parameters.

Benchmark config: batch=4, 3×224×224, float32, deterministic matched weights, cross-entropy loss.

The table below predates the current timing and validation metadata. Regenerate it before using the numbers or the historical correctness label in a publication.

Platform Framework Compile (s) Inference (ms) Latency (ms) Training (ms) Loss
Intel Xeon @ 2.10GHz PyTorch 2.11.0+cu130 (CPU) 60.61 141 40 284 10.10
  ONNX Runtime 1.24.4 (CPU) 0.28 76 18 10.37
  Candle (CPU) 0.00 782 311 6.91
  Meganeura (Vulkan/Lavapipe) 0.98 3906 1192
  Burn (wgpu)  
  JAX (CPU)  
AMD Radeon 890M Graphics PyTorch 2.10.0 (ROCm 7.2.53210) 36.46 48 16 97 6.92
  Burn  
  Inferi  
  Luminal  
  Meganeura (Vulkan) 0.31 58 19 125 6.92
  GGML  
  ONNX Runtime (MIGraphXExecutionProvider) 3.17 29 9.3 6.92
Apple M3 PyTorch 2.11.0 (MPS) 0.00 166 21 274 6.92
  MLX  
  Candle (Metal) 0.01 16 5.3 6.91
  Burn  
  Inferi  
  Luminal  
  Meganeura (Metal) 0.32 63 23 965 6.92
  GGML  
  ONNX Runtime (CoreMLExecutionProvider) 4.87 6.5 2.1 6.92
  JAX (METAL) 0.90 139 9.4 409 6.92
NVIDIA GeForce RTX 5080 PyTorch 2.13.0+cu130 (CUDA 13.0) 9.15 2.5 1.6 4.3 6.92
  Candle (CUDA) 0.17 63 2.3 6.92
  Burn  
  Meganeura (Vulkan) 0.24 5.1 3.4 26 6.92
  GGML  
  ONNX Runtime 1.27.0 (CUDAExecutionProvider) 1.54 2.4 1.3 6.92
  MAX  
  JAX 0.11.0 (GPU) 2.65 2.0 1.3 5.3 6.92
NVIDIA GeForce RTX 3050 (Windows) PyTorch 2.11.0+cu128 (CUDA 12.8) 0.00 12 4.1 36 6.92
  Burn  
  Inferi  
  Luminal  
  Meganeura (Vulkan/DX12) 1.04 19 6.5 52 6.92
  GGML  
  ONNX Runtime (CUDAExecutionProvider) 3.20 13 4.5 6.92
  JAX  
Intel(R) Graphics (RPL-U) PyTorch 2.11.0+xpu (CPU) 0.00 511 127 1048 6.92
  Candle (CPU) 0.36 1187 368 6.92
  Burn  
  Inferi  
  Luminal  
  Meganeura (Vulkan) 0.77 225 73 1049 6.92
  GGML  
  ONNX Runtime (CPUExecutionProvider) 5.75 204 53 6.92
  MAX  
  JAX (CPU) 6.99 436 176 2813 6.92
AMD Radeon RX 7900 XT PyTorch 2.10.0+rocm7.1 (ROCm 7.1.25424) 17.57 7.1 3.8 17 6.92
  Burn  
  Inferi  
  Luminal  
  Meganeura (Vulkan) 0.92 6.5 3.4 26 6.92
  GGML  
  MAX  

Historical correctness: the old PyTorch/ONNX Runtime CLOSE label predates the matched folded-BatchNorm graph and the audited validator.

Caveats: - The benchmark intentionally does not measure training-mode BatchNorm statistics or their backward pass. Both engines run the same inference-folded representation: identity scale plus a trainable channel bias.

Run it yourself: git clone https://github.com/kvark/inferena && cd inferena && ./run.sh -m ResNet-50

openai/whisper-tiny supplies the configuration, but the audited workload is the four-layer encoder only: 8,208,384 total parameters, of which 7,632,384 are trainable. It is not a full speech-to-text pipeline.

Benchmark config: batch=1, synthetic 30-second mel spectrogram (80×3000), float32, deterministic matched weights, and mean-square encoder-output loss. The positional embedding is frozen in both implementations.

The table below predates the current timing and validation metadata and includes runners that measured older or different Whisper workloads. Regenerate it before using the numbers in a publication.

Platform Framework Compile (s) Inference (ms) Latency (ms) Training (ms) Loss
Intel Xeon @ 2.10GHz PyTorch 2.11.0+cu130 (CPU) 39.88 150 371 11.80
  ONNX Runtime 1.24.4 (CPU) 0.84 212 11.80
  Candle (CPU) 0.01 616 0.00
  Meganeura (Vulkan/Lavapipe) 7.84 53467 0.01
  Burn (wgpu)  
  JAX (CPU)  
AMD Radeon 890M Graphics PyTorch 2.10.0 (ROCm 7.2.53210) 17.09 79 63 220 0.00
  Burn  
  Inferi  
  Luminal  
  Meganeura (Vulkan) 0.20 34 33 101 0.01
  GGML  
  ONNX Runtime (MIGraphXExecutionProvider) 23.58 32 0.01
Apple M3 PyTorch 2.11.0 (MPS) 0.00 318 41 127 0.00
  MLX  
  Candle (Metal) 0.01 22 0.00
  Burn  
  Inferi  
  Luminal  
  Meganeura (Metal) 0.15 406 415 1062 0.01
  GGML  
  ONNX Runtime (CoreMLExecutionProvider) 7.93 440 0.01
  JAX (METAL) 2.17 128 315 445 0.01
NVIDIA GeForce RTX 5080 PyTorch 2.13.0+cu130 (CUDA 13.0) 4.14 2.3 2.4 9.7 0.00
  Candle (CUDA) 0.02 45 0.00
  Burn  
  Meganeura (Vulkan) 0.21 3.4 3.4 12 0.01
  GGML (faster-whisper (CTranslate2, CUDA)) 7.98 15 15 0.00
  ONNX Runtime 1.27.0 (CUDAExecutionProvider) 2.04 3.6 0.01
  MAX  
  JAX 0.11.0 (GPU) 17.39 2.1 1.6 5.6 0.01
NVIDIA GeForce RTX 3050 (Windows) PyTorch 2.11.0+cu128 (CUDA 12.8) 0.00 13 13 43 0.00
  Burn  
  Inferi  
  Luminal  
  Meganeura (Vulkan/DX12) 0.66 19 19 43 0.01
  GGML (faster-whisper (CTranslate2, CUDA)) 7.00 40 45 0.00
  ONNX Runtime (CUDAExecutionProvider) 4.82 20 0.01
  JAX  
Intel(R) Graphics (RPL-U) PyTorch 2.11.0+xpu (CPU) 0.00 477 420 899 0.00
  Candle (CPU) 0.02 795 0.00
  Burn  
  Inferi  
  Luminal  
  Meganeura (Vulkan) 0.39 467 466 1594 0.01
  GGML (faster-whisper (CTranslate2, CPU)) 14.76 1036 1104 0.00
  ONNX Runtime (CPUExecutionProvider) 6.18 333 0.01
  MAX  
  JAX (CPU) 5.59 717 686 2681 0.01
AMD Radeon RX 7900 XT PyTorch 2.10.0+rocm7.1 (ROCm 7.1.25424) 5.38 12 6.5 44 0.00
  Burn  
  Inferi  
  Luminal  
  Meganeura (Vulkan) 0.82 4.8 4.8 21 0.01
  MAX  

Historical correctness: the old table’s PyTorch/ONNX Runtime PASS applies to the prior workload and validator, not the audited encoder comparison.

Caveats: - Uses the Whisper-tiny encoder shape (4 layers, d=384), not its decoder

Run it yourself: git clone https://github.com/kvark/inferena && cd inferena && ./run.sh -m Whisper-tiny

Legend: Bold = best among matching frameworks Struck through = different / simplified model = not supported Framework names link to tested revision