Biplov GautamLaya Studio started as a Mac app. It fine-tunes Laya — the open System One model on a ModernBERT-large encoder — with Apple's MLX, which is fast on Apple silicon and nowhere else. To make it run on Windows and Linux, with NVIDIA, AMD or Intel GPUs or just a CPU, we ported the trainer to PyTorch.
A port is only worth having if it trains the same model. So before claiming it, we measured it: the same fine-tune, run once on MLX and once on PyTorch through Metal (MPS), on the same 16 GB MacBook Air.
The setup
| Machine | MacBook Air, Apple M4 (10-core CPU, 8-core GPU), 16 GB unified memory |
| Base model | aac6fef/laya-mlx — Laya English, 421M parameters |
| Data | Emotion, 6 labels: 1,079 training, 121 validation and 600 test rows |
| Question | One choice question: which of sadness, joy, love, anger, fear or surprise the writer expresses |
| Method | LoRA rank 16 (alpha 32) on every encoder layer's attention and MLP projections, decision head trained in full: 33.4M of 428.5M parameters |
| Objective | Proper scoring rule (log + spherical score), as Laya was trained |
| Schedule | 2 epochs, batch 8, gradient accumulation 2 = 136 updates; learning rate 2e-4 for the adapters and 1e-4 for the head; 6% warm-up then cosine decay |
| Precision | Frozen encoder in bfloat16, trainable weights in float32 |
| Software | MLX 0.32.2 with laya-mlx 0.2.0; PyTorch 2.14.0 with laya 0.3.5 and transformers 5.17.0 |
Both runs used the same seed, the same option shuffling and the same batches. Early stopping was off so both trained for exactly two epochs. Each run first scores the base model on the test split, then trains, refits calibration on the validation split, and scores the fine-tuned model on the same test rows.
The first result was wrong
The first PyTorch run reached 75.3% test accuracy. MLX reached 82.7%. Its validation loss after two epochs was 0.78 against MLX's 0.49. Same data, same recipe, same seed: that is not noise, it is a difference in the training.
It was the optimizer. MLX's AdamW defaults to no bias correction. Adam keeps
running averages of the gradient (m) and its square (v), both starting at zero.
PyTorch divides them by 1 − β^t to undo that start; MLX, by default, does not.
On the first step, with β₁ = 0.9 and β₂ = 0.999:
- PyTorch: the update is
lr × 1.0in each coordinate. - MLX:
m = 0.1·g,v = 0.001·g², so the update islr × 0.1 / √0.001 ≈ lr × 3.16.
MLX's early steps are about three times larger, shrinking toward PyTorch's as v fills up. In a long pre-training run that washes out. In a fine-tune of 136 updates it is most of the run.
There was a second, smaller difference: PyTorch's TransformerEncoderLayer, which
the decision head is built from, also applies dropout to the attention weights.
The MLX head does not.
The PyTorch trainer now uses MLX's update rule — decoupled weight decay, then
lr × m / (√v + ε) with no correction — and no attention-weight dropout. Then we
ran it again.
Results
| MLX | PyTorch (Metal) | |
|---|---|---|
| Validation loss, epoch 1 → 2 | 0.589 → 0.488 | 0.600 → 0.461 |
| Test accuracy (base model: 47.5%) | 82.7% | 83.3% |
| Test rows fixed / broken vs the base | 228 / 17 | 230 / 15 |
| Calibration error (ECE), before → after refit | 0.085 → 0.067 | 0.063 → 0.051 |
| Training time, 2 epochs | 4 min 45 s | 7 min 21 s |
| Training throughput | 7.8 decisions/s | 5.0 decisions/s |
| Peak memory | 5.4 GB | 6.3 GB |
| Whole run: baseline, training, test, timing | 6 min 5 s | 11 min 35 s |
Both fine-tunes lift test accuracy from 47.5% to about 83%, a change an exact McNemar test puts at p < 10⁻⁴⁷ for each. The 0.6-point gap between them is one run each and well inside what a different random stream produces: the two frameworks draw different dropout masks and different initial adapter weights from the same seed.
MLX trains about 1.5× faster on this Mac. The peak-memory figures come from each framework's own counter (MLX's peak allocation, PyTorch's Metal driver allocation), so read them as a rough comparison.
Do the two checkpoints agree?
Both trainers write the same file format: model.safetensors in the original
PyTorch parameter names, the tokenizer, and the refitted temperatures. So each
checkpoint can be loaded by the other runtime. We loaded both in MLX, in
PyTorch on Metal and in PyTorch on the CPU, and asked each the same test
questions.
| Checkpoint trained on | Same answer as MLX — PyTorch Metal | Same answer as MLX — PyTorch CPU | Largest probability difference |
|---|---|---|---|
| MLX | 200 of 200 | 60 of 60 | 0.008 |
| PyTorch | 200 of 200 | 60 of 60 | 0.002 |
Every answer is the same, and the probabilities agree to the second or third decimal place. A model trained on a Mac runs unchanged on a Linux server with an NVIDIA card, and the other way round.
How fast does it answer?
Median time for one six-option decision, including tokenization, one request at a time:
| Runtime | Median latency |
|---|---|
| MLX | 54–60 ms |
| PyTorch, Metal | 78–98 ms |
| PyTorch, CPU | about 120 ms |
On a Mac, MLX is the faster runtime for both training and inference, and Laya Studio uses it there by default. PyTorch's value is everywhere else: it is how the same recipe runs on CUDA, ROCm, Intel XPU or a plain CPU.
What this does not show
One dataset, one machine, one run per backend. We have not yet measured the
PyTorch path on an NVIDIA, AMD or Intel GPU; those numbers will get their own
post, from real machines. And "the same recipe" now means the same optimizer rule
as MLX, which differs from PyTorch's stock AdamW — worth knowing if you compare
these runs against a PyTorch fine-tune of your own.
Try it
pip install systemonemodels
systemone run studio
systemone run studio reads your machine, installs the right stack — MLX on
Apple silicon, the PyTorch build for your GPU elsewhere — signs you in to
systemonemodels.tech so a finished run can be published in one click, and
opens Laya Studio. The emotion
dataset is one of its built-in examples. To force PyTorch on a Mac, run the
studio from a checkout with uv sync --extra torch and
LAYASTUDIO_BACKEND=torch uv run layastudio.
Every System One model, with sizes, licences and reported benchmarks, is at System One models.