Machine learning
Sunspot Transformer
A transformer decoder written from scratch in PyTorch, with no pre-trained models and no transformer libraries, trained autoregressively on 275 years of solar observations.
What it does
- Given the last 128 months of sunspot activity, the model predicts the next month’s count, trained the way a GPT model is trained on text.
- 100,161 parameters: 64-dim embeddings, 4 attention heads, 2 layers, 256-dim feedforward, causal masking with residuals and LayerNorm.
- Mean Absolute Error of 16.4 sunspots on a held-out validation set spanning 1973–2026, on a 0–398 scale.
- Ablations over attention-head count and over dropout / weight-decay combinations, with the small baseline holding up against a 3.5× larger variant.
- Data: 3,329 monthly observations from SILSO, Royal Observatory of Belgium (1749 onwards).
Built with
- PyTorch
- NumPy
- Google Colab
- SILSO dataset