I trained a 113M-parameter earthquake LLM from absolute scratch

I understand you have created this for teaching - to help other researchers use LLMs better.

But, in its current state of development, is it of any use in earthquake preparedness or immediate response to an earthquake?

Hi HN,

I am a geophysics researcher and PI making a transition into the ML/AI space. Over the past year, I noticed that while many students and researchers use LLMs, the actual pretraining lifecycle remains a black box to them. Most tutorials start with a clean Hugging Face dataset and end with a trainer.train() call. I wanted to demystify what actually happens in the trenches, so I built nanoGPT-Seis.

This isn't an attempt to build a SOTA foundation model. It is a teaching repository where every single step of the LLM pipeline is implemented and explained block by block, scaled down to run on a single workstation or two A30 GPUs.

I started by writing a concurrent BFS crawler to pull open-access seismology papers via Crossref and Unpaywall. The yak-shaving here was real. To avoid the classic O(N^2) deduplication nightmare, I implemented a MinHash LSH pipeline to cluster near-duplicates before doing exact similarity checks.

Instead of reusing the GPT-2 tokenizer, I wrote a byte-level BPE tokenizer from scratch with a 16k vocabulary, purely to understand how domain-specific compression behaves. For the architecture, I wanted it to mirror modern Llama-style decoders, so the repo includes from-scratch implementations of Grouped-Query Attention (GQA), Rotary Positional Embeddings (RoPE), SwiGLU, and an integration with FlashAttention-2.

One of the most interesting engineering challenges was figuring out the actual scaling behavior. I built a small IsoFLOP scaling-law harness. By implementing Maximal Update Parametrization (muP), I was able to tune the learning rate once on a tiny model and cleanly transfer it across a 90x compute sweep to find the compute-optimal frontier.

A fun empirical finding on data mixing: when I trained the model purely on my scraped research papers, it became highly fluent in "academic register" but generated repetitive gibberish when asked to write plain prose. Injecting a general-text mix (about 76% FineWeb-Edu and Wikipedia) completely restored its plain English fluency while preserving its domain sharpness. It was a very stark demonstration of the specialization versus fluency trade-off.

The repo has all the code, the exact VRAM mathematics for sizing batches before OOMing, and the reasoning behind each architectural choice. I would love to hear your thoughts, answer any questions about the math or the PyTorch engineering, or just chat about the transition from physical sciences to ML.