What's actually in a pretraining dataset
Most explanations of pretraining skip straight to “the model reads a huge amount of text.” That's true, and it hides the part that actually determines how good the model turns out: what “a huge amount of text” means once you go look at it. It's not a curated library. It's a scrape of the public internet, filtered code repositories, digitized books, and academic papers, mixed together in specific proportions that someone chose on purpose. Change the mix and you change the model, even with the exact same architecture and the exact same number of tokens.
Open a random page from an uncleaned web crawl and it looks nothing like the fluent prose you'd expect a language model to learn from. A lot of it is navigation menus repeated on every page of a site, cookie consent banners, “Subscribe to our newsletter” boilerplate, auto-generated product listings that are 90% identical to the listing next to them, and pages that are technically English but are really just keyword-stuffed SEO spam with no real sentence structure underneath. Some of it isn't language at all: minified JavaScript embedded in a page, base64-encoded images, table-of-contents pages that are nothing but links. None of that teaches a model to write or reason well, and a large chunk of the raw crawl is exactly that.
The dataset is not an input you feed into the real work of designing an architecture. Past a certain point of scale, the dataset is the work. Two teams training the identical transformer architecture on differently filtered data end up with models that behave differently in ways no amount of tuning the attention mechanism will fix.
So real pretraining corpora are built, not just collected. Common Crawl, a continuously updated dump of the public web running into hundreds of billions of pages, is the base layer for most of them. On top of that, teams add filtered GitHub repositories for code (with license and quality filters applied, not every repo makes the cut), books from digitized libraries, Wikipedia, and academic papers from sources like arXiv and PubMed. Each of those sources gets its own cleaning pipeline, because the junk in a code repo (auto-generated boilerplate, vendored dependencies, minified bundles) looks nothing like the junk in a web crawl.
The end result, after all that filtering, is measured in trillions of tokens, not gigabytes. A modern frontier model's training set commonly runs to ten trillion tokens or more. That number, and what's actually inside it, matters as much as anything about the model itself, which is exactly what the next section starts putting precise units on.
Batches, tokens, and steps: the units a training run is measured in
The Loss and Backpropagation article walked through one training step: feed a batch of examples through the model, compute the loss, run backpropagation, update the weights. That description is correct and it's also, at the scale of a real pretraining run, a single tick of a clock that runs for weeks. The question this section answers is what that tick actually contains, and how many times it ticks.
A batchis a fixed number of token sequences processed together in one forward and backward pass, exactly the “batch of examples” from that earlier article, just at production scale. Each sequence has a fixed sequence length, the number of tokens the model looks at in one shot, commonly somewhere between 2,048 and 32,000 for current models. The batch size is how many of those sequences get processed in parallel before a single weight update happens. Multiply the two together and you get the number of tokens one step actually touches.
Tokens processed per step, and across a full run
1batch_size = 1024 # sequences processed per step2seq_length = 4096 # tokens per sequence3num_steps = 500_000 # total weight updates over the whole run45tokens_per_step = batch_size * seq_length6total_tokens = tokens_per_step * num_steps78print(tokens_per_step) # 4,194,304 tokens in this one step9print(total_tokens) # 2,097,152,000,000 tokens over the full run
Read that last number out loud: roughly two trillion tokens, seen once each, across the whole run. That's the arithmetic behind every “trained on N trillion tokens” claim you'll see in a model card: it's just batch size times sequence length times number of steps, nothing more exotic than that. Push any one of those three numbers up and the total goes up too, which is exactly why they're the numbers labs report when they talk about a run.
Rule: bigger batch size means a more accurate gradient estimate per step (you're averaging the loss over more examples) but each step costs proportionally more compute. Labs pick batch size based on how much they can fit across their GPUs at once, not as a free knob, doubling it doesn't make training free, it just changes how the same total compute gets divided into steps.
One more distinction worth being precise about: “seeing a token” doesn't mean the model memorizes it. It means that token contributed one next-token-prediction example to one gradient update, alongside millions of others in the same step. Most tokens in a trillion-token run are seen exactly once, maybe twice if the dataset gets a second pass. There usually isn't time or need for more than that, which is part of why the raw size and quality of the dataset matters so much, covered later in this article.
Compute budgets: what a training run costs in practice
“We trained it on a big dataset” makes pretraining sound like a data problem. It's also, unavoidably, a hardware and money problem. Every one of those training steps from the previous section is a fixed amount of arithmetic, matrix multiplications for every layer, forward and backward, and that arithmetic has to physically happen on a GPU, one floating-point operation at a time. Compute budget is just: how many of those operations does this run need, and how fast can the hardware you have actually do them.
There's a rule of thumb worth memorizing, because it turns “train a big model” into an actual number: training a transformer costs roughly 6 FLOPs per parameter per token. Not 1, because a forward pass touches every parameter once but training also needs a backward pass to compute gradients, which costs roughly twice the forward pass, and there's a further pass to propagate those gradients back through every layer. Add it up and the constant that falls out is close to 6. So total training FLOPs is approximately:
The compute rule of thumb behind every model card's FLOPs number
total_flops ≈ 6 × num_parameters × total_tokens
Plug in real numbers and the scale of “frontier model” training stops being abstract. A 70-billion-parameter model trained on 15 trillion tokens needs roughly 6 × 70 × 109 × 15 × 1012, about 6.3 × 1024floating-point operations, total. A single high-end GPU today delivers on the order of a few hundred trillion floating-point operations per second in practice, not the marketing number on the spec sheet, real utilization is closer to 30 to 50 percent of peak once you account for communication between GPUs and the parts of the computation that just aren't matrix multiplication. Divide the first number by the second and you land on GPU-time measured in the hundreds of thousands of GPU-days. That is the actual reason frontier labs run training on thousands of GPUs for months rather than one machine for a weekend: the arithmetic itself demands it, no amount of clever engineering makes 1024 floating-point operations fast on one card.
The calculator below runs that same arithmetic live. Move the parameter count and token count sliders and watch total FLOPs, the single-GPU-equivalent time, and the wall-clock time at a chosen GPU count all update together. Try dragging token count up while holding parameters fixed, that's literally what “train longer on more data” means in terms of cost, and it isn't free just because the model itself didn't get bigger.
wall-clock training time, scaled against a 24-month bar
Assumes ~400 TFLOPS/sec effective per GPU, roughly 40% of one H100's peak bf16 throughput, a realistic model FLOPs utilization for a well-tuned run. Real numbers vary with interconnect, precision, and framework overhead.
In practice:the 400 TFLOPS/GPU figure in the calculator is an assumption, not a universal constant, real achieved throughput depends on GPU generation, interconnect speed, numeric precision, and how well the training code overlaps computation with communication. Labs report a metric called Model FLOPs Utilization, the ratio of achieved throughput to a GPU's theoretical peak, precisely because that gap is large and worth tracking. A well-optimized run at 50% MFU finishes in roughly half the wall-clock time of the same run at 25% MFU, on identical hardware.
Deduplication, filtering, and why more data isn't automatically better
The obvious assumption is that more tokens always means a better model, since the compute rule of thumb from the previous section has total_tokens sitting right there as a multiplier. Feed it more, get more compute credit, get a better model. That assumption breaks down once you notice what a web crawl actually contains: the same terms-of-service boilerplate copy-pasted across ten million different sites, the same press release syndicated to five hundred news aggregators, the same Stack Overflow answer scraped by a dozen different tutorial sites. None of those repeats are new information. They're the same tokens, counted multiple times.
Train on that without deduplicating and the model doesn't just waste compute re-reading the same sentence. It actively learns the wrong lesson: a phrase that appears ten thousand times across near-identical pages looks, statistically, ten thousand times more important than it actually is. The model starts assigning that boilerplate a higher probability than it deserves, and in the worst cases it memorizes long exact stretches of it verbatim, the same way a student who reads one over-repeated practice question ten times can recite the answer without having learned the underlying method. That's overfitting to duplication, and it looks like good performance on the training data while actively hurting how well the model generalizes to text it hasn't seen.
Rule: a duplicate isn't free extra training signal, it's a vote for the model to weight that content more heavily than it should. Deduplication isn't a nice-to-have cleanup pass, it's what keeps the token count in the compute formula honest.
So real pipelines run several layers of cleanup before a token ever reaches a training step. Exact and near-duplicate detection removes pages that are identical or a few words apart, usually with hashing techniques that can compare billions of documents against each other without doing a full pairwise comparison. Quality filtering scores documents on things like sentence structure, vocabulary diversity, and the ratio of real prose to boilerplate, and drops the ones that look like link farms or keyword-stuffed spam rather than actual writing. And data mixing decides the proportion of each source in the final blend by hand, code and books get deliberately upweighted far beyond their raw share of the crawl, because a page of clean Python or a well-edited book chapter teaches the model more per token than an average scraped web page does.
Put together, this is why two labs training the same architecture on the same total token count can end up with meaningfully different models, and why the field has largely stopped chasing raw token count as the headline number. Past a certain scale, the marginal token from a sloppy crawl is worth less than the marginal token from a carefully filtered and deduplicated one, and no amount of extra compute fixes that. Article 14 (Scaling Laws) picks this back up with the formal relationship between data, model size, and compute, this section is just the operational version: what the filtering pipeline is actually doing to the dataset before any of that math applies.