Pretraining HobbyLM from scratch

Derisking the full run

Before committing four H200s to the main run, we tested the pipeline on a smaller proxy model. Its training loss fell quickly, but broader evaluation moved in the wrong direction.

The source datasets were individually valid. The problem was their delivery. The model saw extended stretches from one source family before moving to another, so the batches did not represent the distribution we intended to train on over time. Falling loss showed that the model was learning the current batches. It said little about whether behavior was holding up across the wider corpus.

That result clarified what we needed from a proxy run. A small model could not forecast the final model's benchmark scores with much precision. It could test the machinery around the model while failures were still cheap: data loading, ordering, validation, checkpointing and recovery.

We corrected the delivery behavior and repeated the proxy experiment. General evaluation recovered enough to proceed with the larger pipeline. The implementation is internal. The public lesson is to inspect the sequence the model receives alongside the list of datasets in its configuration.

The proxy result also changed how we read training loss. We continued to use it as an optimization signal, but no longer treated a falling curve as sufficient evidence that the complete training system was improving.

Infrastructure gates

The first infrastructure tests failed before producing a training loss. GPU processes stayed alive, making the job appear active, but the run had not reached a useful step.

We isolated the launch path and retested it in stages. A short single-GPU run had to produce a loss and write a checkpoint. A multi-GPU run then had to demonstrate that the distributed configuration worked. The final sanity run used the complete configuration and real staged data.

Each test had explicit exit conditions. The process needed to train, validate, save a checkpoint and show the expected GPU activity. The preflight also checked available storage, experiment-tracking authentication and remote checkpoint backup. We tested recovery before starting the main job instead of assuming that a saved file would work after a failure.

These checks took time, but each was cheaper than the full run it protected. By launch, the code, data loader, validation path and checkpoint flow had already worked together under realistic conditions.

Data and model design

Data pool

The main corpus drew from four source families: FineWeb-Edu, DCLM, code and FineMath. Together they supplied educational web text, broader web text, program code and mathematical material. Annealing used the narrower Cosmopedia-v2 and FineMath-4plus source families.

All text passed through the GPT-2 byte-level BPE tokenizer configuration used by the released model. Before training, we checked that the loader could read the prepared data, that the expected sources were present and that samples decoded correctly. Validation data remained separate from training data.

We relied heavily on filtering and deduplication performed by the upstream dataset maintainers. We did not run one uniform project-level deduplication pass across every source. The phrase “approximately 100 billion tokens” therefore describes the amount consumed by the training schedule. It does not mean 100 billion unique or uniformly cleaned tokens.

Architecture

HobbyLM is a decoder-only sparse MoE transformer with 20 layers. Layer 0 has a dense feed-forward block. Layers 1 through 19 use MoE blocks.

Each MoE block has 64 routed experts and one shared expert. For every token, the router selects eight routed experts while the shared expert remains active. This gives the model access to a larger parameter pool than it evaluates in a single forward pass.

The attention blocks use grouped-query attention with 16 query heads and eight key-value heads. The public configuration also includes RMSNorm, per-head QK normalization, SwiGLU feed-forward blocks, RoPE with a base of 10,000, and tied token-embedding and language-model-head weights. The tokenizer configuration has a padded vocabulary of 50,304 tokens, and the model was pretrained with a 1,024-token context window.

The router uses sigmoid scores and an auxiliary-loss-free expert-bias mechanism. The bias affects expert selection but is stored as a buffer rather than a trainable parameter.¹

Main pretraining and annealing

The two phases consumed approximately 100 billion tokens. The main phase ran for about 66 hours on four H200 GPUs, or approximately 264 GPU-hours.

The duration figures cover the two training phases end to end and are intentionally approximate. GPU-hours are the number of GPUs multiplied by each phase's elapsed duration.

The original main-run log contains 8,107 sampled training records, 324 periodic validation measurements and 82 checkpoint-save records. It covers the run from initialization through the final main-phase checkpoint. Because the raw log survived, the main validation trajectory can be reconstructed without relying on live dashboard access.

FineWeb validation loss was 4.3557 at the first recorded validation point and generally declined over the run. Individual measurements rose and fell as the validation sample and nearby training batches changed in difficulty. The final main-phase measurement was 3.4112. The broad trend supports continued optimization; an isolated rise in the curve is not evidence of instability by itself.

Checkpointing continued throughout the phase. Important milestones and the final state were backed up remotely, which gave us recovery points as well as candidates for later evaluation.

Annealing

Annealing resumed from the main-phase checkpoint and moved to the narrower Cosmopedia-v2 and FineMath-4plus source families. We verified that the process resumed from the trained state rather than initialization. The phase added about 10 hours, bringing the complete schedule to roughly 76 hours and 304 GPU-hours.

The change in data distribution mattered when we evaluated the result. The immutable final result.json records a FineWeb validation loss of 3.5487, higher than the main-phase endpoint of 3.4112. FineWeb loss alone would therefore have favored the earlier checkpoint.

The downstream comparison pointed the other way.

Selecting the release checkpoint

We evaluated the pre-annealing and post-annealing checkpoints on the same seven-task zero-shot suite: HellaSwag, OpenBookQA, WinoGrande, ARC Challenge, ARC Easy, BoolQ and PIQA. For each task, the reported score uses normalized accuracy where available and accuracy otherwise. The headline number is the unweighted mean of the seven task metrics.

The surviving records report an average of 44.71 before annealing and 47.61 afterward, a gain of 2.90 points. Six of the seven task scores increased. BoolQ fell from 56.24 to 49.54.

Lower is better for validation loss; higher is better for the reported task scores. The table places the measurements together because their disagreement determined the release choice, not because their numerical scales are directly comparable.

We chose the annealed checkpoint because it had the higher average under this protocol. This is a specific release decision, not a claim that the checkpoint is better on every task or distribution. The suite contains seven zero-shot tasks, and one of them declined.

FineWeb validation loss and the downstream suite measure different things. The first measures next-token prediction on a held-out web distribution. The second samples performance across seven task formats. Annealing changed the training distribution toward synthetic textbook and mathematical material, so the divergence is plausible. We did not run the controlled ablation required to attribute the BoolQ decline, or the aggregate increase, to a particular part of that mixture.

There is also an evidence limitation. We retained the task list, zero-shot setting and aggregation procedure in code, while the scores survived in project records and the released model card. The original machine-readable evaluation JSON and exact package version were not retained. We therefore treat these as reported evaluation results, not a reproduced evaluation.

The two checkpoints are still useful together. Their disagreement shows why checkpoint selection needs a declared objective. A held-out loss can track optimization on its own distribution without determining which model performs better under a separate downstream protocol.

What we are releasing

The public release includes:

• The approved architecture and implementation.
• The HobbyLM model and GPT-2 tokenizer configuration.
• Selected milestone checkpoints from pretraining and annealing.
• The final result.json and the main training log.
• The approved routing-dynamics dataset.

The release contains selected checkpoints rather than every state saved during training. A single important checkpoint is approximately 8.5 GB, and an exhaustive upload would add substantial storage while revealing more of the internal training trajectory. The selected states cover meaningful stages for inspection and evaluation.

We name the approved source families but do not redistribute upstream text. Exact data mixture and training-recipe details (including sampling, optimizer and learning-rate construction, and efficiency techniques) remain internal. This boundary reflects both upstream licences and our own development roadmap, and will apply similarly to future work on context extension, supervised fine-tuning and reinforcement fine-tuning.

The open artifacts let researchers inspect the architecture, run the model and compare selected stages, but they do not reproduce Fuel Labs' complete data and optimization pipeline.

Limitations and lessons

The data total is a measure of scheduled consumption, not a count of unique documents. Cleaning and deduplication varied by upstream source because the project did not apply one uniform pass to the entire pool. The exact mixture, scheduling and preparation process is not public.

The checkpoint comparison is also narrow. It covers seven zero-shot tasks and has not been independently reproduced. The original evaluation JSON and exact package version are missing, so the published values remain reported results. The annealing time series was not retained; only its final validation result is presented here.

Three practices carried most of the engineering value from this run.

First, a proxy model is most useful when it tests the whole path around training. The smaller run found a data-delivery problem at a scale where we could still correct and repeat the experiment.

Second, expensive jobs need observable gates. A live process is not proof of progress. Our launch checks required a loss, validation, checkpoint output and expected hardware activity before the full configuration passed.

Third, evaluation output deserves the same retention discipline as model weights. Future runs should save the command, model revision, framework revision, raw JSON and a short manifest showing how each headline number was calculated. Those files are tiny beside a checkpoint and would have made the pre/post comparison independently reconstructable.

The main result is a trained model, but the more reusable outcome is the procedure that allowed us to trust the run. We tested the data stream before scaling it, required the system to prove that it could train and recover, and selected the release checkpoint against an explicit evaluation objective. Where the record does not support a broader conclusion, we have left it open.