Horizon: Architecture Screening and Scaling of a 20M-Parameter Language Model Under a One-Hour Consumer-GPU Budget

Executive Summary

Horizon began as an intentionally overtrained small-model experiment: approximately 20 million parameters and a planned 2 billion base-training tokens, or roughly 100 tokens per parameter. That question was scientifically interesting but operationally incompatible with the project’s actual constraint. The best measured synthetic training rate on the target RTX 5070 was 206,233 tokens/s. At that optimistic ceiling, 2 billion tokens alone require 2.694 hours. Rather than silently abandon the original endpoint after seeing model-quality results, we archived it and adopted a one-hour principal experimental protocol before the main architecture comparison.

The revised study asks a different question: if a small independent lab has one consumer GPU-hour to spend on architecture research, where should that hour go? We treat the hour as a program budget, not a single-run budget. Candidate mechanisms are screened quickly, rejected when they fail the wall-clock objective, and the remaining compute is reallocated to the strongest survivor. This makes the paper explicitly exploratory and adaptive. It is not a claim of statistical significance from a single seed; it is a measured account of what one hour of disciplined iteration can and cannot buy.

The central empirical result is unusually clean. Every tested attempt to compress or complicate the 20M-class baseline lost on the study’s primary objective. Recurrent block reuse made the model smaller but worse and slower end-to-end. Factorizing the lexical embedding and reinvesting the saved parameters into the FFN preserved parameter count almost exactly, yet validation perplexity at 25M tokens increased from 218.60 to 438.95. External memory slightly improved raw validation loss, but memory-v3 required 6.15 minutes for the same 25M-token run that dense D8 completed in 2.82 minutes; memory-v6 required 17.42 minutes. Dense scaling, by contrast, produced a large and monotonic gain: 25M → 200M tokens reduced validation loss by 2.402 and perplexity by 11.0x.

Table 1. Principal one-hour program ledger. These eight runs, and only these eight runs, are counted toward the 3,600-second post-amendment budget.

Contributions

• A compute-budget-grounded protocol revision recorded before the main architecture results, preserving the original 2B-token endpoint as a superseded artifact instead of retroactively rewriting the experiment.
• A compute-matched recurrent-weight-sharing ablation covering 8, 4, 2, and 1 unique Transformer blocks at fixed effective depth eight.
• A parameter-matched factorized-embedding ablation that isolates whether lexical-table compression can be profitably reinvested into FFN width at the 20M-parameter scale.
• A parameter-matched external-memory comparison with routing telemetry, measured energy, and explicit separation of raw loss from wall-clock efficiency.
• A dense scaling trajectory to 200M tokens, including exact continuation semantics, validation checkpoints, energy accounting, and marginal-gain analysis.
• A checkpoint-by-checkpoint factual-logit probe showing that factual answer rank evolves irregularly even while aggregate validation loss improves smoothly.
• A reproducibility artifact containing 332 copied research files, 55 hashed checkpoint records, source/config snapshots, environment provenance, and an archive-level SHA-256.

1. Introduction

Small language models are often discussed as if “small” were a single axis. It is not. A model can be small in parameters yet expensive to train, small in training FLOPs yet slow on a particular GPU, or cheap to run once yet painfully slow to iterate on during development. For a small independent lab, the last axis can dominate. The practical bottleneck is frequently the latency between an architectural idea and enough evidence to decide whether the idea survives.

This paper treats wall-clock iteration time as a first-class scientific constraint. Horizon is not optimized primarily for inference latency, benchmark prestige, or parameter minimality. The target is language-model quality gained per wall-clock hour on one consumer GPU. This changes what counts as a successful mechanism. A modification that saves parameters but lowers hardware utilization can lose. A mechanism that slightly improves loss but doubles runtime can lose. Conversely, a conventional dense model can win simply because it converts the available hour into more useful optimization steps.

The study was motivated by an initially different experiment. Horizon was specified as an approximately 20M-parameter model trained for as many as 2B tokens to observe extreme overtraining and saturation. During pilot calibration, however, the best synthetic throughput was 206,233 tokens/s. A 2B-token run would therefore require at least 161.6 minutes of pure training, before real-corpus I/O, validation, checkpointing, and other overhead. The mismatch was large enough that no reasonable implementation detail could rescue the one-hour goal. We archived the 2B protocol and formally adopted a hard one-GPU-hour program budget before the main architecture comparison.

The resulting research process is deliberately adaptive. The initial screen isolates recurrent weight sharing. Subsequent factorized-embedding and external-memory screens were added as orthogonal exploratory mechanisms while budget remained. Once none improved the primary wall-clock objective, a documented execution amendment allocated the remaining budget to the dense D8 survivor. Because these later choices are informed by earlier observations, the work should be read as a budgeted exploratory study, not a confirmatory trial. We make no p-value or single-seed superiority claims.

The outcome is less glamorous than a new architecture and more useful for the stated constraint. Aggressive weight sharing improves parameter efficiency but degrades absolute validation loss and does not speed the run. Factorized embeddings at width 160 are a clear loss. External memory produces marginally lower 25M-token loss but severe runtime and energy penalties. Continued dense training dominates all of these interventions. At 200M cumulative tokens, D8 reaches validation loss 2.9855 and perplexity 19.80, compared with 5.3872 and 218.60 at 25M.

The paper also documents a second lesson: low validation perplexity is not equivalent to factual usefulness. A direct checkpoint sweep of eight factual prompts shows that some correct tokens move rapidly toward the top of the distribution, others improve non-monotonically, and several remain poorly ranked even at 200M. The 200M model can assign the token “4” rank 7 after “2 + 2 =” and the first token of “Jupiter” rank 16 after a Solar-System prompt, while “Berlin” remains rank 1,649 and “Paris” rank 271. Greedy generation correspondingly collapses into syntactically plausible repetition rather than reliable answers.

2. Related Work and Positioning

2.1 Dense decoder-only Transformers and scaling

Horizon uses the decoder-only Transformer family introduced by Vaswani et al. [1] and common in autoregressive language modeling. Its purpose is not to propose a new attention primitive. Instead, the dense baseline is intentionally conventional: RMSNorm [9], rotary position embeddings [10], SwiGLU-style feed-forward activations [11], grouped-query attention [8], tied lexical weights, and AdamW [12]. This makes the baseline a useful reference point for asking whether more unusual mechanisms earn their wall-clock cost at a 20M-parameter scale.

Language-model scaling work emphasizes predictable improvements with model size, data, and compute [4,5]. Horizon differs in objective and scale. We do not search for a globally compute-optimal model family. We ask what can be learned inside a fixed local wall-clock research budget on one specific consumer GPU. Hardware utilization, compiler behavior, and implementation maturity therefore matter directly, even when theoretical FLOPs appear favorable.

2.2 Weight sharing and recurrent depth

Reusing the same block across depth has precedents in Universal Transformers [2] and parameter-sharing methods such as ALBERT [3]. The attraction is clear: recurrent application of a small set of parameters can produce greater effective depth without storing an independent block at every layer. The systems risk is equally clear. Parameter count and per-token compute are not the same quantity. If the same block is executed eight times, the arithmetic work remains close to eight block applications, while compiler scheduling and memory behavior may become less favorable. Horizon therefore evaluates sharing at fixed effective depth and width, explicitly separating parameter efficiency from wall-clock efficiency.

2.3 Factorized lexical embeddings

ALBERT popularized factorized embedding parameterization as a way to decouple vocabulary embedding width from hidden-state width [3]. At larger vocabulary-to-hidden ratios, this can free substantial parameters. Horizon asks a narrower question: at a 16,384-token vocabulary and d_model 384, is a 160-dimensional lexical space sufficient if the saved parameters are reinvested into a wider FFN? The resulting F160-D8 variant is matched to the dense control within 4,096 parameters, making the comparison about allocation rather than total model size.

2.4 External and recurrent memory

External differentiable memory has a long lineage, including Neural Turing Machines [13], while recurrent state mechanisms such as Transformer-XL [14] attack context extension from another direction. Horizon’s memory-v3 and memory-v6 are local experimental mechanisms derived from earlier SliceLine memory research rather than implementations of those papers. They add 16 persistent 128-dimensional slots updated every 64 tokens and read at selected layers. The critical evaluation criterion is not whether memory can lower raw loss by a few thousandths, but whether the gain survives measured wall-clock and energy accounting.

2.5 Systems-aware experimental methodology

PyTorch [15], ahead-of-time graph compilation, fused optimizers, and GPU-specific kernel behavior can make nominally similar architectures behave very differently in practice. Horizon therefore records rolling throughput, end-to-end throughput, peak VRAM, power telemetry, GPU-active time, validation duration, checkpoint events, environment metadata, and hashes. We deliberately report implementation effects instead of pretending that architecture exists independently of the software stack on which it is trained.

3. Experimental System

3.1 Hardware and software

All principal experiments were executed on one NVIDIA GeForce RTX 5070 with 11.94 GiB addressable VRAM. The captured environment used Windows 11, Python 3.13.14, PyTorch 2.13.0+cu130, and CUDA 13.0. The GPU reports compute capability 12.0 and 48 multiprocessors. CPU performance was not a controlled independent variable; the study reports end-to-end GPU-active wall time as observed on the complete host system.

Training used bfloat16, fused AdamW where supported, sequence length 512, microbatch 32 for production screens, no gradient accumulation, gradient clipping at 1.0, and torch.compile for dense/shared/factorized variants. The memory variants were run eager because that was the fastest supported implementation of those mechanisms at the time of the experiment. This difference is intentional for the wall-clock objective but is also a limitation: the memory result measures the architecture-plus-implementation available to the project, not an implementation-independent theoretical lower bound.

Table 2. Core execution environment and training settings.

3.2 Dense D8 architecture

D8 contains eight independently parameterized Transformer blocks at model width 384. Attention uses six query heads and two KV heads, each of dimension 64, giving native grouped-query attention. The FFN hidden width is 1,152. Input and output lexical embeddings are tied over a 16,384-token vocabulary. RMSNorm, Q/K normalization, rotary positional encoding with theta 10,000, and zero dropout are used throughout. The maximum configured context is 4,096 tokens, although all paper training runs use 512-token sequences for throughput and comparability.

Table 3. Dense D8 architecture.

3.3 Corpus and tokenizer

The training artifact exposes 2,495,442,941 eligible training tokens across 491 shards. The source mixture is 1,247,568,736 FineWeb-Edu tokens, 499,112,938 Wikipedia tokens, 399,281,562 Cosmopedia tokens, 249,594,504 code tokens, and 99,885,201 general FineWeb tokens. Horizon reuses the Antares base-data recipe and tokenizer unchanged to enable controlled lineage comparisons. The tokenizer has 16,384 entries and SHA-256 57ab170c2d268d34… .

The validation artifact contains 939,709 tokens. Its source mixture is not identical to the training mixture: it contains FineWeb-Edu, Wikipedia, and code, but no Cosmopedia or general FineWeb entries in the captured validation manifest. Aggregate validation loss should therefore be interpreted as a consistent experimental comparator, not a perfectly source-matched estimate of the entire training distribution.

Table 4. Corpus composition recorded by the D8 run manifest.

3.4 Telemetry and scientific record

Each run writes append-only step, validation, event, system, and power streams, plus immutable run/config/environment/data manifests and checkpoint metadata. Checkpoints are hashed. Derived summaries are regenerated from those records. This design follows a simple principle: terminal output is disposable; JSON/JSONL is the experiment. It avoids relying on screenshots or manually transcribed numbers when a later paper pass needs to reconstruct a claim.

Throughput is reported with explicit semantics. Rolling tokens/s is a short-window measure useful for steady-state kernel behavior; end-to-end tokens/s divides actual processed tokens by the complete measured run duration and therefore includes startup, compilation, validation, checkpointing, and other overhead. The smoke run exposed why this distinction matters: late rolling throughput reached roughly 34k tok/s while the end-to-end rate was only about 23k tok/s.

4. Protocol and Budget Accounting

4.1 Superseding the 2B-token endpoint

The original protocol targeted 2B tokens, approximately 99.7 tokens per parameter, to examine extreme overtraining and eventual saturation. The pilot microbatch sweep provided a direct feasibility test. At the best measured synthetic rate of 206,233 tok/s, 2B tokens require 9,697.8 s, or 161.6 minutes. This is already 2.694x the entire one-hour budget before any real-world overhead. The protocol revision was therefore based on measured compute cost, not downstream model quality, and was timestamped before the main architecture comparison.

The archived 2B configuration and experiment design remain part of the evidence bundle. We do not treat the abandoned endpoint as a failed result; it is a different experiment that does not satisfy the present wall-clock constraint and remains future work.

4.2 One-hour program design

The adopted hard budget is 3,600 GPU-active seconds for the principal post-amendment experiment. The initial plan used progressive elimination: train all shared-core candidates to 25M tokens, continue finalists rather than restarting them, and truncate later stages if real throughput threatened the wall-clock limit. During execution, the research program remained adaptive. After shared-core sharing failed, factorized embeddings and external memory were screened as orthogonal mechanisms. Once none improved the primary quality-per-wall-clock objective, the remaining budget was assigned to the dense D8 continuation.

This adaptive behavior must be distinguished from preregistration. The switch from 2B to one hour was ex ante with respect to the main architecture results. The later choice of which mechanisms to test and the decision to scale D8 were informed by earlier observations and are therefore exploratory. The paper treats the resulting negative and positive findings as measured evidence for this hardware/software setting, not as universally significant architecture rankings.

5. Experiment I — Recurrent Weight Sharing

5.1 Design

All four candidates execute eight effective Transformer passes with the same width, attention geometry, FFN width, tokenizer, data order, optimizer, sequence length, microbatch, and 25M-token budget. Only the number of independently parameterized blocks changes. D8 uses eight unique blocks. S4x2 cycles four blocks twice. S2x4 alternates two blocks four times. S1x8 applies one block eight times. The comparison is intentionally compute-matched at the block-application level rather than parameter-matched by widening shared blocks, which would add FLOPs and confound the effect of reuse.

Table 5. 25M-token shared-core screen. Lower validation loss is better; higher throughput and loss reduction per GPU-hour are better.

5.2 Sharing saves parameters but degrades absolute quality

The loss ordering is monotonic in the amount of sharing. D8 finishes at 5.3872. S4x2 is 0.1400 worse, S2x4 is 0.3346 worse, and S1x8 is 0.5469 worse. Perplexity increases from 218.60 to 251.44, 305.45, and 377.71 respectively. The degradation is already visible at 5M and 10M tokens and widens by the 25M checkpoint, so the result is not driven by a single noisy endpoint.

Parameter efficiency tells a different story. Because the shared models remove large numbers of trainable weights while preserving eight block applications, validation-loss reduction per million parameters rises as sharing becomes more aggressive. S1x8 extracts more reduction per stored parameter than D8. That is a legitimate advantage if memory footprint or model size is the dominant constraint. It is not the objective of this paper, where final quality and wall-clock research efficiency dominate.

5.3 Sharing does not create an end-to-end speed win

Despite fewer trainable parameters, every shared model is slower end-to-end than D8. D8 processes 147,594 tok/s end-to-end; S4x2 reaches 125,779, S2x4 123,270, and S1x8 119,559. Their final rolling windows can be fast, but total run time grows from 2.82 minutes to 3.31–3.49 minutes. Recurrent parameter reuse therefore reduces storage without reducing the eight sequential block applications that dominate compute, and the actual software stack does not turn the smaller parameter state into a runtime advantage.

For the stated objective, all shared variants are eliminated. S4x2 remains the most interesting negative result because it removes roughly one third of parameters for a moderate loss penalty; a future parameter-matched widened-shared experiment could ask whether that storage advantage can be converted back into quality. Such a test would answer a different question and was not charged to the remaining one-hour budget.

6. Experiment II — Factorized Lexical Embeddings

6.1 Parameter-matched reinvestment

The factorized experiment asks whether a full-width lexical vector is the right use of parameters at this scale. F160-D8 compresses the 16,384-token embedding from width 384 to width 160 and uses a tied 160↔384 projection. The saved parameters are reinvested into the feed-forward network, widening the FFN from 1,152 to 1,544. Total trainable parameters are 20,064,640, only 4,096 more than D8, a difference of roughly 0.020%. Attention geometry, eight unique blocks, data stream, seed, optimizer, sequence length, microbatch, and token budget are fixed.

Table 6. Parameter-matched dense versus factorized-embedding screen at 25M tokens.

6.2 Result: lexical compression is a clear loss at width 160

F160-D8 finishes at validation loss 6.0844 compared with 5.3872 for D8, a deterioration of 0.6972 nats. Perplexity is 438.95 versus 218.60, almost exactly a 2.01x penalty. The factorized model is also slower end-to-end: 127,890 tok/s versus 147,594 tok/s. This is not a close trade-off. At 25M tokens, the compressed lexical space has not preserved enough information for the wider FFN to compensate.

The result does not establish that all factorized embeddings are bad. It rejects this specific 160-dimensional allocation under this training regime. Intermediate widths such as 256 or 320 could define a smoother trade-off, but spending main-budget minutes on that sweep was difficult to justify after the 160-dimensional point failed by such a large margin. The mechanism was therefore eliminated immediately.

7. Experiment III — External Memory

7.1 Memory mechanism and fairness boundary

The memory variants preserve the dense eight-block Transformer backbone but add 16 persistent memory slots of dimension 128, updated in 64-token chunks and read at selected layers. FFN width is reduced to approximately match the dense parameter count. Memory-v3 uses a simpler routing/update mechanism and exhibits strong slot concentration. Memory-v6 adds anti-collapse objectives and normalized routing intended to distribute writes across the slot set.

A crucial systems caveat is that the dense model is compiled while the memory path runs eager. Per-chunk CPU telemetry transfers were removed before the experiment, so the measured result reflects the fastest supported implementation available to the project, but not an architecture-independent kernel optimum. This is acceptable for the paper’s operational objective—what should the lab train now?—and insufficient for a claim that external memory is intrinsically 2–6x slower on all implementations.

Table 7. 25M-token external-memory comparison. Memory variants slightly improve raw loss but sharply worsen time and energy efficiency.

7.2 The only raw loss win is economically tiny

Memory-v3 achieves validation loss 5.3736, 0.0136 lower than D8, while memory-v6 reaches 5.3839, 0.0033 lower. These are the only tested architecture modifications that beat the dense baseline on raw endpoint loss. The differences are far too small, and the experiment far too under-replicated, to claim statistical superiority. They are nonetheless useful because they show that the memory mechanism is not simply nonfunctional; it can move the loss in the desired direction.

The wall-clock price is severe. Memory-v3 takes 368.8 seconds, 2.18x the D8 run duration. Memory-v6 takes 1,045.5 seconds, 6.17x D8. Measured GPU energy rises from 8.51 Wh for D8 to 17.03 Wh and 28.14 Wh. Validation-loss reduction per kWh falls from 515.9 for D8 to 259.1 for v3 and 156.3 for v6. Under a one-hour research budget, the few thousandths of loss are overwhelmed by the opportunity cost of not training more dense tokens.

7.3 Routing diagnostics: diversity is not the same as usefulness

Routing telemetry explains an important internal difference. V3 has mean route entropy 0.184, only 3.34 active slots on average, and mean maximum-slot probability 0.938. It is heavily collapsed. V6 reaches mean route entropy 2.600, activates all 16 slots, and lowers the maximum-slot probability to 0.143. For 16 slots, the maximum categorical entropy is ln(16)=2.773, so v6 is genuinely close to broad slot use.

Yet v6 is not the better language model. Its final validation loss is slightly worse than v3 and its runtime is dramatically worse. The anti-collapse machinery succeeds at its local routing objective without producing a corresponding gain in the global next-token objective. This is a useful warning against treating a healthy-looking auxiliary metric as evidence that the mechanism helps the task.

For the main objective, both memory variants are rejected. V3 is retained as the strongest mechanism-level negative result because it hints that memory can lower raw loss; a future implementation with conditional invocation, chunk-level fusion, or compiled kernels could revisit whether that signal can be obtained without paying a universal per-token tax.

8. Cross-Architecture Screen

The architecture screens are most informative when viewed together. At the same 25M-token training horizon, the variants form three different failure modes. Shared cores save parameters but lose quality and wall time. Factorized embeddings keep parameter count fixed but lose a large amount of quality. External memory preserves or slightly improves raw loss but consumes much more time and energy. Dense D8 is the only candidate that sits near the best region on all three axes relevant to the study: loss, time, and implementation simplicity.

9. Dense D8 Scaling to the Budget Limit

9.1 Exact continuation semantics

The 25M D8 run is not replayed. The continuation restores model weights, AdamW moments, exact training-loader RNG/data-stream state, global step, and cumulative token count. Stage counters reset, and a documented 1% warmup followed by cosine decay is restarted over the 175M-token continuation. Planned cumulative observations are 50M, 75M, 100M, 150M, and 200M tokens. The execution controller reserves 120 seconds for finalization and would stop early if reaching 200M threatened the 3,600-second program cap.

Table 8. Dense D8 validation scaling curve. The 25M checkpoint is continued rather than restarted.

9.2 Scaling dominates architecture surgery

The dense scaling trajectory is the largest effect observed anywhere in the program. From 25M to 50M tokens, validation loss drops by 0.9956 and perplexity by 2.71x. By 100M tokens, loss is 3.3688 and perplexity 29.04. At 200M tokens, loss reaches 2.9855 and perplexity 19.80. Relative to 25M, the final model reduces perplexity by approximately 11.0x. None of the architecture interventions produced an effect remotely comparable to simply spending the remaining budget on more dense training.

The diminishing-return pattern matters for the archived 2B-token question. The last observed 50M-token interval still improves loss, but much less than earlier intervals. Extrapolating this small curve to 2B would be unjustified; the paper therefore reports the bend without claiming a saturation exponent. The original long-run experiment remains the appropriate way to study that regime.

9.3 Training dynamics and systems behavior

9.4 Final budget accounting

The eight principal runs consume 3452.0 GPU-active seconds, or 57.53 minutes. This is 95.89% of the 60-minute hard budget, leaving 148.0 seconds of headroom. Measured GPU energy is 147.83 Wh. The program therefore reaches the nominal 200M cumulative D8 checkpoint without violating the budget.

Reproducing the final dense model is cheaper than discovering it. The D8 25M screen plus its 175M continuation consume about 20.52 GPU-active minutes and 67.47 Wh. The remaining roughly 37.01 minutes and 80.36 Wh are discovery cost: shared-core, factorized, and memory alternatives that were rejected. Reporting both numbers avoids a common accounting ambiguity in which only the final training run is described while the experimental search that selected it disappears.

10. Perplexity Is Not Factual Recall

10.1 Why probe logits instead of only generation

After the 200M run, greedy generation exposed an uncomfortable mismatch. “The capital of France is” did not produce “Paris”; it produced a self-referential continuation of the phrase “the capital of France.” “The capital of Germany is” behaved similarly. Stochastic top-k samples remained syntactically plausible but did not reliably recover the facts. Generation alone cannot distinguish absent knowledge from a correct token that is present but narrowly outranked by generic continuations, so we inspect the next-token distribution directly.

Eight prompts were evaluated at every available dense checkpoint: France→Paris, Paris→France, Germany→Berlin, Italy→Rome, largest planet→Jupiter, water freezes at→0, author of Hamlet→William Shakespeare, and 2+2→4. For multi-token answers, the first token is the primary ranking diagnostic and later tokens are evaluated teacher-forced. The checkpoint sweep was conducted after training and is diagnostic rather than a training-selection metric.

Table 9. Direct factual-logit probes at 200M cumulative tokens. For multi-token answers, rank/probability refer to the first answer token.

10.2 Knowledge emerges unevenly and non-monotonically

The checkpoint trajectories are not simple scaled versions of validation loss. The token “4” improves from rank 3,840 at initialization to rank 99 at 5M, rank 14 at 25M, and rank 5 at 50M, but later oscillates between ranks 5 and 13 before finishing at rank 7. The first token of “Jupiter” improves much more dramatically after 25M, reaching rank 40 at 75M and rank 15 at 150M. By contrast, “Berlin” is worse at 50–100M than at several earlier checkpoints before recovering to rank 1,649 at 200M. “Paris” improves sharply only after 100M, from rank 1,661 to 268 at 150M and 271 at 200M.

Multi-token answers reveal another pattern. At 200M, the first token “ J” of “Jupiter” is only rank 16, but once teacher-forced, “up” and “iter” are both rank 1. For “William Shakespeare,” “ William” is rank 100 while the following “ Shakespeare” is rank 8. The model can therefore strongly encode suffix structure conditional on an initial token that it does not itself select confidently. This is one reason free generation can understate partially learned associations while still correctly exposing that the association is not robust enough to dominate decoding.

10.3 Interpretation

These probes do not constitute a benchmark suite. Eight hand-selected prompts are too few for population-level claims, and some prompt forms may be rare or distributionally awkward in the pretraining data. Their value is mechanistic: they demonstrate directly that a validation perplexity of 19.80 does not mean the 20M base model is a dependable question-answering system. Horizon at this stage is a base language model, not an instruction-tuned assistant.

The diagnostic also provides a useful baseline for future SFT. If instruction tuning later causes Paris to appear reliably without materially changing the underlying next-token rank of “ Paris” in the base completion form, the improvement would primarily reflect prompt/task shaping. If factual ranks themselves improve after additional pretraining, that would be evidence of knowledge acquisition rather than decoding style. Preserving the base checkpoint sweep makes that distinction testable.

11. Pilot Lessons: Feeding the GPU Matters

The first real-data smoke run was intentionally small and exposed an implementation trap. It processed only 2,048 tokens per optimizer step and completed 5.001M tokens in 217.25 seconds, about 23k tok/s end-to-end. Late rolling throughput was roughly 34k tok/s. Because checkpoint-plus-validation overhead accounted for only about 3.26 seconds, telemetry and evaluation were not the primary cause. The GPU was simply underfed.

The subsequent synthetic sweep demonstrated a much higher hardware ceiling around 200k tok/s, and production switched to sequence length 512 with microbatch 32. The dense 25M screen then reached 147.6k tok/s end-to-end and roughly 180.7k tok/s in its final rolling window. The lesson is methodological: before interpreting a slow architecture as evidence about modeling, establish that the training loop is operating in a sensible hardware regime.

12. Discussion

12.1 Capacity-limited beats parameter-clever at this scale

Across the tested mechanisms, the 20M-class model appears more capacity-limited than parameter-inefficient. Removing independent blocks hurts. Compressing lexical vectors hurts even when the saved parameters are reinvested. External memory can shave a tiny amount from raw loss, but not enough to justify its universal runtime tax. The simplest interpretation is that, at this scale and token horizon, independently parameterized dense transformations and full-width lexical representations are still highly valuable.

This does not imply that the same architecture is optimal at 1M, 100M, or 1B parameters. It also does not imply that sharing, factorization, or memory are intrinsically poor ideas. Horizon deliberately prices the mechanisms on one RTX 5070 with the implementation stack that actually exists. A mechanism can fail here because its asymptotic benefit begins at a different scale, because its optimization is harder, because its kernel support is immature, or because the saved parameters are being reinvested poorly. The paper’s claim is narrower and more operational: given this one-hour decision, none earned more budget than dense scaling.

12.2 Wall-clock efficiency can reverse conclusions based on parameter count

The shared-core ablation makes this reversal explicit. S1x8 is dramatically more parameter-efficient than D8 in terms of loss reduction per million stored parameters, but worse in final quality, end-to-end throughput, and loss reduction per GPU-hour. If deployment memory were scarce, S1x8 or S4x2 might deserve further work. If research iteration time is scarce, they do not. “Efficient” is therefore incomplete unless the resource being optimized is named.

12.3 Auxiliary mechanism health is not task success

Memory-v6 is a particularly useful cautionary example. It was designed to prevent slot collapse, and the telemetry says it succeeds: near-maximal routing entropy, all 16 slots active, low maximum-slot concentration. Yet its language-model endpoint is not better than v3 and its runtime is much worse. The auxiliary objective solved the auxiliary problem. That is not the same as solving language modeling.

12.4 The strongest result is an allocation result

The program’s most consequential decision was to stop architecture roulette. Once the screens failed, continuing D8 produced orders-of-magnitude larger perplexity improvement than any tested mechanism. This suggests a general strategy for compute-constrained independent research: make mechanism screens cheap, define kill criteria, preserve negative results, and redirect compute quickly. The scientific contribution can be a map of what not to spend the next hour on.

12.5 Final model quality is still limited

Horizon’s 19.80 validation perplexity is a large improvement over its early checkpoints, but the factual probes and generations show that the model is not a reliable assistant. It has learned substantial linguistic structure; it has not acquired robust factual recall across even simple prompts. This distinction is important for small-model work, where smooth loss curves can create an illusion of capability that disappears as soon as the model is asked to commit to a specific token.

13. Limitations and Threats to Validity

Single seed. The principal screens use one seed and one data order. Endpoint differences smaller than ordinary run-to-run variance could reverse under replication. This is especially relevant to the 0.0136 and 0.0033 raw-loss advantages of memory-v3 and memory-v6. We therefore describe them as observed differences, not statistically significant wins.

One hardware/software stack. Wall-clock conclusions are intentionally hardware-specific. A different GPU, compiler, custom CUDA kernel, or framework version could change the ranking, especially for memory. Dense/shared/factorized variants are compiled while the memory mechanisms are eager. That asymmetry is part of the operational measurement and a confound for architecture-intrinsic speed claims.

Short training horizon relative to corpus. The final D8 model processes only 200M of 2.495B eligible training tokens, approximately 8% of the available corpus and 9.97 tokens per parameter. The study does not measure the archived 2B-token saturation regime and should not extrapolate its final marginal-gain curve over an order of magnitude in tokens.

Validation mixture mismatch. The captured validation set contains FineWeb-Edu, Wikipedia, and code, while training also includes Cosmopedia and general FineWeb. Aggregate validation loss is consistent across compared runs but is not an exactly source-matched corpus estimate.

No downstream benchmark suite in the principal hour. The paper does not report HellaSwag, PIQA, ARC, Winogrande, or OpenBookQA results for these checkpoints. The one-hour program focused on architecture screening and base pretraining. The factual-logit suite is diagnostic and intentionally not presented as a substitute for standardized evaluation.

Adaptive exploratory design. The one-hour revision itself precedes the main architecture results, but later factorized/memory screens and the decision to allocate the remainder to D8 are adaptive. This is a feature of the research-budget question, not a confirmatory preregistration. Independent replication should pre-specify the complete candidate set if hypothesis testing is desired.

Energy boundary. GPU energy is measured; whole-system energy, electricity price, hardware amortization, and cloud-equivalent costs remain null because no assumptions were supplied. The paper therefore does not convert Wh into currency or claim a full environmental footprint.

14. Reproducibility and Evidence Provenance

The supplied paper-evidence archive was generated at 2026-08-21T17:54:04.494231+00:00 and contains 332 copied files out of 387 discovered paper-relevant artifacts. It inventories 55 checkpoint binaries without embedding the large .pt files and records their SHA-256 hashes. The archive itself has SHA-256 8d7cabd0013a6a43fb043bebab7651ba15b33e6ff8aa8c5abf297abb2f9d9dac. The final 200M checkpoint has SHA-256 1e2baaa6874f47eb71c35add82de2c67ba6329aac7cb87c8df2e17cda88ddaf7.

The archive includes run manifests, resolved configurations, data manifests and shard fingerprints, environment snapshots, raw training/validation/system/power/event streams, derived screen summaries, protocol amendments, benchmark calibration records, source code, tokenizer metadata, and the checkpoint-level factual-logit sweep. A later paper pass can therefore reconstruct the figures and claims without relying on conversational memory.

Table 10. Primary provenance identifiers.

15. Conclusion

Horizon asks a deliberately practical architecture question: what should a small lab do with one consumer GPU-hour if the goal is to improve a scratch-trained 20M-parameter language model as much as possible while learning something scientifically useful? On the tested stack, the answer is not recurrent weight sharing, not 160-dimensional lexical factorization, and not universally active external memory. Weight sharing saves parameters but hurts quality and wall time. Factorization misallocates capacity. Memory hints at raw-loss value but spends too much time obtaining it. Dense D8, trained longer, wins decisively.

The one-hour constraint did more than save time. It forced architecture ideas to compete against their opportunity cost. The complete principal program—seven 25M-class screening runs plus a 175M-token continuation—used 57.53 GPU-active minutes and 147.83 Wh. The final model reached 200.016M cumulative tokens, validation loss 2.9855, and perplexity 19.80. Yet direct logits still rank “Paris” 271st after a canonical capital-of-France completion. The model is substantially better at modeling text than at behaving like a factual assistant.

The strongest methodological lesson is therefore dual: measure the resource you actually care about, and measure capability at the level where you intend to use it. Parameter count is not wall-clock efficiency, auxiliary routing health is not language-model quality, and validation perplexity is not factual reliability. A disciplined negative result can be more valuable than a complicated mechanism kept alive because it looks novel.

After several hours of increasingly sleep-deprived experimentation, the authors identified a highly promising architecture: a datacenter. Funding inquiries are therefore strongly encouraged.

References

[1] A. Vaswani et al., “Attention Is All You Need,” NeurIPS, 2017.

[2] M. Dehghani et al., “Universal Transformers,” arXiv:1807.03819, 2018.

[3] Z. Lan et al., “ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations,” ICLR, 2020.

[4] J. Kaplan et al., “Scaling Laws for Neural Language Models,” arXiv:2001.08361, 2020.

[5] J. Hoffmann et al., “Training Compute-Optimal Large Language Models,” arXiv:2203.15556, 2022.

[6] G. Penedo et al., “The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale,” NeurIPS Datasets and Benchmarks, 2024.

[7] L. Ben Allal et al., “SmolLM2: When Smol Goes Big — Data-Centric Training of a Small Language Model,” arXiv:2502.02737, 2025.

[8] J. Ainslie et al., “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints,” EMNLP, 2023.

[9] B. Zhang and R. Sennrich, “Root Mean Square Layer Normalization,” arXiv:1910.07467, 2019.

[10] J. Su et al., “RoFormer: Enhanced Transformer with Rotary Position Embedding,” arXiv:2104.09864, 2021.

[11] N. Shazeer, “GLU Variants Improve Transformer,” arXiv:2002.05202, 2020.

[12] I. Loshchilov and F. Hutter, “Decoupled Weight Decay Regularization,” ICLR, 2019.

[13] A. Graves, G. Wayne, and I. Danihelka, “Neural Turing Machines,” arXiv:1410.5401, 2014.

[14] Z. Dai et al., “Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context,” ACL, 2019.

[15] A. Paszke et al., “PyTorch: An Imperative Style, High-Performance Deep Learning Library,” NeurIPS, 2019.

Appendix A. Full 25M Architecture Screen

This appendix consolidates the validation trajectories and efficiency measurements for all main 25M-token architecture screens. It is included so that the negative results remain visible rather than being reduced to a single endpoint table.

Appendix B. Full Factual-Probe Trajectory

The table below reports first-answer-token rank at every dense checkpoint. Lower rank is better. These diagnostics were collected after training and did not influence the architecture or scaling decisions.

Table 11. Correct first-token rank across dense checkpoints.

Appendix C. Checkpoint and Artifact Integrity

The evidence collector hashed all 55 checkpoint binaries without embedding them into the paper archive. The following table lists a compact identifier for each checkpoint. Full 64-character SHA-256 values remain in CHECKPOINTS.json; truncated values here are for visual indexing only and must not be used as the sole integrity check.

Table 12. Checkpoint inventory (55 binaries). Full hashes are preserved in the evidence bundle.

Appendix D. Unscheduled Toy Experiments

During the final scaling run, we conducted two deliberately pathological toy experiments at extreme model and vocabulary scales. These experiments were not part of the main Horizon protocol, were not used for model selection or quantitative claims, and are included solely as illustrative curiosities about degenerate language-model objectives.

D.1 Horizon-Atom

Horizon-Atom uses a one-token vocabulary and one trainable scalar. Because the vocabulary admits no alternative prediction, the cross-entropy objective is exactly zero regardless of the scalar. The observed toy run saw 5,120 tokens in approximately 2.710 ms, with loss 0, perplexity 1, and gradient 0. This is not “learning”; it is a degenerate objective with zero entropy by construction.

D.2 Horizon-Binary

Horizon-Binary takes the opposite pathological direction: two vocabulary symbols but exactly 16,384 trainable parameters. A 2×4,096 input embedding contributes 8,192 parameters and an untied 2×4,096 output matrix contributes another 8,192. The training language alternates deterministically, 0→1 and 1→0. The toy CPU implementation saw 819,200 tokens in 7.369 s, reached 100% accuracy on optimization step 2, and finished at effectively zero loss and perplexity 1.

The point is deliberately narrow: sufficiently low-entropy task design can make almost any model appear perfect. These experiments therefore serve as a humorous control on benchmark interpretation, not as evidence for small-language-model capability.

Appendix E. Additional Systems Diagnostics

Appendix F. Reproducibility Checklist

Table 13. Reproducibility checklist for the supplied Horizon evidence artifact.