Two Years of Small Language-Model Development: From Early Recurrent Models to Antares

1. Comparison principles

The models in this history were trained for different objectives and with very different data. A parameter-count table is therefore descriptive, not a controlled benchmark. Direct performance comparisons are only made where the same evaluation harness was used. Historical claims are separated into measured artifact facts, contemporaneous public observations, and reconstruction.

2. Model-lineage overview

Table 1. High-level model lineage. “Scale” is not a quality ranking.

3. Phase I: learning the mechanics

The earliest notebooks focused on fundamentals: tokenization, attention, positional encoding, encoder-decoder masking, training loops, and generation. The June 2024 GPT notebook trained a 20.98M-parameter Transformer on 99,951 movie-dialog samples and already showed a recognizable overfit pattern: training loss continued to improve while validation loss bottomed out and then rose. This phase supplied implementation experience rather than a deployable general language model.

4. Phase II: recurrent conversational scaling

MICRO-3 is the clearest demonstration that scale without data breadth was insufficient. At 167.9M parameters it was larger than Antares, but it trained on a compact dialogue corpus and its public generation contains long near-verbatim fragments from that corpus. The author’s original July 2024 note already called it “a bit overfit.” This is an important methodological result because it was identified at the time rather than imposed retrospectively.

Miranda then explored a different balance. The preserved 50.61M LSTM produced a much richer training trace, while the public MirandaGPT announcement described a far smaller 6.36M model that was fast on CPU and produced cleaner short completions. The period therefore began moving the project away from “larger is better” and toward practical model/data fit.

5. Phase III: causal decoders and deployment

Zeon was the first well-preserved end-to-end model stack in the archive. Zeon-1 used a 54.8M causal decoder with a FineWeb-derived tokenizer/corpus pipeline, was exposed through an API, and had measured server throughput around one token per second. Its launch post showed both correct and incorrect answers, establishing that conversational surface competence had advanced faster than factual reliability.

Zeon-2 kept essentially the same deployed model size but changed the system behavior through SFT. It learned to emit calculator/time tool tokens; deterministic runtime code executed the requested operation. This was a systems solution to a small-model limitation rather than an attempt to force exact arithmetic into 54M parameters. The architecture also became a stable deployment target, which made model/runtime attribution possible.

6. Phase IV: Antares and standardized evaluation

Antares increased model scale to 108,552,960 parameters and adopted a more systematic experimental record: checkpoint provenance, architecture specifications, optimizer/memory ablations, SFT audits, and explicit separation of model behavior from runtime routing. Most importantly for cross-model comparison, the final Antares Base and v0 checkpoints were evaluated with the same lm-evaluation-harness tasks used for GPT-2 Small under zero-shot raw causal-likelihood scoring, with no chat template, system prompt, or tools.

Table 2. Full-run zero-shot benchmark results. Headline mean uses the displayed task metric for each benchmark.

Antares v0 is only 0.10 percentage points behind GPT-2 Small on the five-task mean while using 15,886,848 fewer parameters (12.8% fewer). It exceeds GPT-2 on ARC-Easy and OpenBookQA, trails on HellaSwag and PIQA, and is close on WinoGrande. The aggregate should be interpreted as descriptive rather than proof of equivalence on all language capabilities.

7. SFT as a capability-shaping stage

Because Antares v0 is the SFT-trained continuation of Antares Base, the pre/post comparison is direct. SFT improves the headline mean from 39.15% to 42.37%, a gain of 3.22 percentage points, and improves every included benchmark. The gain closes approximately 97% of the original 3.32-point mean gap between Antares Base and GPT-2 Small.

Table 3. Antares Base to v0 benchmark change after SFT.

This is a more interesting result than a simple release-model score. The evaluation contains no chat formatting or tools, so the SFT stage changed the model’s raw continuation likelihoods in ways that improved these multiple-choice tasks. The current artifact set does not establish which SFT subsets contributed most, and the report therefore avoids attributing the gain to a specific behavior bucket.

8. What improved across the lineage

Table 4. Methodological progression across the project.

9. Discussion

9.1 Parameter count was not the dominant variable

The most visually striking comparison is MICRO-3 versus later models: 167.9M parameters did not produce the broad usefulness achieved by smaller Zeon or Antares systems. This is not a controlled scaling result because the architectures and data differ radically, but it is a strong engineering reminder that parameters are only useful when the objective, data, optimization, and evaluation are aligned.

9.2 Deployment changed the research questions

Once Zeon was publicly deployed, questions shifted from “does it generate text?” to “does it answer reliably, stop correctly, route tools, and run fast enough on available hardware?” Those product constraints influenced training. Zeon-2’s calculator/time routing is a direct example: the system improved arithmetic reliability without asking the language model to become a deterministic arithmetic engine.

9.3 Better evidence made the later work stronger

The Antares result is more defensible not merely because the model is better, but because the evaluation is better. The same harness, zero-shot conditions, raw likelihood scoring, explicit parameter counts, and pre/post-SFT checkpoints make it possible to state exactly what improved and what did not. That methodological progress is itself part of the model-development trajectory.

10. Limitations

No single standardized benchmark was run across MICRO, Miranda, Zeon, and Antares at the time each model was developed. Some historical checkpoints are missing, and several parameter counts come from public posts or code reconstruction rather than surviving weight files. Data reuse across early conversational experiments prevents clean causal comparison. The five-task Antares/GPT-2 mean is a compact diagnostic, not a substitute for a broad benchmark suite or statistical confidence intervals. The retrospective therefore treats the early models qualitatively and reserves direct numerical comparison for the Antares/GPT-2 runs that share an evaluation protocol.

11. Conclusion

The SliceLine model lineage shows a clear shift in what “progress” meant. In 2024, progress meant getting custom recurrent and Transformer models to generate recognizable dialogue. By Zeon, it meant training a causal decoder on web text, deploying it, measuring runtime performance, and teaching it to use deterministic tools. With Antares, it meant preserving a base checkpoint, auditing SFT, separating runtime behavior from model behavior, and running standardized full benchmarks. The strongest quantitative endpoint currently recovered is Antares v0: 42.37% on the five-task zero-shot mean versus 42.47% for GPT-2 Small, with 12.8% fewer parameters. The more important longitudinal result is how the project reached that point: not by monotonic scaling, but by progressively improving the relationship between architecture, data, training objective, system design, and evidence.

References

[1] Radford, A., et al. (2019). Language Models are Unsupervised Multitask Learners.

[2] Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS.

[3] Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation.

Primary project artifacts

[A1] Early foundation notebooks: TinyLanguageModel.ipynb, LightGPT.ipynb, GPT.ipynb.

[A2] MICRO-1, MICRO-3, MICRO-4 preserved notebooks and MICRO-3 public post.

[A3] Miranda.ipynb, MirandaGPT.ipynb, public MirandaGPT post, and Kaggle training source.

[A4] Zeon.ipynb and deployed Zeon archive (configs, logs, runtime code).

[A5] Antares preserved checkpoint/report artifacts and full GPT-2 vs Antares benchmark logs supplied by the project author.