1. Why separate prototypes from model releases
A historical archive can easily overstate continuity if every notebook is retrospectively called a “model generation.” The evidence supports a more precise taxonomy: external-model inference experiments, architecture exercises, trained custom models, and public/deployed releases. Only the latter two categories should be compared as project language models.
| Category | Example | What it establishes |
|---|---|---|
| External-model inference | TinyLanguageModel.ipynb using CohereForAI/aya-101 | API/model-loading experimentation; not own weights |
| Architecture exercise | Short custom Transformer notebooks | Implementation practice; no capability claim |
| Trained custom model | LightGPT / GPT notebook | Own architecture and training run |
| Later public model | MICRO / Miranda / Zeon | Separate papers in this series |
Table 1. Artifact taxonomy used in this report.
2. March 2024: external-model experimentation
TinyLanguageModel.ipynb loads CohereForAI/aya-101 with Hugging Face Transformers and performs multilingual generation. The notebook begins downloading the public checkpoint shards. Because the model weights are external, this artifact should not appear in a parameter-growth chart for SliceLine’s own models. Its relevance is that it predates the custom training work and shows early experimentation with tokenizers, generation APIs, and multilingual language models.
3. LightGPT: custom Transformer construction
LightGPT.ipynb implements multi-head attention, sinusoidal positional encoding, encoder and decoder layers, masking, and an encoder-decoder Transformer in TensorFlow/Keras. It trains a SubwordTextEncoder vocabulary of 1,706 tokens over local text, uses a maximum sequence length of 256, four layers, eight heads, a 1,024-unit feed-forward network, batch size 64, and a configured five epochs. The notebook demonstrates a full custom Transformer path rather than wrapping a pretrained model.
| Field | LightGPT |
|---|---|
| Tokenizer vocabulary | 1,706 before start/end tokens |
| Sequence length / d_model | 256 |
| Layers | 4 |
| Heads | 8 |
| Feed-forward units | 1,024 |
| Batch size | 64 |
| Configured epochs | 5 |
Table 2. Selected LightGPT settings.
The implementation couples model width to the maximum sequence length (d_model=MAX_LENGTH). That is a legitimate early simplification for experimentation but not a design pattern retained by later models. The main historical value of LightGPT is that the core attention stack, masking, and training loop were implemented directly.
4. GPT notebook: a larger dialogue Transformer
The later GPT.ipynb uses the Cornell Movie-Dialogs corpus, builds a subword vocabulary of 8,173 tokens, filters/pads dialogue pairs to length 200, and trains a custom encoder-decoder Transformer with two layers, model width 512, four heads, and 512 feed-forward units. The model summary reports exactly 20,977,133 parameters. The dataset preparation prints 99,951 question-answer samples.
| Field | GPT notebook |
|---|---|
| Dataset | Cornell Movie-Dialogs |
| Samples | 99,951 |
| Vocabulary | 8,173 |
| Maximum length | 200 |
| Transformer layers | 2 encoder + 2 decoder |
| Model width | 512 |
| Heads | 4 |
| FFN units | 512 |
| Parameters | 20,977,133 |
Table 3. GPT notebook configuration.
4.1 Preserved training behavior
The run was configured for 500 epochs, but the notebook preserves 22 complete epochs and part of epoch 23. Each complete epoch took roughly ten minutes in the recorded environment. Training loss falls steadily from 0.4146 to 0.2260 by epoch 22, while validation loss improves initially from 0.3617 to about 0.3347 around epoch 6 and then rises to 0.3693 by epoch 22. This is a clear early example of the project observing the distinction between training fit and validation behavior.
| Epoch | Train loss | Validation loss | Validation accuracy |
|---|---|---|---|
| 1 | 0.4146 | 0.3617 | 0.0189 |
| 6 | 0.3011 | 0.3347 | 0.0211 |
| 10 | 0.2731 | 0.3423 | 0.0211 |
| 22 | 0.2260 | 0.3693 | 0.0201 |
Table 4. Selected preserved GPT-notebook training metrics.
The absolute accuracy values are not directly comparable to later language-model benchmarks because they are token-level metrics inside a sequence-to-sequence training setup. The useful observation is the trend: the model continued fitting training data after validation loss had stopped improving.
5. Architecture exercises and short-lived branches
The archive also contains brief notebooks that instantiate Transformer blocks, synthetic random-token datasets, or experimental objectives. For example, the notebook titled MoronGPT defines custom positional encoding and Transformer classes but trains on randomly generated token sequences for only two epochs. Such files are evidence of architecture experimentation, not language capability. Keeping this distinction explicit prevents the research record from inflating the number of substantive model releases.
6. Transition into the own-model lineage
By mid-2024 the project had moved through three useful stages: using public language models to understand inference; implementing attention/sequence-to-sequence systems from first principles; and training custom dialogue models long enough to observe overfitting. The MICRO series then shifted toward recurrent free-text completion on a local conversation corpus, Miranda explored both recurrent and Transformer variants, and Zeon eventually standardized on a causal decoder-only architecture with web-text pretraining. In that sense, the early archive is best read as the engineering foundation for the later model family rather than as a set of products that need to be defended by modern benchmark standards.
7. Limitations
This report audits selected notebooks with preserved content. The Drive folder contains additional 2023-2024 files whose names indicate chatbot, classification, reinforcement-learning, and neural-network experiments, but their technical contents have not all been reconstructed in the present pass. The report therefore avoids assigning architectures or results to those files from titles alone. Some training runs lack final checkpoints or complete output histories.
8. Conclusion
The pre-MICRO archive shows a rapid progression from consuming public models to implementing and training custom sequence models. LightGPT established a from-scratch Transformer implementation path. The larger GPT notebook produced a 20.98M-parameter dialogue model and preserved enough training history to expose emerging overfit behavior. These were not yet general-purpose language models, but they provided the practical knowledge required for the project’s later scratch-trained systems. Presenting them as foundations rather than overstating their capability gives the SliceLine lineage a stronger and more credible beginning.
References
[1] Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS.
[2] Danescu-Niculescu-Mizil, C., & Lee, L. (2011). Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. Workshop on Cognitive Modeling and Computational Linguistics. (Cornell Movie-Dialogs corpus.)
Primary project artifacts
[A1] TinyLanguageModel.ipynb, created 12 March 2024, modified 23 March 2024.
[A2] LightGPT.ipynb, created 14 June 2024, modified 17 June 2024.
[A3] GPT.ipynb, created 18 June 2024, modified 3 July 2024.
[A4] MoronGPT.ipynb, created 1 August 2024; architecture/synthetic-data exercise.