MirandaGPT: Iterative Conversational Language-Model Development in 2024

1. Historical scope

The Miranda archive contains several artifacts created within weeks of each other. Their parameter counts and architectures differ materially, and no surviving manifest maps every file to a public release. Accordingly, the report uses artifact-specific names such as “Miranda LSTM notebook,” “MirandaGPT Drive Transformer,” and “public 6.36M MirandaGPT.” This avoids inventing a chronology that the evidence does not support.

Table 1. Miranda-family artifacts should be treated as separate iterations unless stronger checkpoint lineage is recovered.

2. Conversational data

The recurrent and later Kaggle experiments use a conversation corpus formatted as alternating “Human 1:” and “Human 2:” turns. The corpus contains casual workplace and social dialogue, travel, food, hobbies, holidays, jokes, and occasional technical topics. It is useful for learning conversational surface form but is not a broad factual pretraining corpus. The repeated greeting structure and recurring topics also make overfitting easy to detect in generated text.

Evidence note. The human_chat corpus is preserved in the project archive and is also mirrored publicly on Kaggle and Hugging Face. This report treats the supplied copy as the training-data artifact; it does not infer broader pretraining data unless a notebook explicitly loads it.

3. Miranda: 50.61M recurrent language model

3.1 Architecture and runtime

Miranda.ipynb implements a TensorFlow/Keras recurrent language model using a subword tokenizer, a 2,048-dimensional embedding, a 2,048-unit LSTM, and a vocabulary projection. The tokenizer vocabulary observed in the model summary is 4,161. Sequence length is 500 and batch size is 16. The notebook initializes a TPU strategy and records eight logical TPU devices. The model summary reports exactly 50,610,241 trainable parameters.

Table 2. Preserved Miranda LSTM configuration.

3.2 Preserved training trajectory

The notebook preserves a long training trace rather than only final weights. Early loss declines from 1.6746 at epoch 1 to 1.2603 at epoch 20, while validation loss moves from 1.6253 to 1.2848. Later logged epochs reach validation losses below 0.5 with token-level accuracy above 0.9. These metrics are internally consistent with learning the local conversation corpus, but they should not be interpreted as broad language-model accuracy because the data distribution is narrow and neighboring text chunks are highly related.

Table 3. Selected logged Miranda training points.

Generated text is recognizable as conversation-like English but remains noisy, with abrupt topic changes and malformed tokens. This is consistent with a recurrent model learning local turn structure without the data breadth needed for general-purpose language modeling.

4. MirandaGPT Drive Transformer experiment

The Drive notebook replaces the recurrent model with a custom PyTorch encoder-decoder Transformer: six encoder layers and six decoder layers, model width 768, eight attention heads, feed-forward width 1,024, sequence length 50, dropout 0.1, and a word-level vocabulary of 727 tokens. The notebook logs only 13,612 source characters and 3,214 word tokens. Reconstructing the parameter count from the exact code and dimensions gives 63,140,311 trainable parameters. This count is a code-derived reconstruction rather than a notebook-emitted measurement.

Table 4. Drive MirandaGPT configuration and data scale.

4.1 Preserved behavior

The notebook preserves next-token distributions and free generation. For “I love”, the model assigns 0.6156 probability to “noodle” and 0.3512 to “love”; for “In the future,” it assigns effectively all probability to “the”. A preserved generation from “I love” repeatedly cycles through phrases such as “prefer tea” and “prefer warmer climates”. These outputs are useful because they expose the narrow corpus directly rather than relying on a subjective conversational-quality claim.

5. Implementation audit and metric interpretation

Several implementation details substantially limit what can be inferred from the Drive Transformer’s training metrics. They are reported here as historical engineering findings, not as criticism of a production system: this was an early custom Transformer implementation used to learn the mechanics of language-model training.

• Encoder-side target visibility: each dataset item uses src=t0…t49 and tgt=t1…t50, while the decoder is trained against later positions of tgt. Because the encoder attends bidirectionally over src, many supervised target tokens are already present somewhere in the encoder input. Cross-attention can therefore exploit information unavailable in true next-token generation.
• Positional-encoding layout mismatch: the PositionalEncoding implementation constructs sequence-first positional tensors but receives batch-first embeddings. It slices positions using x.size(0), which is the batch dimension. During single-example generation, every token therefore receives the same position-0 vector.
• Overlapping-window validation: the dataset creates every adjacent 50-token window and then random-splits windows. Neighboring train and validation examples can share almost all tokens, making validation loss an optimistic measure of held-out generalization.

These issues explain why low loss or high-confidence next-token distributions should be read as evidence that the network fit the training construction, not as a general language benchmark. The artifacts remain valuable precisely because later projects moved toward cleaner causal-decoder objectives and more explicit evaluation.

6. Public 6.36M MirandaGPT

A public post dated 16 August 2024 describes MirandaGPT as a 6,365,070-parameter model created to improve Rustybyte AI’s NLP capability. The post states that it ran quickly on CPU and could produce short essays, answer Rustybyte-related questions, explain basic concepts, and generate simple conversations. Because no exact matching 6.36M source checkpoint has yet been identified in the recovered notebooks, this announcement is treated as an independent contemporaneous artifact rather than equated with the 50.61M or 63.14M experiments.

The key historical point is the scale direction: the publicly presented MirandaGPT was far smaller than MICRO-3 and was nevertheless described at the time as a major practical step forward. That observation supports the later SliceLine emphasis on architecture, data, and deployment constraints rather than parameter count alone.

7. Kaggle MirandaGPT continuation

A later recovered Kaggle script uses the human_chat corpus directly with a three-encoder/three-decoder Transformer, width 768, six heads, feed-forward width 4,090, sequence length 500, batch size 32, and a configured 50 epochs. Its preserved output ends in KeyboardInterrupt during training. The artifact therefore establishes the attempted architecture and data path but not a completed model or final metric. It also demonstrates that the MirandaGPT name continued to be used for experimentation rather than denoting one frozen specification.

8. Discussion

The Miranda period is best understood as an architectural transition. The project moved from very large recurrent models toward Transformer experiments and, in the public 6.36M variant, toward a model small enough to run comfortably on CPU. The surviving notebooks also exposed evaluation pitfalls that later SliceLine work could avoid: narrow-corpus accuracy is not broad capability; random splits of overlapping language windows are weak validation; and train/inference alignment matters as much as raw loss.

The artifacts should not be used to claim that MirandaGPT approached contemporary general-purpose language models. Their value is different: they document the point at which conversational generation became coherent enough to motivate a model product, and they preserve concrete implementation lessons that informed later decoder-only systems.

9. Limitations

The public 6.36M MirandaGPT source/checkpoint has not yet been matched to a preserved notebook, so its architecture cannot be reconstructed from the announcement alone. The Drive Transformer’s parameter count is reconstructed from code. The Kaggle run is interrupted. The human_chat corpus is narrow and was reused across multiple experiments, which complicates direct model-to-model comparison. No standardized benchmark suite was run at the time.

10. Conclusion

MirandaGPT was not one model but an iterative development phase. The recurrent Miranda notebook demonstrated that a 50.61M LSTM could learn strong local conversational statistics; the custom Transformer experiments explored attention-based generation and exposed important train/evaluation alignment issues; and the public 6.36M MirandaGPT demonstrated a shift toward practical CPU deployment and cleaner short completions. Read together, these artifacts form a credible bridge between the MICRO experiments and the later Zeon decoder-only family without requiring inflated capability claims.

References

[1] Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS.

[2] Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation.

Primary project artifacts

[A1] Miranda.ipynb, created 29 July 2024, modified 7 August 2024.

[A2] MirandaGPT.ipynb, created 5 August 2024, modified 12 August 2024.

[A3] MirandaGPT public announcement, 16 August 2024; supplied by the project author.

[A4] Kaggle MirandaGPT training source and KeyboardInterrupt trace; supplied by the project author.

[A5] human_chat.txt conversation corpus; preserved project copy and public mirrors supplied by the project author.