The MICRO Models: Early Recurrent Language Modeling on Conversational Data

1. Scope

This report focuses on MICRO artifacts that can be reconstructed from surviving notebooks and public examples. MICRO-2 is not assigned specifications because a matching artifact has not yet been recovered in the current archive. MICRO-4 is included as an architecture experiment but excluded from capability comparisons because its loss/output construction differs from a standard language-model objective.

2. Shared data regime

The MICRO notebooks load local .txt files and train directly on conversation-form text. MICRO-3’s public sample and the recovered human_chat corpus share long identical or nearly identical stretches, including the “missing cat,” holiday suit, and sleepy-homework conversations. The corpus is rich in casual dialogue style but small and repetitive relative to the size of MICRO-3.

3. MICRO-1

MICRO-1 uses TensorFlow/Keras with a SubwordTextEncoder tokenizer, sequence length 100, batch size 512, a 512-dimensional embedding, and a 760-unit LSTM. The notebook-emitted model summary reports a 13,228-token vocabulary and exactly 20,709,164 trainable parameters. The model is configured for up to 500 epochs with Adam and an early-stopping callback.

Table 1. MICRO-1 configuration from the preserved notebook.

The notebook also preserves a highly noisy generation containing many malformed or rare subword fragments. That output is consistent with an early model still struggling to model the combined text distribution; no standardized evaluation was recorded.

4. MICRO-3

4.1 Architecture and scale

MICRO-3 dramatically increases recurrent capacity: 4,090-dimensional token embeddings, a 4,090-unit LSTM, a 4,161-token vocabulary, 300-token sequences, and batch size 16. The public announcement gives 167,882,301 parameters. That figure is exactly consistent with the notebook architecture and a 4,161-token vocabulary, so it can be treated as a verified architecture-level count. The notebook loads micro-3_83.weights.h5, indicating that a checkpoint from epoch 83 existed; the training call itself is commented in the preserved notebook, so the original full training trace is not available.

Table 2. MICRO-3 configuration.

4.2 Contemporaneous self-assessment

The 18 July 2024 public post is notable for its restraint. It states that the model was trained on a conversation dataset, had a 300-token context, generated some conversations, was “a bit overfit,” and was not considered especially powerful. This contemporaneous description is consistent with the recovered evidence and avoids the need for harsher retrospective language.

4.3 Training-data reuse in generation

The public MICRO-3 generation contains long passages that match the human_chat corpus. This is stronger evidence than merely noting topical similarity. The following examples occur in both the model output and the supplied training corpus with only minor or no changes:

Table 3. Examples of near-verbatim training-corpus reuse in the public MICRO-3 sample.

The most conservative interpretation is that MICRO-3 learned conversation structure but also reproduced memorized spans under generation. Given the very large recurrent state and small dialogue corpus, this supports the author’s original overfitting diagnosis. It does not mean that every generated token was copied or that the model learned nothing; rather, the public sample cannot be used as evidence of broad conversational generalization.

5. MICRO-4 as a short-lived architecture experiment

MICRO-4 uses a 5,000-word Keras tokenizer, a 128-dimensional embedding, a 128-unit LSTM, and a single sigmoid output trained with binary cross-entropy directly against padded token-ID sequences. That construction is not a standard next-token language-model objective: a single sigmoid output cannot represent a 5,000-way token distribution. The notebook is therefore best classified as an exploratory prototype rather than a successor whose quality can be compared numerically with MICRO-1 or MICRO-3.

Treating MICRO-4 this way is important for historical accuracy. Early research archives naturally contain dead ends and code experiments; a technical retrospective should distinguish them from trained release candidates rather than implying that every notebook was a production model.

6. What the MICRO series established

• Large parameter count did not compensate for a narrow training distribution. MICRO-3 was larger than later Zeon and Antares variants, yet its preserved evidence is dominated by local corpus fitting rather than broad task capability.
• Conversation formatting can be learned without instruction tuning when the raw corpus repeatedly contains explicit speaker labels, but this is not equivalent to general instruction following.
• Generation samples are more informative when compared directly with the training corpus. This turns an intuitive “overfit” impression into a documentable overlap finding.
• The project’s later move toward causal decoder-only Transformers, larger web-text corpora, and explicit train/evaluation separation addressed limitations that were already visible here.

7. Limitations

The original MICRO training datasets are not fully versioned, and the current archive does not include MICRO-2. MICRO-3’s notebook references an epoch-83 weight file but does not preserve the original training log. The public generation is a single qualitative sample, so corpus reuse can be demonstrated for that output but not converted into a model-wide memorization rate. MICRO-1 and MICRO-3 use different vocabularies and batch regimes, making direct training-efficiency comparison inappropriate.

8. Conclusion

The MICRO models are useful historical baselines precisely because their limitations are well documented. MICRO-1 established a working recurrent text-generation pipeline at 20.7M parameters. MICRO-3 scaled that idea to 167.9M parameters and produced recognizable conversation, while its own launch note correctly flagged overfitting and the surviving sample shows substantial corpus reuse. These experiments made clear that the next gains would need to come from better objectives, data breadth, and architecture rather than parameter growth alone. That lesson is visible in the subsequent Miranda, Zeon, and Antares work.

References

[1] Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation.

Primary project artifacts

[A1] Micro-1.ipynb, created 3 July 2024, modified 6 August 2024.

[A2] MICRO-3.ipynb, created 12 July 2024, modified 12 August 2024.

[A3] MICRO-4.ipynb, created 11 August 2024, modified 12 August 2024.

[A4] MICRO-3 public model post, 18 July 2024; supplied by the project author.

[A5] human_chat.txt corpus supplied by the project author.