Zeon: Development and Deployment of 54M-Parameter Language Models with Tool-Augmented Post-Training

1. Scope and evidence

The objective is historical and technical: establish what Zeon actually was, which artifacts correspond to deployed models, and which claims are directly supported. Preserved artifacts take priority over later recollection. Where a training configuration exists without a completion log, this report describes it as configured or attempted rather than completed.

Table 1. Evidence classes used for the Zeon reconstruction.

2. Zeon-1

2.1 Data preparation

The preserved Zeon preparation configuration identifies the raw corpus as HuggingFaceFW/fineweb, configuration sample-10BT. The data pipeline imposed a two-billion-character ingestion cap, ignored source texts shorter than 200 characters, trained an 8,192-token byte-level tokenizer, and packed the resulting token stream into 384-token sequences. The preserved metadata reports 533,757,840 cached tokens across 1,386,384 packed rows. These figures describe the available prepared corpus; they do not imply that the model consumed the entire cache during training.

Table 2. Zeon-1 data-preparation metadata.

2.2 Architecture

Zeon-1 is a conventional causal decoder-only Transformer by small-model standards: 12 blocks at width 512, eight attention heads, RMSNorm, a SwiGLU feed-forward network with hidden width 2,048, learned absolute position embeddings, causal scaled-dot-product attention, and tied input/output token embeddings. The preserved training output prints an exact parameter count of 54,800,896.

Table 3. Zeon-1 architecture.

2.3 Preserved pretraining run

One preserved Colab run used AdamW-style training with a base learning rate of 2e-4, batch size 16, 384-token sequences, and a planned 20,000 optimizer steps. The notebook output records loss falling from 9.1167 at step 1 to 4.6226 at step 4,320. Validation loss fell from 6.8465 at step 200 to 4.5574 at step 4,200. The run then received Ctrl+C and began checkpoint preservation. At step 4,320 the log reports 26,542,080 token presentations. This is strong evidence for one Zeon-1 training trajectory, but the archive does not establish that this interrupted Colab run is identical to every later deployed checkpoint.

Table 4. Selected points from the preserved Zeon-1 pretraining log. Approximate entries are explicitly marked.

3. Public capability snapshot

The 29 April 2026 launch post described Zeon-1 as a 54-million-parameter GPT-style decoder-only Transformer trained on web text. The post was deliberately modest: it characterized the model as basic but functional, highlighted correct conversation formatting, and explicitly called knowledge and arithmetic unreliable. The examples are useful because they preserve both successes and errors rather than presenting a curated success-only sample.

Taken together, these outputs show that Zeon-1 had learned a useful answer format and some factual associations, but retrieval reliability was low. The contemporaneous wording “It can answer questions when the required knowledge is present” is therefore a fair description of the observed state rather than a post-hoc downgrade.

4. Deployment performance

The launch post reported roughly 16 tokens per second on a Ryzen 5 3600 and roughly one token per second on the server. The archived production log independently supports the second figure. Several 64-token generations took between 55.6 and 65.2 seconds, corresponding to 0.98-1.15 tokens/s; a 32-token generation completed at 1.29 tokens/s. The same log prints 54,800,896 parameters. This is unusually good provenance for a small experimental deployment because the public performance statement can be checked against a surviving runtime record.

Table 5. Selected production throughput records from the Zeon-1 server log.

5. Zeon-2

5.1 Deployed architecture continuity

The deployed Zeon-2 runtime loads a ChatLM with the same core architectural family as Zeon-1: 12 layers, width 512, eight-head multi-head attention with a combined QKV projection, learned absolute position embeddings, RMSNorm, SwiGLU with 2,048 hidden units, and tied embeddings. The runtime loads the checkpoint with strict state-dictionary matching. With the deployed tokenizer size of 8,202, this architecture contains 54,806,016 parameters. Zeon-2 was therefore primarily a training and behavior iteration at essentially the same parameter scale rather than a straightforward scale-up.

5.2 Preview and final SFT

The recovered SFT records show a two-stage process. The preview used 600 steps, a 2e-5 learning rate, a 15% validation split, and aggressive weighting of the first assistant token and tool-opening tokens. It recorded 496,734 supervised tokens. The final run used up to 4,000 steps at 5e-5, trained on all available SFT lines with validation disabled, and recorded 3,698,586 supervised tokens. Both runs used assistant-only supervision: the user/system context was present, but the loss was concentrated on assistant output tokens.

Table 6. Preserved Zeon-2 SFT settings.

The final configuration’s routing weights are mostly 1x; only sequence termination receives additional weight. The preview, in contrast, strongly emphasized tool selection. This suggests that the preview stage was used to make routing behavior easy to acquire, while the longer final stage broadened behavior and reinforced clean termination. That interpretation is plausible but not directly proven by an ablation.

6. Model-level tool selection and runtime execution

Zeon-2 introduced explicit calculator and time tokens: <|tool:calc|>...<|tool:calc_end|> and <|tool:time|>...<|tool:time_end|>. The training set includes examples where the assistant is supervised to emit these tokens. In the production API, generation begins normally; when the model emits a tool opener, the stream processor captures the tool payload and executes a deterministic calculator or time backend. Thus tool selection and argument generation are model behaviors, while the numerical/time result itself is produced by runtime code.

Table 7. Attribution of Zeon-2 tool behavior.

This distinction matters for historical evaluation. Zeon-2’s improved arithmetic should not be described as the neural network learning reliable multiplication internally. The engineering improvement was a learned route into a deterministic calculator, which is a legitimate system-level capability and was one of the project’s explicit post-training goals.

7. Alternate Zeon-2 architecture branch

A separate recovered Zeon-2 pretraining program defines a more LLaMA-like 54M model with RoPE, grouped-query attention (eight query heads and two KV heads), a 2,304-wide SwiGLU FFN, zero dropout, and depth-scaled residual initialization. It is a meaningful design experiment, but it should not be conflated with deployed Zeon-2. The deployed SFT/runtime model expects parameters named qkv/proj plus a learned pos_emb; the alternate model uses separate q_proj/k_proj/v_proj/o_proj tensors and no learned positional table. Because the deployment loader uses strict=True, the two checkpoint layouts are not directly interchangeable.

Evidence note. The archived production runtime therefore establishes the older ChatLM family as the deployed Zeon-2 architecture. The RoPE/GQA implementation is reported as an alternate development branch unless a future checkpoint-conversion artifact demonstrates otherwise.

8. Discussion

Zeon’s main contribution to the SliceLine lineage was not raw benchmark performance. It was the point where the project assembled the pieces of a complete language-model product: a scratch-trained causal decoder, a sizeable web-text preparation pipeline, a tokenizer, conversational SFT, a streaming API, measured CPU/server inference, and model-triggered deterministic tools. The public examples also established a useful norm for later work: show failures alongside successes.

The 54M scale made weaknesses highly visible. A model could learn the grammatical form of a capital-city answer while substituting an unrelated token, or learn the social form of helpfulness without robust knowledge. Zeon-2 addressed one class of weakness by moving exact arithmetic and time retrieval into tools rather than trying to force a small parametric model to be a calculator. That systems lesson carries directly into later SliceLine work.

9. Limitations

The surviving archive is incomplete. Some checkpoints visible on the original server are not included in the current zip, so parameter counts for deployed variants are reconstructed from exact runtime architecture/tokenizer dimensions where an emitted count is unavailable. The Zeon-1 Colab log documents one interrupted run but does not by itself prove that the same checkpoint was deployed. The final Zeon-2 SFT run disabled validation, so its training loss cannot establish generalization. The alternate RoPE/GQA branch lacks evidence tying it to production. No standardized zero-shot benchmark comparable to the later Antares evaluation has yet been recovered for Zeon-1 or Zeon-2.

10. Conclusion

Zeon was a compact but important development stage. Zeon-1 established a 54.8M-parameter decoder-only baseline trained from scratch on a FineWeb-derived corpus and deployed it as an interactive model. Zeon-2 kept essentially the same deployed architecture while investing heavily in assistant-only SFT and learned calculator/time routing. The result was not a claim of broad language-model competence; it was a working small-model stack whose strengths and weaknesses were visible in production. The preserved artifacts make that progression unusually auditable and provide a clean bridge between the earlier conversational experiments and Antares.

References

[1] Radford, A., et al. (2019). Language Models are Unsupervised Multitask Learners.

[2] Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS.

[3] Penedo, G., et al. FineWeb dataset and associated technical documentation, Hugging Face.

Primary project artifacts

[A1] Zeon.ipynb, created 13 April 2026, modified 26 April 2026; preserved Colab notebook.

[A2] deployed/1/meta.json and prepare_config.json; preserved Zeon-1 preparation metadata.

[A3] deployed/1_pt/model-api-19002.log; preserved production throughput log.

[A4] Zeon-1 public launch post, 29 April 2026; supplied by the project author.

[A5] deployed/v2/train_config.json, progress.json, model_api.py; final Zeon-2 SFT/deployment artifacts.

[A6] deployed/v2_preview/train_config.json and progress.json; Zeon-2 preview artifacts.

[A7] Alternate RoPE/GQA Zeon-2 pretraining source supplied by the project author.