Antares: Resource-Constrained Pretraining and Post-Training Analysis of a 108M-Parameter Language Model

1. Introduction

Language-model training studies commonly report parameter count, token count, floating-point operations, or downstream benchmark performance. For a small independent training effort, however, the dominant experimental constraint can be wall-clock iteration time. Antares was developed under this constraint: candidate methods were evaluated primarily by the useful language-model quality obtained per unit of elapsed training time on consumer hardware, rather than by theoretical compute efficiency alone.

Antares was also intended to remain a scratch-trained model rather than an adaptation of a larger public checkpoint. Earlier planning work placed a dense decoder-only model in the 100–120M parameter range near the practical center of this regime for a corpus containing a few billion tokens (Artifact A9). The subsequent design process therefore focused on reducing the cost of scratch pretraining while retaining a conventional architecture with mature kernel support.

The development record contains both positive and negative results. Explicit external memory improved synthetic associative recall per token but introduced routing and throughput costs. A short local optimizer comparison favored Muon during early training, whereas the preserved final base checkpoint originated from a one-pass AdamW branch. Base-model probes showed nontrivial lexical, factual, and code structure but substantial retrieval asymmetry. Supervised post-training imposed assistant-format behavior efficiently, while a planned multi-level reasoning-mode dataset was rejected after audit because the supervision contained extensive recurring metacognitive templates. These outcomes motivate a broader methodological point: at small scale, model architecture, data quality, and runtime orchestration can each dominate apparent capability.

1.1 Research questions

1. Does explicit external memory improve learning efficiency sufficiently to offset its additional wall-clock cost?
2. Which architectural and optimization choices are defensible for a scratch-trained model near 100M parameters on a single consumer GPU?
3. What capabilities are present in the preserved base model prior to instruction tuning?
4. Which behaviors are readily imposed by small supervised fine-tuning datasets, and which claims are not supported by those datasets?
5. How should deployed tool behavior be attributed when the serving runtime performs deterministic routing outside the model?

1.2 Contributions

• A preliminary comparison of Transformer and external-memory architectures under matched-token and matched-wall-clock conditions on a synthetic associative-recall task.
• A locally measured early-training comparison between AdamW and Muon in a 36.6M-parameter Antares pilot.
• A documented 108.55M-parameter final architecture and a set of next-token probes examining lexical, factual, arithmetic, and code associations before SFT.
• A quantitative audit of the post-training corpora, including a 1,000-example five-level reasoning-mode dataset that was excluded from reasoning claims because of template contamination.
• A runtime-attribution analysis demonstrating that deterministic orchestration can confound evaluations of model-level tool selection.

2. Study Scope and Evidence

The work was conducted as an iterative engineering study rather than a preregistered benchmark campaign. Accordingly, evidence is reported with explicit provenance. Numerical results from preserved logs are treated as measured; deployment transcripts are treated as observations; values retained only in project-conversation records are treated as reconstructed; proposed but unrecovered experiments are treated as planned; and published results from other groups are treated as external evidence. This distinction is maintained throughout the paper to avoid converting design intent into achieved results.

Table 1 | Evidence classes used in this study.

2.1 Hardware objective

The later Antares design targeted a single NVIDIA GeForce RTX 5070 with 12 GB of VRAM. The objective was end-to-end pretraining throughput and time-to-quality, not inference latency. Pre-run estimates were used to select architecture and sequence-length candidates, but unverified throughput and memory estimates are excluded from the main result tables.

3. Model and Pretraining Design

3.1 Final architecture

Table 2 | Antares architecture specification (Artifact A10).

The preserved probe logs independently confirm the checkpoint depth, model width, and vocabulary size for outputs\antares_16l_onepass_adamw_b12\latest.pt (Artifact A4). The 108.55M parameter value is taken from the architecture specification; a parameter-sum printout for the final checkpoint was not recovered in the current artifact set.

3.2 Data plan

The main Antares plan used a 2.5B-token target mixture combining educational web text, encyclopedic text, synthetic educational material, code, and general web text. The mixture in Table 3 records the intended composition rather than a reconstructed final exposure ledger.

Table 3 | Planned 2.5B-token pretraining mixture.

Project conversation notes later referred to approximately 2.35B pretraining tokens. Because a complete end-to-end token ledger was not recovered, the exact final exposure remains unresolved and is not used as a primary measured quantity in this paper.

3.3 Architecture selection

Candidate designs included recurrent and state-space architectures, attention-reduction variants, small mixture-of-experts models, multi-token prediction, shared-depth schemes, and low-precision training. The selection criterion required evidence of improved quality per wall-clock hour on the target hardware. Methods with favorable theoretical FLOP counts but uncertain kernel efficiency, additional routing overhead, or insufficient equal-wall-clock evidence were deferred. This process favored a dense Transformer because of its predictable implementation, mature attention kernels, and straightforward checkpoint semantics (Artifact A10).

4. Preliminary External-Memory Experiments

4.1 Synthetic associative-recall task

Before the final Antares architecture was fixed, a series of hybrid models augmented a Transformer with an explicit external-memory pathway. The synthetic benchmark embedded key/value associations across conversation-like histories. Typical variants used 16 candidate keys and 16 values, with chance accuracy in the range of approximately 6.25–12.5% depending on the task configuration. The benchmark was intended as an inexpensive architecture filter and should not be interpreted as a natural-language long-context evaluation.

Approximately six memory variants were explored, including learned reads and writes, repeated reads, delta-style updates, separate address and content representations, normalized cosine addresses, straight-through single-slot writes, and anti-collapse regularization. An oracle-routing diagnostic was subsequently introduced to separate memory-content limitations from learned-addressing failures. The raw output files for these experiments were not recovered; values in this section are therefore reconstructed from the project conversation record (Artifact A1).

4.2 Matched-token and matched-time results

Table 4 | Reconstructed associative-recall comparison between the Transformer baseline and Hybrid v3 (Artifact A1).

Figure 1 | Final associative-recall accuracy for the reconstructed Transformer and Hybrid-v3 comparisons. The larger matched-token gain indicates improved sample efficiency, whereas lower throughput reduces the wall-clock benefit.

The hybrid model produced a larger advantage when training exposure was matched by tokens than when it was matched by elapsed time. The result therefore supports a narrow interpretation: the explicit-memory pathway improved sample efficiency on the synthetic task, but the additional computation reduced its practical advantage under the wall-clock objective.

4.3 Router refinement and failure mode

Table 5 | Reconstructed matched-token result and approximate throughput for the v6 router experiment (Artifact A1).

The v6 router provided only a modest matched-token improvement over v3 while further reducing throughput. Long-history diagnostics also exhibited severe address collisions: approximately 14–15 collisions among 16 addresses in the recorded 32-turn configuration, with route entropy near 1.0 and weak oracle-router accuracy. A separate ablation produced an approximately 26-percentage-point degradation when the memory signal was removed, indicating that the model was using the memory pathway even though routing remained unreliable. These findings motivated a return to a dense Transformer for Antares v0.

5. Optimization Experiments

5.1 AdamW versus Muon

The strongest locally measured pretraining-efficiency result is a short optimizer comparison on a four-layer Antares pilot containing 36,576,000 parameters. The training loop processed 16,384 tokens per optimizer step and sustained approximately 100–105k tokens/s in the recovered records. AdamW and Muon were evaluated using elapsed time, processed tokens, and periodic validation cross-entropy (Artifacts A2–A3).

Table 6 | Measured early-training optimizer comparison.

Figure 2 | Validation cross-entropy as a function of processed tokens in the recovered AdamW and Muon pilot runs. Muon is lower at each shared validation checkpoint. The experiment is short and single-run; it does not establish long-horizon superiority.

Muon reached lower validation cross-entropy at all recovered matched-token checkpoints and did so within a shorter recorded run. The experiment therefore provides evidence that optimizer choice materially affected early learning in this regime. It does not support a universal speedup claim: no multi-seed replication was performed, the comparison is short relative to the intended full pretraining horizon, and the final preserved base checkpoint was not produced by this Muon run.

5.2 Progressive depth and context curricula

A separate design track proposed whole-stack depth growth. Under the planned operator, a trained four-layer sequence [A,B,C,D] would be expanded to 16 layers by repeating the full stack four times. Published work on depthwise stacking and masked structural growth provided external motivation for this strategy [2,3]. The Antares design also considered staged sequence-length growth to reduce early attention cost. However, no recovered artifact provides a clean local fixed-16 versus 4->16 control. These techniques are therefore recorded as planned experiments rather than achieved Antares improvements.

The preserved checkpoint path includes antares_16l_onepass_adamw_b12, indicating a one-pass 16-layer AdamW branch. Consequently, neither progressive depth nor Muon should be attributed to the provenance of the preserved base weights.

6. Base-Model Capability Probes

6.1 Checkpoint provenance

The pre-SFT base checkpoint was preserved separately and hash-recorded as SHA-256 745547A1A944244387BF068EEA38394C206976EB078BF2723A62056E69CAF4DB (Artifact A5). The probe results below were obtained from the 16-layer, width-768, 16,384-vocabulary checkpoint before supervised post-training (Artifact A4).

6.2 Lexical and factual next-token probes

Teacher-forced next-token probabilities were used in addition to free-form generation. This analysis distinguishes a correct association that is represented in the distribution from an answer that happens to be emitted under sampling. Selected results are shown in Table 7.

Table 7 | Selected pre-SFT next-token probes (Artifact A4).

The results show substantial heterogeneity. Common fixed expressions and some directional factual relations are strong, whereas reverse factual prompts are markedly weaker. Multi-token entities also exhibit a first-subtoken bottleneck: once the initial token of “Jupiter” is selected, the remaining subtokens become nearly deterministic. These patterns distinguish representation strength from prompt-dependent accessibility and argue against treating isolated free-generation successes as calibrated factual competence.

6.3 Code-completion probes

Table 8 | Selected pre-SFT code-completion probes (Artifact A4).

The model learned several local Python syntax and idiom patterns, including simple function completion, recursion syntax, list-comprehension structure, and a common binary-search update. Cross-line variable binding and higher-order idioms were substantially weaker. These probes indicate that code structure was present in pretraining without establishing general programming or debugging competence.

7. Supervised Post-Training

7.1 Training format

The post-training pipeline used explicit special tokens for system, user, assistant, tool-call, and tool-result messages. Supervision was assistant-only: system text, user text, and tool results were masked from the language-model loss, while assistant tokens contributed to the objective. This choice reduces direct pressure to reproduce user or runtime context as assistant output and is consistent with causal-LM instruction-tuning practice.

7.2 Dataset inventory

Table 9 | Uploaded post-training corpora.

The corpus is large enough to impose repeated formatting and behavioral patterns but small relative to pretraining. Its most defensible role is behavioral adaptation rather than acquisition of broad factual knowledge. The tool-only data also contain synthetic tool traces, including fabricated entities and example-domain results. Such examples can teach schema and control flow but do not constitute factual grounding data.

7.3 Five-level reasoning-mode dataset

A separate 1,000-example dataset attempted to train five reasoning modes—instant, medium, high, xhigh, and max—under fixed, automatically selected, user-selected, and auto-disabled policies. The intended output format used explicit reasoning and final-answer markers. The design was not retained as evidence of improved reasoning because manual inspection and lexical audit identified extensive template repetition across otherwise unrelated tasks.

Table 10 | Distribution of selected reasoning modes and mode-policy conditions.

Table 11 | Repeated markers in the 1,000 reasoning targets.

The frequency of repeated metacognitive scaffolding is not, by itself, proof that each example is invalid. It does, however, create a substantial confound for a 108M-parameter model: successful imitation of recurring “verification” or “correction” language could be misinterpreted as improved underlying reasoning. For this reason, the dataset is documented as an abandoned post-training experiment rather than as support for a reasoning-capability claim.

8. Deployment Evaluation and Runtime Attribution

8.1 Observed post-SFT failures

Deployment traces show that assistant-format behavior did not imply robust factual or instructional competence. Examples included an incorrect definition of pi, failure to continue the decimal prefix 3.1415, unsupported political claims after a correction cue, and first-person participation in historical events. These outputs are treated as observations rather than causal diagnoses because multiple layers—base model, SFT data, system prompts, sampling, and runtime tool context—can affect the final text.

Table 12 | Runtime-aware taxonomy of selected deployment failures.

8.2 Deterministic tool routing in the current server

Inspection of the current serving implementation changes the interpretation of visible tool behavior (Artifact A7). Pure arithmetic is recognized by deterministic regular expressions and evaluated by a restricted Python AST calculator; the numeric result is returned directly without requiring the model to select or reproduce the tool result. Similarly, messages containing freshness cues such as “current”, “latest”, or “today”, and queries about dynamic offices such as president or CEO, can trigger a forced web search before model generation.

The web result is serialized into a <|tool_result|> JSON payload and appended to the prompt. Only after these deterministic checks can the model independently emit a tool-call token. Consequently, an interface event such as “Used web” is not sufficient evidence that Antares selected the web tool. Evaluation must distinguish at least two variables: the source of the route decision and the source of the final answer.

One historical arithmetic trace is particularly useful for provenance analysis. A deployment configuration answered “What is 1821 + 12831?” with an unknown-term fallback rather than 14652. The current server would deterministically invoke the calculator and return the exact result. The failure therefore belongs to an earlier or different serving path and cannot be used to characterize the current calculator route.

9. Zero-Shot Benchmark Comparison with GPT-2 Small

The preserved Antares Base and Antares v0 checkpoints were evaluated against GPT-2 Small using the same lm-evaluation-harness tasks under raw causal-likelihood scoring. The run was zero-shot and used no chat template, no system prompt, and no tools. This removes the deployment/runtime attribution confounds discussed in Section 8 and permits a direct model-level comparison on the selected tasks.

Table 15. Full zero-shot benchmark results. The headline mean uses HellaSwag acc_norm, PIQA acc_norm, ARC-Easy acc_norm, WinoGrande acc, and OpenBookQA acc_norm.

Antares v0 reaches a 42.37% five-task mean versus 42.47% for GPT-2 Small: a descriptive gap of only 0.10 percentage points while Antares uses 15,886,848 fewer parameters (12.8% fewer). Antares v0 exceeds GPT-2 Small on ARC-Easy and OpenBookQA, is close on WinoGrande, and trails on HellaSwag and PIQA. This result supports a narrow claim of GPT-2-Small-class performance on this benchmark aggregate; it is not a claim of general capability equivalence.

9.1 Effect of supervised fine-tuning

Antares v0 is the SFT-trained continuation of the preserved Antares Base checkpoint. The direct pre/post comparison therefore measures the change associated with SFT for this lineage. The headline mean rises from 39.15% to 42.37%, a gain of 3.22 percentage points, and every included task improves.

Table 16. Antares Base to v0 change under identical benchmark conditions.

The SFT gain closes approximately 97% of the original 3.32-point mean gap between Antares Base and GPT-2 Small. Because the benchmark bypasses chat formatting and tools, the change is not merely a nicer response surface: SFT altered raw continuation likelihoods in a direction that improved these multiple-choice tasks. The current report does not isolate which SFT subsets caused the gain, so no category-level causal claim is made.

9.2 Interpretation and statistical caution

The five-task mean is a compact engineering summary rather than a formal statistical equivalence test. OpenBookQA contains only 500 items, and small task-level differences can be noisy. Future work should preserve item-level outputs and report paired bootstrap intervals. The robust conclusions supported by the current run are the exact observed scores, the positive Base-to-v0 movement on all five tasks, and the near-equality of the descriptive aggregate to GPT-2 Small under the stated harness.

10. Discussion

10.1 Sample efficiency and wall-clock efficiency are distinct objectives

The external-memory and optimizer experiments illustrate complementary cases. The hybrid memory model improved associative recall per token but lost part of that gain through slower steps. Muon, in contrast, improved validation cross-entropy sufficiently in the recovered pilot to improve both token efficiency and elapsed-time progress. These results support direct measurement of time-to-target rather than reliance on tokens processed or theoretical FLOPs as proxies for practical training efficiency.

10.2 Representation, retrieval, and instruction-level use are separable

The base probes show that a relation may be strongly encoded in one prompt direction and weakly accessible in another. The model strongly predicts “France” after “Paris is the capital of” but assigns substantially less probability to “Paris” in the reverse formulation. Likewise, continuation of a multi-token entity can become nearly deterministic after the first subtoken has been selected. For small models, it is therefore useful to distinguish (i) representation of an association, (ii) prompt-dependent accessibility, and (iii) reliable use under an instruction or dialogue format.

10.3 Small-data SFT has high behavioral leverage

The post-training corpora are small relative to pretraining but highly structured and repetitive. Such data can efficiently impose response formats, tool schemas, system-instruction conventions, and fallback phrases. The same leverage increases sensitivity to synthetic-data artifacts: repeated metacognitive templates or narrow fallback responses may become disproportionately salient. At approximately 100M parameters, dataset diversity and applicability conditions are therefore first-order design variables rather than cosmetic curation details.

10.4 Runtime orchestration can obscure model capability

Deterministic routing is a reasonable production strategy for a small model, particularly for exact arithmetic and freshness-sensitive facts. It nevertheless changes the object being measured. A benchmark that attributes a forced web call or runtime-computed arithmetic answer to the neural model evaluates the orchestration layer rather than tool-selection competence. Future evaluation should log route_source = model|runtime and answer_source = model|runtime, or equivalent provenance fields, for each tool-mediated response.

10.5 The preserved checkpoint does not incorporate every promising ablation

The recovered evidence indicates that Muon outperformed AdamW during the short pilot and that progressive depth had substantial external support, yet the preserved final base checkpoint is associated with a one-pass AdamW branch. The available artifacts do not establish why the better-performing or better-motivated alternatives were not incorporated into that run. The scientifically appropriate conclusion is prospective: these methods remain candidates for controlled replication in a subsequent Antares training run, not retrospective properties of the existing checkpoint.

13. Recommended Experimental Controls for a Subsequent Run

• Replicate the AdamW–Muon comparison for multiple random seeds and extend it far enough to compare time-to-fixed-validation-loss rather than only early checkpoints.
• Run a controlled fixed-depth versus 4->16 depth-growth experiment with identical data order, tokenizer, optimizer family, effective batch size, and validation schedule.
• Preserve a machine-readable training ledger containing cumulative tokens, elapsed wall time, sequence length, learning rate, validation metrics, and checkpoint hash at every evaluation point.
• Evaluate SFT by held-out behavioral tasks rather than by stylistic resemblance to the supervision. Separate tool routing, tool-argument correctness, tool-result grounding, and final-answer correctness.
• Replace templated reasoning-mode supervision with outcome-validated task traces only if a reasoning-control experiment is repeated. Report improvements on task accuracy rather than on production of reasoning markers.
• Evaluate both teacher-forced probability probes and free generation. The former detects weak but real representations; the latter exposes calibration, instruction-following, and compounding-generation failures.
• Record runtime provenance for every tool-mediated evaluation so that deterministic orchestration cannot be misattributed to the model.

13. Limitations

The study is limited by incomplete experimental preservation. The external-memory results are reconstructed from project conversation records because the raw v1–v6 output artifact is unavailable. The optimizer comparison is short and apparently single-seed. Although the final base checkpoint is preserved and hash-identified, the recovered artifacts do not provide a complete end-to-end ledger of final token exposure, total wall-clock training time, every sequence-length transition, and every validation checkpoint.

No clean local progressive-depth control was recovered, so reported benefits of model growth are drawn only from external literature and are not attributed to Antares. The base-model probes are diagnostic prompts rather than a standardized public benchmark suite and should not be aggregated into a general capability score. Deployment traces likewise span more than one serving configuration; the current API establishes present routing semantics but cannot reconstruct the exact runtime used for every historical response.

Finally, the external-memory benchmark measures synthetic associative recall. Its throughput and routing results do not establish the behavior of external-memory architectures on natural-language long-context tasks, larger models, or different hardware.

13. Conclusion

Antares provides a compact case study of scratch language-model development under a strict consumer-hardware iteration constraint. The external-memory experiments show that improved sample efficiency does not necessarily translate into improved wall-clock efficiency. The Muon pilot indicates that optimizer choice can materially affect early time-to-loss in this scale regime, while also illustrating the need for longer, replicated comparisons. The preserved 108M base model contains measurable lexical, factual, arithmetic, and code associations, but these representations are unevenly accessible and do not yield reliable instruction-level behavior.

Post-training further demonstrates that small supervised datasets can strongly shape response format and interaction protocols without supplying robust factual grounding or general reasoning. The rejected five-level reasoning-mode dataset illustrates the corresponding methodological risk: repeated synthetic reasoning templates can create the appearance of structured deliberation without evidence of improved task performance. Finally, deterministic runtime routing shows that product-level tool behavior must be separated from neural-model capability during evaluation.

The principal outcome of the Antares work is therefore methodological rather than benchmark-based. For small scratch-trained language models, direct wall-clock measurement, explicit evidence provenance, careful SFT-data auditing, and runtime attribution are necessary to determine which apparent improvements belong to the model itself and which arise from training or orchestration artifacts.

References

[1] Hoffmann, J. et al. (2022). Training Compute-Optimal Large Language Models. Advances in Neural Information Processing Systems 35.

[2] Du, W., Luo, T., Qiu, Z., Huang, Z., Shen, Y., Cheng, R., Guo, Y., & Fu, J. (2024). Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training. NeurIPS 2024. arXiv:2405.15319.

[3] Yao, Y., Zhang, Z., Li, J., & Wang, Y. (2024). Masked Structural Growth for 2x Faster Language Model Pre-training. ICLR 2024. arXiv:2305.02869.

[4] Liu, J. et al. (Moonshot AI). (2025). Muon Is Scalable for LLM Training. Technical report.

[5] Li, Z., Liu, L., Liang, C., Chen, W., & Zhao, T. (2025). NorMuon: Making Muon More Efficient and Scalable. arXiv:2510.05491.

[6] Li, J., Fan, Y., & Li, H. (2026). Variance-Adaptive Muon: Accelerating LLM Pretraining with NSR-Modulated and Variance-Scaled Momentum. arXiv:2601.14603.

Project artifacts

[A1] SliceLine project conversation record: external-memory v1–v6 synthetic key/value recall experiments, August 2026. Values in Section 4 are reconstructed; raw output file not recovered.

[A2] Antares AdamW 10M-token pilot record, August 20, 2026: 100.14 s; validation checkpoints through 9.83M tokens; checkpoint outputs/bakeoff_adamw_10m/final.pt.

[A3] Antares Muon 7.21M-token pilot record, August 20, 2026: 86.83 s; validation checkpoints through 6.55M tokens; checkpoint outputs/bakeoff_muon_7p2m/final.pt.

[A4] scripts/15_probe_logits.py output for outputs/antares_16l_onepass_adamw_b12/latest.pt, August 20–21, 2026.

[A5] Frozen pre-SFT base-checkpoint hash record: SHA-256 745547A1A944244387BF068EEA38394C206976EB078BF2723A62056E69CAF4DB.

[A6] Post-training corpora: generated(4).jsonl, generated_plain_chat(1).jsonl, generated_tools_only(1).jsonl, antares_system_following_gold_1000_messages(1).jsonl, and antares_thinking_modes_gold_1000(1).jsonl.

[A7] Current serving implementation: model_api_antares(2).py, including deterministic arithmetic and freshness-sensitive routing.

[A8] Recorded Antares deployment transcripts, August 21, 2026.

[A9] SliceLine internal deep-research-report.md, early small-LM training/data/SFT planning.

[A10] Antares architecture and efficient-pretraining research notes, August 2026, including the 108.55M architecture specification, progressive-growth plan, context curriculum, optimizer survey, and low-precision analysis.

Appendix A. Evidence Ledger

Table A1 | Claim-level evidence status.

Appendix B. SFT Mixture Snapshot

The broad mixed SFT file and tool-only file contain the following recorded category counts. These distributions characterize post-training exposure; category labels are dataset metadata rather than externally standardized task definitions.

Table B1 | Largest categories in `generated(4).jsonl`.

Table B2 | Category counts in `generated_tools_only(1).jsonl`.

Appendix C. Current Runtime Routing Semantics

The serving implementation is part of the evaluation environment because several decisions occur outside the model. In simplified form:

• Pure arithmetic and direct calculate/compute/evaluate requests containing only supported arithmetic characters are evaluated by a restricted AST calculator, and the numeric result is returned directly.
• Queries containing freshness terms such as current/latest/today/now, or dynamic-role patterns such as president/prime minister/CEO, can trigger a forced web query.
• Web results are serialized into a <|tool_result|> JSON payload before the next assistant generation.
• Only after deterministic routing can the model independently emit a <|tool_call|> token and JSON request.
• The default chat system prompt identifies the model as Antares and requests direct answers without invented facts; forced tool routes use narrower tool-specific system prompts.

These semantics define the unit of evaluation. A response generated after a runtime-forced tool call can measure tool-result use and answer synthesis, but it cannot by itself measure model-level tool selection.