melise · about

← back to chat

Melise is a hobby project: a complete language model built from scratch — no pretrained weights, no fine-tuned Llama. The tokenizer, model architecture, training loops, RL harness, evals, and this website were all written and run from first principles, and the whole model was trained on a single rented GPU for about the price of a nice pair of shoes. She is small, earnest, and frequently wrong — that's the charm.

the model

Melise is a miniature of Moonshot AI's Kimi K3 architecture (arXiv:2607.24653): a hybrid attention stack where three linear-time Kimi Delta Attention layers alternate with one global gated multi-latent attention layer, position handled by the KDA layers alone (no positional embeddings). Instead of a plain residual stream, every layer reads an attention-weighted mix of all earlier layer outputs (“attention residuals”). The feed-forward layers are a sparse mixture-of-experts: 40 small experts of which 4 fire per token, plus a shared expert — so of her 163M parameters, only about 78M are active on any given token.

parameters
163M total · 78M active/token
width / depth
d_model 512 · 13 attention layers
attention
16 heads · 3:1 KDA:MLA hybrid · NoPE
feed-forward
40 experts, top-4 routed + shared
tokenizer
custom BPE, 8,192 vocab
context
2,048 tokens

the training

Everything ran on a single NVIDIA L4 — one mid-range cloud GPU — over about 11½ days, in three stages:

  1. Pretraining · 2.2B tokens. Next-token prediction over a curated mix: filtered web text (44%), Wikipedia (16%), forty-five public-domain books (15%), code (8%), dialogue (7%), plus math and reference material — 268,500 steps with a warmup-stable-decay learning-rate schedule.
  2. Chat tuning (SFT) · 25,000 steps. Supervised fine-tuning on ~900MB of open conversation data (SmolTalk, Dolly, OASST1, PersonaChat, BlendedSkillTalk) plus a small identity corpus — with 3% pretraining data replayed so she doesn't forget how to read while learning to talk.
  3. Reinforcement learning · 600 GRPO steps. RL on tasks a program can verify: copying text exactly, small arithmetic, parity, letter counting, word counting, fact recall, and reading facts back out of her own context window. Reward climbed to 0.89 out of 1.

Total cost: roughly $230 of on-demand GPU time, zero crashes. She now serves replies from a small CPU container at a stately 3–4 tokens per second.

what to expect

At 163M parameters — roughly ten thousand times smaller than a frontier model — Melise knows her name, the date, some geography, and her party tricks, and answers tersely (ask about France, get “Paris.”). Beyond that she confabulates cheerfully: facts she wasn't drilled on, code, and poetry are all adventures. Treat her as a working demonstration of how language models are made, at a scale one person can build — not as a source of truth.