The heart of the dice: what cross-entropy per token measures, and where the idea really comes from
Demiton is building a small open kernel, Thorp, and scoring it on one number: cross-entropy per token. Quentin Luck gives you the formula, which is a receipt, and then the idea, which is a king in a forest who could count the leaves on a tree without counting them. What the number measures, what it does not, and why it is the only honest score in the field.
Read post