Tag: context window

How byte-pair encoding turns text into tokens, walked merge by merge on a four-word corpus, plus why it breaks arithmetic, spelling and…