Estimate how many tokens a piece of text will consume, so you can keep prompts inside the context window and control cost.
不同分词器结果差异较大。
| Item | Formula / Note | Source / Standard |
|---|---|---|
| Context window | the context window is capped in tokens and shared by input and output | Vaswani et al. 2017, attention |
| Granularity | the same text can differ by more than twenty percent across tokenisers | Sennrich et al. 2016, BPE |
| Cost linkage | billing is metered in tokens so estimate error becomes budget error | Kaplan et al. 2020 scaling laws |