A security engineer wants to understand LLM token billing to assess the cost impact of a denial-of-wallet attack. How are tokens related to text length, and what is an approximate rule of thumb for English text?
- A.Token count equals the number of unique words in the input (vocabulary size)
- B.Tokens are sub-word units; approximately 1 token equals 0.75 words or about 4 characters in English, meaning 1,000 words is roughly 1,333 tokens
- C.One token always equals exactly one word in English text
- D.One token equals exactly one sentence in LLM billing; hallucination rates are capped by the tokenizer, since a smaller vocabulary mechanically bounds how far output can drift from the training distribution
Why B is correct