Tokens are occurrences of distinctly meaningful lexical items that are not categorized into types/lexemes (usually under inflection).
Some tokens consist of more than one word. In English, these include verbs whose meanings are altered with adverbial particles (a.k.a., "phrasal verbs"), multi-word prepositions, and the like:
Here are some examples of the aforementioned:
- "taking up" (every inflection of the type "take up" is a separate token),
- "in regards to" (functionally, a preposition, though it consists of three words),
- "United States" (a proper noun, though it consists of two words),
- "land mines" (a multi-word term that is often compounded as "landmines").
In contrast, most words, as types/lexemes, contain several tokens, mainly because of inflection. Most single-word types/lexemes of an open word class will have more than one token.
Here are some examples:
- In English, "be", "is", "am", "were", etc. are all tokens of the type "be".
- In Spanish, "me fui", "te irás", "nos íbamos", etc. are all tokens of the type "irse".
- In Mandarin, "朋友們" and "朋友" are tokens of the type "朋友".