Token
In AI, a token is the basic unit of text that a model processes, which can be as short as a character or as long as a word.
How it works
Models break down input text into these tokens to understand and generate language. The total number of tokens used affects both the model's processing cost and its ability to handle long conversations or documents.
Examples
- 1.The phrase 'Hello, world!' might be broken into tokens like ['Hello', ',', ' world', '!'].
- 2.When you hit a usage limit on an AI tool, you are often hitting a maximum token count for your input and output.
- 3.A long document must be 'chunked' into token-sized pieces to fit within a model's context window.
