tiktoken pypi
tiktoken is a fast BPE tokeniser for use with OpenAI's models
Releases in the last 7 days (0)
No release in the last 7 days. Latest known version: 0.14.0.
tiktoken is a fast BPE tokeniser for use with OpenAI's models
No release in the last 7 days. Latest known version: 0.14.0.