GitHubVC
Trending Repos
Trending Builders
Portfolio
AttentionVC
Sign In
Back to Trending
Star
huggingface
/
tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
Rust
bert
gpt
language-model
natural-language-processing
natural-language-understanding
+2 more
Stars
10.8k
+1 today
+2 /wk
+5 /mo
Forks
1.1k
Issues
180
Watchers
10.8k
Star History
30d
90d
Repository Info
License
Apache-2.0
Created
Nov 1, 2019
Last push
5/22/2026
Homepage
huggingface.co/docs/tokenizers
Copy Link
Share on X
Open on GitHub
