Global Rank#4821Aug 21, 2026
Global Rank#4821Aug 21, 2026
since Nov 2019
huggingface/tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
RustApache-2.0
11k
1.2k
150
Weekly
0stars
·2pushes
Stars
Contributors
New Stars
New Pushes
Forks
star-history.com
Trended 99 days · best rank #2 on Apr 25, 2024
May 13, 2026Rust#14
May 12, 2026Rust#7
Oct 18, 2025Rust#3
Oct 16, 2025Rust#22
Sep 14, 2025Rust#13
Jul 23, 2025Rust#23
Jul 14, 2025Rust#10
Jul 13, 2025Rust#15
Jun 20, 2025Rust#6
Jun 11, 2025Rust#10
May 30, 2025Rust#18
May 28, 2025Rust#7
May 22, 2025Rust#4
Apr 11, 2025Rust#2
Feb 1, 2025Rust#24
Jan 28, 2025Rust#21
Dec 27, 2024Rust#7
Dec 26, 2024Rust#22
Dec 12, 2024Rust#11
Dec 10, 2024Rust#15
Oct 28, 2024Rust#23
Oct 9, 2024Rust#24
Oct 5, 2024Rust#23
Aug 24, 2024Rust#19
Aug 14, 2024Rust#21
Jun 19, 2024Rust#13
Jun 17, 2024Rust#21
Apr 26, 2024Rust#5
Apr 25, 2024Rust#2
Apr 8, 2024Rust#19
Mar 16, 2024Rust#12
Jan 17, 2024Rust#24
Dec 7, 2023Rust#13
Oct 19, 2023Rust#15
Sep 22, 2023Rust#16
Sep 21, 2023Rust#16
Sep 19, 2023Rust#22
Aug 4, 2023Rust#25
Aug 3, 2023Rust#13
Aug 2, 2023Rust#13
Jul 28, 2023Rust#21
Jun 25, 2023Rust#18
Jun 21, 2023Rust#20
Jun 20, 2023Rust#24
May 26, 2023Rust#10
May 15, 2023Rust#24
Apr 7, 2023Rust#16
Apr 6, 2023Rust#21
Mar 10, 2023Rust#21
Mar 9, 2023Rust#22
Jan 10, 2023Rust#18
Dec 28, 2022Rust#13
Dec 27, 2022Rust#21
Nov 4, 2022Rust#20
Nov 3, 2022Rust#21
Sep 23, 2022Rust#20
Sep 22, 2022Rust#3
Sep 20, 2022Rust#9
Jun 17, 2022Rust#15
May 14, 2022Rust#5
May 13, 2022Rust#5
May 12, 2022Rust#5
Apr 2, 2022Rust#15
Apr 1, 2022Rust#13
Mar 31, 2022Rust#10
Mar 30, 2022Rust#24
Mar 29, 2022Rust#3
Mar 17, 2022Rust#22
Mar 1, 2022Rust#23
Feb 19, 2022Rust#6
Feb 18, 2022Rust#3
Feb 17, 2022Rust#7
Dec 25, 2021Rust#16
Dec 4, 2021Rust#20
Nov 11, 2021Rust#7
Nov 10, 2021Rust#18
Sep 2, 2021Rust#25
Sep 1, 2021Rust#23
Jun 11, 2021Rust#16
Jun 10, 2021Rust#6
May 26, 2021Rust#9
May 22, 2021Rust#6
May 21, 2021Rust#8
Apr 2, 2021Rust#21
Apr 1, 2021Rust#8
Mar 31, 2021Rust#12
Mar 24, 2021Rust#10
Mar 19, 2021Rust#20
Mar 13, 2021Rust#17
Mar 12, 2021Rust#23
Feb 26, 2021Rust#20
Feb 25, 2021Rust#16
Feb 17, 2021Rust#8
Feb 6, 2021Rust#6
Feb 5, 2021Rust#14
Jan 20, 2021Rust#22
Jan 8, 2021Rust#5
Jan 7, 2021Rust#9
Jan 6, 2021Rust#11
Both badges
GitHub README
<p align="center"> <a href="https://www.star-history.com/huggingface/tokenizers"> <picture><source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/badge?repo=huggingface/tokenizers&type=rank&theme=dark" /><source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/badge?repo=huggingface/tokenizers&type=rank" /><img alt="Star History Rank" src="https://api.star-history.com/badge?repo=huggingface/tokenizers&type=rank" /></picture> <picture><source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/badge?repo=huggingface/tokenizers&type=trending&theme=dark" /><source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/badge?repo=huggingface/tokenizers&type=trending" /><img alt="GitHub Trending Repository of the Day" src="https://api.star-history.com/badge?repo=huggingface/tokenizers&type=trending" /></picture> </a> </p>
Markdown (Light)
[](https://www.star-history.com/huggingface/tokenizers) [](https://www.star-history.com/huggingface/tokenizers)
Markdown (Dark)
[](https://www.star-history.com/huggingface/tokenizers) [](https://www.star-history.com/huggingface/tokenizers)
HTML (Light)
<p align="center"> <a href="https://www.star-history.com/huggingface/tokenizers"><img src="https://api.star-history.com/badge?repo=huggingface/tokenizers&type=rank" alt="Star History Rank" /> <img src="https://api.star-history.com/badge?repo=huggingface/tokenizers&type=trending" alt="GitHub Trending Repository of the Day" /></a> </p>
HTML (Dark)
<p align="center"> <a href="https://www.star-history.com/huggingface/tokenizers"><img src="https://api.star-history.com/badge?repo=huggingface/tokenizers&type=rank&theme=dark" alt="Star History Rank" /> <img src="https://api.star-history.com/badge?repo=huggingface/tokenizers&type=trending&theme=dark" alt="GitHub Trending Repository of the Day" /></a> </p>
Global Rank
GitHub README
<p align="center"> <a href="https://www.star-history.com/huggingface/tokenizers"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/badge?repo=huggingface/tokenizers&type=rank&theme=dark" /> <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/badge?repo=huggingface/tokenizers&type=rank" /> <img alt="Star History Rank" src="https://api.star-history.com/badge?repo=huggingface/tokenizers&type=rank" /> </picture> </a> </p>
Markdown (Light)
[](https://www.star-history.com/huggingface/tokenizers)
Markdown (Dark)
[](https://www.star-history.com/huggingface/tokenizers)
HTML (Light)
<p align="center"> <a href="https://www.star-history.com/huggingface/tokenizers"><img src="https://api.star-history.com/badge?repo=huggingface/tokenizers&type=rank" alt="Star History Rank" /></a> </p>
HTML (Dark)
<p align="center"> <a href="https://www.star-history.com/huggingface/tokenizers"><img src="https://api.star-history.com/badge?repo=huggingface/tokenizers&type=rank&theme=dark" alt="Star History Rank" /></a> </p>
GitHub Trending Repository of the Day
GitHub README
<p align="center"> <a href="https://www.star-history.com/huggingface/tokenizers"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/badge?repo=huggingface/tokenizers&type=trending&theme=dark" /> <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/badge?repo=huggingface/tokenizers&type=trending" /> <img alt="GitHub Trending Repository of the Day" src="https://api.star-history.com/badge?repo=huggingface/tokenizers&type=trending" /> </picture> </a> </p>
Markdown (Light)
[](https://www.star-history.com/huggingface/tokenizers)
Markdown (Dark)
[](https://www.star-history.com/huggingface/tokenizers)
HTML (Light)
<p align="center"> <a href="https://www.star-history.com/huggingface/tokenizers"><img src="https://api.star-history.com/badge?repo=huggingface/tokenizers&type=trending" alt="GitHub Trending Repository of the Day" /></a> </p>
HTML (Dark)
<p align="center"> <a href="https://www.star-history.com/huggingface/tokenizers"><img src="https://api.star-history.com/badge?repo=huggingface/tokenizers&type=trending&theme=dark" alt="GitHub Trending Repository of the Day" /></a> </p>