| Port details |
- py-tokenizers Fast state-of-the-art tokenizers optimized for research and production
- 0.23.2 textproc
 =0      0.23.1_2Version of this port present on the latest quarterly branch. - Maintainer: tagattie@FreeBSD.org
- Port Added: 2024-02-12 08:36:07
- Last Update: 2026-09-04 01:16:10
- Commit Hash: e65a2c4
- Also Listed In: python
- License: APACHE20
- WWW:
- https://github.com/huggingface/tokenizers
- Description:
- Provides an implementation of today's most used tokenizers, with a
focus on performance and versatility.
Main features:
- Train new vocabularies and tokenize, using today's most used
tokenizers.
- Extremely fast (both training and tokenization), thanks to the Rust
implementation. Takes less than 20 seconds to tokenize a GB of text
on a server's CPU.
- Easy to use, but also extremely versatile.
- Designed for research and production.
- Normalization comes with alignments tracking. It's always possible
to get the part of the original sentence that corresponds to a given
token.
- Does all the pre-processing: Truncate, Pad, add the special tokens
your model needs.
  ¦  ¦  ¦  ¦ 
- Manual pages:
- FreshPorts has no man page information for this port.
- pkg-plist: as obtained via:
make generate-plist - There is no configure plist information for this port.
- USE_RC_SUBR (Service Scripts)
- no SUBR information found for this port
- Dependency lines:
-
- ${PYTHON_PKGNAMEPREFIX}tokenizers>0:textproc/py-tokenizers@${PY_FLAVOR}
- To install the port:
- cd /usr/ports/textproc/py-tokenizers/ && make install clean
- To add the package, run one of these commands:
- pkg install textproc/py-tokenizers
- pkg install py312-tokenizers
NOTE: If this package has multiple flavors (see below), then use one of them instead of the name specified above.
NOTE: This is a Python port. Instead of py312-tokenizers listed in the above command, you can pick from the names under the Packages section.- PKGNAME: py312-tokenizers
- Package flavors (<flavor>: <package>)
- distinfo:
- TIMESTAMP = 1788469406
SHA256 (tokenizers-0.23.2.tar.gz) = 7f0f085686b9de0d0079e6f874ae053600db64c5d13049e0bbc0119926d25aac
SIZE (tokenizers-0.23.2.tar.gz) = 385745
Packages (timestamps in pop-ups are UTC):
- Dependencies
- NOTE: FreshPorts displays only information on required and default dependencies. Optional dependencies are not covered.
- Build dependencies:
-
- py312-maturin>=1.0<2.0 : devel/py-maturin@py312
- rust>=1.97.0 : lang/rust
- pkgconf>=1.3.0_1 : devel/pkgconf
- python3.12 : lang/python312
- py312-build>=0 : devel/py-build@py312
- py312-installer>=0 : devel/py-installer@py312
- Test dependencies:
-
- py312-requests>0 : www/py-requests@py312
- py312-numpy>0 : math/py-numpy@py312
- py312-datasets>0 : misc/py-datasets@py312
- py312-pytest-benchmark>0 : devel/py-pytest-benchmark@py312
- py312-pytest>=7,1 : devel/py-pytest@py312
- python3.12 : lang/python312
- Runtime dependencies:
-
- py312-huggingface-hub>=0.16.4<2.0 : misc/py-huggingface-hub@py312
- python3.12 : lang/python312
- Library dependencies:
-
- libonig.so : devel/oniguruma
- This port is required by:
- for Run
-
- databases/py-chromadb
- misc/colibri
- misc/comfyui
- misc/py-cohere
- misc/py-crewai
- misc/py-langchain-huggingface
- misc/py-litellm
-
Deleted ports which required this port:
- * - deleted ports are only shown under the This port is required by section. It was harder to do for the Required section. Perhaps later...
Configuration Options:- No options to configure
- Options name:
- textproc_py-tokenizers
- USES:
- cargo python
- FreshPorts was unable to extract/find any pkg message
- Master Sites:
|
| Commit History - (may be incomplete: for full details, see links to repositories near top of page) |
| Commit | Credits | Log message |
|---|
0.23.2
04 Sep 2026 01:16:10
 |
Hiroki Tagato (tagattie) |
textproc/py-tokenizers: Update to 0.23.2
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.23.2
Reported by: Repology |
0.23.1_3
21 Jul 2026 06:47:24
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.97.1
PR: 296645 |
0.23.1_2
04 Jun 2026 08:52:51
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.96.0
PR: 295694 |
0.23.1_1
05 May 2026 19:29:37
|
Max Brazhnikov (makc) |
*/*: bump PORTREVISION after switching to NumPy 2.x |
0.23.1
05 May 2026 19:29:37
 |
Max Brazhnikov (makc) |
*/*: Switch to NumPy 2.x
PR: 294328
Exp-run: antoine |
0.23.1
30 Apr 2026 11:30:30
|
Hiroki Tagato (tagattie) |
textproc/py-tokenizers: Update to 0.23.1
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.23.1
Reported by: Repology |
0.22.2_3
27 Apr 2026 12:26:47
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.95.0
PR: 294615 |
0.22.2_2
03 Apr 2026 18:52:22
|
Charlie Li (vishwin) |
math/py-numpy: switch all consumers to math/py-numpy1
...and bump PORTREVISIONs of consumers with hard RUN_DEPENDS
PR: 281470 |
0.22.2_2
23 Mar 2026 07:33:11
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.94.0
PR: 293615 |
0.22.2_1
28 Jan 2026 08:48:51
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.93.0
PR: 292360 |
0.22.2
06 Jan 2026 08:20:50
|
Hiroki Tagato (tagattie) |
textproc/py-tokenizers: Update to 0.22.2
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.22.2
Reported by: portscout |
0.22.1_3
16 Dec 2025 12:12:31
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.92.0
PR: 291582 |
0.22.1_2
11 Nov 2025 11:08:12
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.91.1
PR: 290816 |
0.22.1_1
03 Oct 2025 08:16:50
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.90.0
PR: 289709 |
0.22.1
22 Sep 2025 07:13:25
|
Hiroki Tagato (tagattie) |
textproc/py-tokenizers: update to 0.22.1
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.22.1
Reported by: Repology |
0.22.0_1
01 Sep 2025 08:25:04
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.89.0
PR: 288923 |
0.22.0
31 Aug 2025 01:40:55
|
Hiroki Tagato (tagattie) |
textproc/py-tokenizers: update to 0.22.0
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.22.0
Reported by: portscout |
0.21.4
05 Aug 2025 06:43:18
|
Hiroki Tagato (tagattie) |
textproc/py-tokenizers: update to 0.21.4
Changelog:
- https://github.com/huggingface/tokenizers/releases/tag/v0.21.3
- https://github.com/huggingface/tokenizers/releases/tag/v0.21.4
Reported by: Repology |
0.21.2_1
03 Jul 2025 08:46:01
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.88.0
PR: 287766 |
0.21.2
26 Jun 2025 09:16:07
|
Hiroki Tagato (tagattie) |
textproc/py-tokenizers: update to 0.21.2
While here, refactor test-related part so that it can execute both
python and rust tests.
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.21.2
Reported by: Repology |
0.21.1_2
05 Jun 2025 07:52:53
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.87.0
PR: 286829 |
0.21.1_1
08 Apr 2025 08:41:13
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.86.0
PR: 285840 |
0.21.1
22 Mar 2025 08:35:24
|
Hiroki Tagato (tagattie) |
textproc/py-tokenizers: update to 0.21.1
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.21.1
Reported by: portscout |
0.21.0_2
24 Feb 2025 07:55:52
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.85.0
PR: 284884 |
0.21.0_1
20 Jan 2025 11:06:50
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.84.0
PR: 283962 |
0.21.0
06 Dec 2024 02:24:32
|
Hiroki Tagato (tagattie) |
textproc/py-tokenizers: update to 0.21.0
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.21.0
Reported by: Repology |
0.20.3_2
01 Dec 2024 09:24:18
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.83.0
PR: 283000 |
0.20.3_1
08 Nov 2024 08:24:20
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.82.0
PR: 282516 |
0.20.3
07 Nov 2024 12:30:23
|
Hiroki Tagato (tagattie) |
textproc/py-tokenizers: update to 0.20.3
Changelog:
- https://github.com/huggingface/tokenizers/releases/tag/v0.20.2
- https://github.com/huggingface/tokenizers/releases/tag/v0.20.3
Reported by: portscout |
0.20.1
18 Oct 2024 05:12:30
|
Hiroki Tagato (tagattie) |
textproc/py-tokenizers: update to 0.20.1
While here, add LICENSE_FILE.
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.20.1
Reported by: Repology |
0.20.0_2
10 Sep 2024 11:00:34
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.81.0
PR: 281300 |
0.20.0_1
10 Sep 2024 10:58:07
|
Mikael Urankar (mikael)
Author: Siva Mahadevan |
*/*: remove STRIP_CMD calls in rust based ports
This is not needed after bc4fedc1fec0d359365c04d43be9e32bf101a50e
PR: 246993
Differential Revision: https://reviews.freebsd.org/D46503 |
0.20.0_1
26 Aug 2024 08:08:01
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.80.1
PR: 280490 |
0.20.0
10 Aug 2024 22:35:02
|
Hiroki Tagato (tagattie) |
textproc/py-tokenizers: update to 0.20.0
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.20.0
Reported by: portscout |
0.19.1_2
18 Jun 2024 10:59:14
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.79.0
PR: 279707 |
0.19.1_1
13 May 2024 11:03:24
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.78.0
PR: 278834 |
0.19.1
21 Apr 2024 08:18:00
|
Hiroki Tagato (tagattie) |
textproc/py-tokenizers: update to 0.19.1
Changelog:
- https://github.com/huggingface/tokenizers/releases/tag/v0.19.0
- https://github.com/huggingface/tokenizers/releases/tag/v0.19.1
Reported by: Repology |
0.15.2_2
23 Mar 2024 09:41:46
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.77.0
PR: 277786 |
0.15.2_1
19 Feb 2024 11:59:23
|
Mikael Urankar (mikael) |
lang/rust: Bump revisions after 1.76.0
PR: 276920 |
0.15.2
14 Feb 2024 09:17:15
|
Hiroki Tagato (tagattie) |
textproc/py-tokenizers: update to 0.15.2
While here, enable tests.
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.15.2
Reported by: portscout |
0.15.1
12 Feb 2024 08:34:14
|
Hiroki Tagato (tagattie) |
textproc/py-tokenizers: add port: Fast state-of-the-art tokenizers optimized for
research and production
Provides an implementation of today's most used tokenizers, with a
focus on performance and versatility.
Main features:
- Train new vocabularies and tokenize, using today's most used
tokenizers.
- Extremely fast (both training and tokenization), thanks to the Rust
implementation. Takes less than 20 seconds to tokenize a GB of text
on a server's CPU.
- Easy to use, but also extremely versatile.
- Designed for research and production.
- Normalization comes with alignments tracking. It's always possible
to get the part of the original sentence that corresponds to a given
token.
- Does all the pre-processing: Truncate, Pad, add the special tokens
your model needs.
WWW: https://github.com/huggingface/tokenizers |
As an Amazon Associate I earn from qualifying purchases.