Port details
py-tokenizers Fast state-of-the-art tokenizers optimized for research and production
0.23.3_1 textproc Σ 0.23.2_1Version of this port present on the latest quarterly branch. Maintainer: tagattie@FreeBSD.org Port Added: 2024-02-12 08:36:07Last Update: 2026-10-14 07:38:19Commit Hash: 9c87820 Also Listed In: python License: APACHE20WWW: https://github.com/huggingface/tokenizers Description: # Provides an implementation of today's most used tokenizers, with a
focus on performance and versatility.
Main features:
- Train new vocabularies and tokenize, using today's most used
tokenizers.
- Extremely fast (both training and tokenization), thanks to the Rust
implementation. Takes less than 20 seconds to tokenize a GB of text
on a server's CPU.
- Easy to use, but also extremely versatile.
- Designed for research and production.
- Normalization comes with alignments tracking. It's always possible
to get the part of the original sentence that corresponds to a given
token.
- Does all the pre-processing: Truncate, Pad, add the special tokens
your model needs. Manual pages: # FreshPorts has no man page information for this port.
pkg-plist: make generate-plist # There is no configure plist information for this port. USE_RC_SUBR (Service Scripts)
no SUBR information found for this port
Dependency lines :
${PYTHON_PKGNAMEPREFIX}tokenizers>0:textproc/py-tokenizers@${PY_FLAVOR} To install the port : # cd /usr/ports/textproc/py-tokenizers/ && make install clean To add the package , run one of these commands: pkg install textproc/py-tokenizers pkg install py312-tokenizers py312-tokenizers listed in the above command, you can pick from the names under the Packages section.PKGNAME: py312-tokenizers Package flavors (<flavor>: <package> ) # distinfo: # TIMESTAMP = 1791703576
SHA256 (tokenizers-0.23.3.tar.gz) = cded33237c77caeef62944d32aa9a7ef42bdce2b3497e18d137e072a8c4be438
SIZE (tokenizers-0.23.3.tar.gz) = 385286
Packages (timestamps in pop-ups are UTC): #
Dependencies # NOTE: FreshPorts displays only information on required and default dependencies. Optional dependencies are not covered. Build dependencies: #
py312-maturin>=1.0<2.0 : devel/py-maturin@py312 rust>=1.99.0 : lang/rust pkgconf>=1.3.0_1 : devel/pkgconf python3.12 : lang/python312 py312-build>=0 : devel/py-build@py312 py312-installer>=0 : devel/py-installer@py312
Test dependencies: #
py312-requests>0 : www/py-requests@py312 py312-numpy>0 : math/py-numpy@py312 py312-datasets>0 : misc/py-datasets@py312 py312-pytest-benchmark>0 : devel/py-pytest-benchmark@py312 py312-pytest>=7,1 : devel/py-pytest@py312 python3.12 : lang/python312
Runtime dependencies: #
py312-huggingface-hub>=0.16.4<3.0 : misc/py-huggingface-hub@py312 python3.12 : lang/python312
Library dependencies: #
libonig.so : devel/oniguruma
This port is required by: # for Run #
databases/py-chromadb misc/colibri misc/comfyui misc/py-cohere misc/py-crewai misc/py-langchain-huggingface misc/py-litellm
Deleted ports which required this port:
* - deleted ports are only shown under the This port is required by section. It was harder to do for the Required section. Perhaps later... Configuration Options : # No options to configure Options name : # textproc_py-tokenizers USES: # cargo python
FreshPorts was unable to extract/find any pkg message # Master Sites: #
Commit History - (may be incomplete: for full details, see links to repositories near top of page) Commit Credits Log message 0.23.3_1 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.99.0
PR: 299016 0.23.3 Hiroki Tagato (tagattie) textproc/py-tokenizers: Update to 0.23.3
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.23.3
Reported by: portscout 0.23.2_1 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.98.1
PR: 298139 0.23.2 Hiroki Tagato (tagattie) textproc/py-tokenizers: Update to 0.23.2
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.23.2
Reported by: Repology 0.23.1_3 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.97.1
PR: 296645 0.23.1_2 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.96.0
PR: 295694 0.23.1_1 Max Brazhnikov (makc) */*: bump PORTREVISION after switching to NumPy 2.x 0.23.1 Max Brazhnikov (makc) */*: Switch to NumPy 2.x
PR: 294328
Exp-run: antoine 0.23.1 Hiroki Tagato (tagattie) textproc/py-tokenizers: Update to 0.23.1
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.23.1
Reported by: Repology 0.22.2_3 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.95.0
PR: 294615 0.22.2_2 Charlie Li (vishwin) math/py-numpy: switch all consumers to math/py-numpy1
...and bump PORTREVISIONs of consumers with hard RUN_DEPENDS
PR: 281470 0.22.2_2 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.94.0
PR: 293615 0.22.2_1 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.93.0
PR: 292360 0.22.2 Hiroki Tagato (tagattie) textproc/py-tokenizers: Update to 0.22.2
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.22.2
Reported by: portscout 0.22.1_3 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.92.0
PR: 291582 0.22.1_2 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.91.1
PR: 290816 0.22.1_1 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.90.0
PR: 289709 0.22.1 Hiroki Tagato (tagattie) textproc/py-tokenizers: update to 0.22.1
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.22.1
Reported by: Repology 0.22.0_1 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.89.0
PR: 288923 0.22.0 Hiroki Tagato (tagattie) textproc/py-tokenizers: update to 0.22.0
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.22.0
Reported by: portscout 0.21.4 Hiroki Tagato (tagattie) textproc/py-tokenizers: update to 0.21.4
Changelog:
- https://github.com/huggingface/tokenizers/releases/tag/v0.21.3
- https://github.com/huggingface/tokenizers/releases/tag/v0.21.4
Reported by: Repology 0.21.2_1 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.88.0
PR: 287766 0.21.2 Hiroki Tagato (tagattie) textproc/py-tokenizers: update to 0.21.2
While here, refactor test-related part so that it can execute both
python and rust tests.
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.21.2
Reported by: Repology 0.21.1_2 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.87.0
PR: 286829 0.21.1_1 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.86.0
PR: 285840 0.21.1 Hiroki Tagato (tagattie) textproc/py-tokenizers: update to 0.21.1
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.21.1
Reported by: portscout 0.21.0_2 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.85.0
PR: 284884 0.21.0_1 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.84.0
PR: 283962 0.21.0 Hiroki Tagato (tagattie) textproc/py-tokenizers: update to 0.21.0
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.21.0
Reported by: Repology 0.20.3_2 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.83.0
PR: 283000 0.20.3_1 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.82.0
PR: 282516 0.20.3 Hiroki Tagato (tagattie) textproc/py-tokenizers: update to 0.20.3
Changelog:
- https://github.com/huggingface/tokenizers/releases/tag/v0.20.2
- https://github.com/huggingface/tokenizers/releases/tag/v0.20.3
Reported by: portscout 0.20.1 Hiroki Tagato (tagattie) textproc/py-tokenizers: update to 0.20.1
While here, add LICENSE_FILE.
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.20.1
Reported by: Repology 0.20.0_2 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.81.0
PR: 281300 0.20.0_1 Mikael Urankar (mikael) Siva Mahadevan */*: remove STRIP_CMD calls in rust based ports
This is not needed after bc4fedc1fec0d359365c04d43be9e32bf101a50e
PR: 246993
Differential Revision: https://reviews.freebsd.org/D46503 0.20.0_1 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.80.1
PR: 280490 0.20.0 Hiroki Tagato (tagattie) textproc/py-tokenizers: update to 0.20.0
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.20.0
Reported by: portscout 0.19.1_2 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.79.0
PR: 279707 0.19.1_1 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.78.0
PR: 278834 0.19.1 Hiroki Tagato (tagattie) textproc/py-tokenizers: update to 0.19.1
Changelog:
- https://github.com/huggingface/tokenizers/releases/tag/v0.19.0
- https://github.com/huggingface/tokenizers/releases/tag/v0.19.1
Reported by: Repology 0.15.2_2 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.77.0
PR: 277786 0.15.2_1 Mikael Urankar (mikael) lang/rust: Bump revisions after 1.76.0
PR: 276920 0.15.2 Hiroki Tagato (tagattie) textproc/py-tokenizers: update to 0.15.2
While here, enable tests.
Changelog: https://github.com/huggingface/tokenizers/releases/tag/v0.15.2
Reported by: portscout 0.15.1 Hiroki Tagato (tagattie) textproc/py-tokenizers: add port: Fast state-of-the-art tokenizers optimized for
research and production
Provides an implementation of today's most used tokenizers, with a
focus on performance and versatility.
Main features:
- Train new vocabularies and tokenize, using today's most used
tokenizers.
- Extremely fast (both training and tokenization), thanks to the Rust
implementation. Takes less than 20 seconds to tokenize a GB of text
on a server's CPU.
- Easy to use, but also extremely versatile.
- Designed for research and production.
- Normalization comes with alignments tracking. It's always possible
to get the part of the original sentence that corresponds to a given
token.
- Does all the pre-processing: Truncate, Pad, add the special tokens
your model needs.
WWW: https://github.com/huggingface/tokenizers
