Port details
py-tokenizer Tokenizer for Icelandic text
3.6.4 textproc Σ 3.6.4Version of this port present on the latest quarterly branch. Maintainer: otis@FreeBSD.org Port Added: 2025-08-27 11:15:08Last Update: 2026-07-12 10:46:32Commit Hash: 1647234 Also Listed In: python License: MITWWW: https://github.com/mideind/Tokenizer Description: # Tokenizer: A tokenizer for Icelandic text
Tokenization is a necessary first step in many natural language processing
tasks, such as word counting, parsing, spell checking, corpus generation, and
statistical analysis of text. Manual pages: # FreshPorts has no man page information for this port.
pkg-plist: make generate-plist # There is no configure plist information for this port. USE_RC_SUBR (Service Scripts)
no SUBR information found for this port
Dependency lines :
${PYTHON_PKGNAMEPREFIX}tokenizer>0:textproc/py-tokenizer@${PY_FLAVOR} To install the port : # cd /usr/ports/textproc/py-tokenizer/ && make install clean To add the package , run one of these commands: pkg install textproc/py-tokenizer pkg install py312-tokenizer py312-tokenizer listed in the above command, you can pick from the names under the Packages section.PKGNAME: py312-tokenizer Package flavors (<flavor>: <package> ) # distinfo: # TIMESTAMP = 1783688101
SHA256 (tokenizer-3.6.4.tar.gz) = 1bc7423853e0d2e1430ffdff2c1a4048db3aedd480480c7542ecd2ff8d5fcf58
SIZE (tokenizer-3.6.4.tar.gz) = 135242
Packages (timestamps in pop-ups are UTC): #
Dependencies # NOTE: FreshPorts displays only information on required and default dependencies. Optional dependencies are not covered. Build dependencies: #
py312-setuptools>0 : devel/py-setuptools@py312 py312-wheel>0 : devel/py-wheel@py312 python3.12 : lang/python312 py312-build>=0 : devel/py-build@py312 py312-installer>=0 : devel/py-installer@py312
Test dependencies: #
python3.12 : lang/python312
Runtime dependencies: #
python3.12 : lang/python312
There are no ports dependent upon this port # Configuration Options : # No options to configure Options name : # textproc_py-tokenizer USES: # python
FreshPorts was unable to extract/find any pkg message # Master Sites: #
Number of commits found: 5 #
Commit History - (may be incomplete: for full details, see links to repositories near top of page) Commit Credits Log message 3.6.4 Juraj Lutter (otis) textproc/py-tokenizer: Fix build
Reported by: pkg-fallout 3.6.4 Juraj Lutter (otis) textproc/py-tokenizer: Update to 3.6.4 3.5.2 Juraj Lutter (otis) textproc/py-tokenizer: Update to 3.5.2
Release notes: https://github.com/mideind/Tokenizer/releases/tag/v3.5.2 3.5.0_1 Juraj Lutter (otis) textproc/py-tokenizer: Correct pkg-descr
A text for different package has slipped through and made it
to pkg-descr of this package.
Put the correct description instead. 3.5.0 Juraj Lutter (otis) textproc/py-tokenizer: Add new port
Add textproc/py-tokenizer, a tokenizer for Icelandic text.
Number of commits found: 5
