| Port details |
- py-tokenizer Tokenizer for Icelandic text
- 3.6.4 textproc
 =0      3.5.2Version of this port present on the latest quarterly branch. - Maintainer: otis@FreeBSD.org
- Port Added: 2025-08-27 11:15:08
- Last Update: 2026-07-12 10:46:32
- Commit Hash: 1647234
- Also Listed In: python
- License: MIT
- WWW:
- https://github.com/mideind/Tokenizer
- Description:
- Tokenizer: A tokenizer for Icelandic text
Tokenization is a necessary first step in many natural language processing
tasks, such as word counting, parsing, spell checking, corpus generation, and
statistical analysis of text.
  ¦  ¦  ¦  ¦ 
- Manual pages:
- FreshPorts has no man page information for this port.
- pkg-plist: as obtained via:
make generate-plist - There is no configure plist information for this port.
- USE_RC_SUBR (Service Scripts)
- no SUBR information found for this port
- Dependency lines:
-
- ${PYTHON_PKGNAMEPREFIX}tokenizer>0:textproc/py-tokenizer@${PY_FLAVOR}
- To install the port:
- cd /usr/ports/textproc/py-tokenizer/ && make install clean
- To add the package, run one of these commands:
- pkg install textproc/py-tokenizer
- pkg install py312-tokenizer
NOTE: If this package has multiple flavors (see below), then use one of them instead of the name specified above.
NOTE: This is a Python port. Instead of py312-tokenizer listed in the above command, you can pick from the names under the Packages section.- PKGNAME: py312-tokenizer
- Package flavors (<flavor>: <package>)
- distinfo:
- TIMESTAMP = 1783688101
SHA256 (tokenizer-3.6.4.tar.gz) = 1bc7423853e0d2e1430ffdff2c1a4048db3aedd480480c7542ecd2ff8d5fcf58
SIZE (tokenizer-3.6.4.tar.gz) = 135242
Packages (timestamps in pop-ups are UTC):
- Dependencies
- NOTE: FreshPorts displays only information on required and default dependencies. Optional dependencies are not covered.
- Build dependencies:
-
- py312-setuptools>0 : devel/py-setuptools@py312
- py312-wheel>0 : devel/py-wheel@py312
- python3.12 : lang/python312
- py312-build>=0 : devel/py-build@py312
- py312-installer>=0 : devel/py-installer@py312
- Test dependencies:
-
- python3.12 : lang/python312
- Runtime dependencies:
-
- python3.12 : lang/python312
- There are no ports dependent upon this port
Configuration Options:- No options to configure
- Options name:
- textproc_py-tokenizer
- USES:
- python
- FreshPorts was unable to extract/find any pkg message
- Master Sites:
|
Number of commits found: 5
| Commit History - (may be incomplete: for full details, see links to repositories near top of page) |
| Commit | Credits | Log message |
|---|
3.6.4
12 Jul 2026 10:46:32
 |
Juraj Lutter (otis) |
textproc/py-tokenizer: Fix build
Reported by: pkg-fallout |
3.6.4
10 Jul 2026 13:33:45
|
Juraj Lutter (otis) |
textproc/py-tokenizer: Update to 3.6.4 |
3.5.2
26 Sep 2025 15:07:41
|
Juraj Lutter (otis) |
textproc/py-tokenizer: Update to 3.5.2
Release notes: https://github.com/mideind/Tokenizer/releases/tag/v3.5.2 |
3.5.0_1
27 Aug 2025 14:40:21
|
Juraj Lutter (otis) |
textproc/py-tokenizer: Correct pkg-descr
A text for different package has slipped through and made it
to pkg-descr of this package.
Put the correct description instead. |
3.5.0
27 Aug 2025 11:13:40
|
Juraj Lutter (otis) |
textproc/py-tokenizer: Add new port
Add textproc/py-tokenizer, a tokenizer for Icelandic text. |
Number of commits found: 5
|
As an Amazon Associate I earn from qualifying purchases.