| Port details |
- py-tokenizer Tokenizer for Icelandic text
- 3.5.2 textproc
 =0      Package not present on quarterly.This port was created during this quarter. It will be in the next quarterly branch but not the current one. - Maintainer: otis@FreeBSD.org
- Port Added: 2025-08-27 11:15:08
- Last Update: 2025-09-26 15:07:41
- Commit Hash: af01614
- Also Listed In: python
- License: MIT
- WWW:
- https://github.com/mideind/Tokenizer
- Description:
- Tokenizer: A tokenizer for Icelandic text
Tokenization is a necessary first step in many natural language processing
tasks, such as word counting, parsing, spell checking, corpus generation, and
statistical analysis of text.
  ¦  ¦  ¦  ¦ 
- Manual pages:
- FreshPorts has no man page information for this port.
- pkg-plist: as obtained via:
make generate-plist - There is no configure plist information for this port.
- USE_RC_SUBR (Service Scripts)
- no SUBR information found for this port
- Dependency lines:
-
- ${PYTHON_PKGNAMEPREFIX}tokenizer>0:textproc/py-tokenizer@${PY_FLAVOR}
- To install the port:
- cd /usr/ports/textproc/py-tokenizer/ && make install clean
- To add the package, run one of these commands:
- pkg install textproc/py-tokenizer
- pkg install py311-tokenizer
NOTE: If this package has multiple flavors (see below), then use one of them instead of the name specified above.
NOTE: This is a Python port. Instead of py311-tokenizer listed in the above command, you can pick from the names under the Packages section.- PKGNAME: py311-tokenizer
- Package flavors (<flavor>: <package>)
- distinfo:
- TIMESTAMP = 1758898772
SHA256 (tokenizer-3.5.2.tar.gz) = 36a425197f0623b68ca1e81f22b283df599f80ed1121af8a88a1a45031eb36a9
SIZE (tokenizer-3.5.2.tar.gz) = 104221
Packages (timestamps in pop-ups are UTC):
- Dependencies
- NOTE: FreshPorts displays only information on required and default dependencies. Optional dependencies are not covered.
- Build dependencies:
-
- py311-setuptools>0 : devel/py-setuptools@py311
- py311-wheel>0 : devel/py-wheel@py311
- python3.11 : lang/python311
- py311-build>=0 : devel/py-build@py311
- py311-installer>=0 : devel/py-installer@py311
- Test dependencies:
-
- python3.11 : lang/python311
- Runtime dependencies:
-
- python3.11 : lang/python311
- There are no ports dependent upon this port
Configuration Options:- No options to configure
- Options name:
- textproc_py-tokenizer
- USES:
- python
- FreshPorts was unable to extract/find any pkg message
- Master Sites:
|
Number of commits found: 3
| Commit History - (may be incomplete: for full details, see links to repositories near top of page) |
| Commit | Credits | Log message |
|---|
3.5.2
26 Sep 2025 15:07:41
 |
Juraj Lutter (otis) |
textproc/py-tokenizer: Update to 3.5.2
Release notes: https://github.com/mideind/Tokenizer/releases/tag/v3.5.2 |
3.5.0_1
27 Aug 2025 14:40:21
|
Juraj Lutter (otis) |
textproc/py-tokenizer: Correct pkg-descr
A text for different package has slipped through and made it
to pkg-descr of this package.
Put the correct description instead. |
3.5.0
27 Aug 2025 11:13:40
|
Juraj Lutter (otis) |
textproc/py-tokenizer: Add new port
Add textproc/py-tokenizer, a tokenizer for Icelandic text. |
Number of commits found: 3
|
As an Amazon Associate I earn from qualifying purchases.