notbugAs an Amazon Associate I earn from qualifying purchases.
Want a good read? Try FreeBSD Mastery: Jails (IT Mastery Book 15)Want a good monitor light? See my photosAll times are UTC		 Ukraine
Port details
py-tiktoken Fast BPE tokeniser for use with OpenAI's models
0.8.0_2 textproc on this many watch lists=2 search for ports that depend on this port Find issues related to this port Report an issue related to this port View this port on Repology. pkg-fallout 0.7.0_3Version of this port present on the latest quarterly branch.
Maintainer: sunpoet@FreeBSD.org search for ports maintained by this maintainer
Port Added: 2023-08-21 17:15:36
Last Update: 2024-12-01 09:24:18
Commit Hash: 1d1fd15
People watching this port, also watch:: jdictionary, py311-Automat, py311-python-gdsii, py39-PyOpenGL, p5-Sane
Also Listed In: python
License: MIT
WWW:
https://github.com/openai/tiktoken
Description:
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Homepage    cgit ¦ Codeberg ¦ GitHub ¦ GitLab ¦ SVNWeb - no subversion history for this port

Manual pages:
FreshPorts has no man page information for this port.
pkg-plist: as obtained via: make generate-plist
There is no configure plist information for this port.
Dependency lines:
  • ${PYTHON_PKGNAMEPREFIX}tiktoken>0:textproc/py-tiktoken@${PY_FLAVOR}
To install the port:
cd /usr/ports/textproc/py-tiktoken/ && make install clean
To add the package, run one of these commands:
  • pkg install textproc/py-tiktoken
  • pkg install py311-tiktoken
NOTE: If this package has multiple flavors (see below), then use one of them instead of the name specified above.
NOTE: This is a Python port. Instead of py311-tiktoken listed in the above command, you can pick from the names under the Packages section.
PKGNAME: py311-tiktoken
Package flavors (<flavor>: <package>)
  • py311: py311-tiktoken
distinfo:
TIMESTAMP = 1728632490 SHA256 (tiktoken-0.8.0.tar.gz) = 9ccbb2740f24542534369c5635cfd9b2b3c2490754a78ac8831d99f89f94eeb2 SIZE (tiktoken-0.8.0.tar.gz) = 35107

Expand this list (62 items)

Collapse this list.

SHA256 (rust/crates/aho-corasick-1.1.3.crate) = 8e60d3430d3a69478ad0993f19238d2df97c507009a52b3c10addcd7f6bcb916 SIZE (rust/crates/aho-corasick-1.1.3.crate) = 183311 SHA256 (rust/crates/autocfg-1.4.0.crate) = ace50bade8e6234aa140d9a2f552bbee1db4d353f69b8217bc503490fc1a9f26 SIZE (rust/crates/autocfg-1.4.0.crate) = 17712 SHA256 (rust/crates/bit-set-0.5.3.crate) = 0700ddab506f33b20a03b13996eccd309a48e5ff77d0d95926aa0210fb4e95f1 SIZE (rust/crates/bit-set-0.5.3.crate) = 14470 SHA256 (rust/crates/bit-vec-0.6.3.crate) = 349f9b6a179ed607305526ca489b34ad0a41aed5f7980fa90eb03160b69598fb SIZE (rust/crates/bit-vec-0.6.3.crate) = 19927 SHA256 (rust/crates/bstr-1.10.0.crate) = 40723b8fb387abc38f4f4a37c09073622e41dd12327033091ef8950659e6dc0c SIZE (rust/crates/bstr-1.10.0.crate) = 380426 SHA256 (rust/crates/cfg-if-1.0.0.crate) = baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd SIZE (rust/crates/cfg-if-1.0.0.crate) = 7934 SHA256 (rust/crates/fancy-regex-0.13.0.crate) = 531e46835a22af56d1e3b66f04844bed63158bc094a628bec1d321d9b4c44bf2 SIZE (rust/crates/fancy-regex-0.13.0.crate) = 85930 SHA256 (rust/crates/heck-0.5.0.crate) = 2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea SIZE (rust/crates/heck-0.5.0.crate) = 11517 SHA256 (rust/crates/indoc-2.0.5.crate) = b248f5224d1d606005e02c97f5aa4e88eeb230488bcc03bc9ca4d7991399f2b5 SIZE (rust/crates/indoc-2.0.5.crate) = 14396 SHA256 (rust/crates/libc-0.2.159.crate) = 561d97a539a36e26a9a5fad1ea11a3039a67714694aaa379433e580854bc3dc5 SIZE (rust/crates/libc-0.2.159.crate) = 755366 SHA256 (rust/crates/memchr-2.7.4.crate) = 78ca9ab1a0babb1e7d5695e3530886289c18cf2f87ec19a575a0abdce112e3a3 SIZE (rust/crates/memchr-2.7.4.crate) = 96670 SHA256 (rust/crates/memoffset-0.9.1.crate) = 488016bfae457b036d996092f6cb448677611ce4449e970ceaf42695203f218a SIZE (rust/crates/memoffset-0.9.1.crate) = 9032 SHA256 (rust/crates/once_cell-1.20.2.crate) = 1261fe7e33c73b354eab43b1273a57c8f967d0391e80353e51f764ac02cf6775 SIZE (rust/crates/once_cell-1.20.2.crate) = 33394 SHA256 (rust/crates/portable-atomic-1.9.0.crate) = cc9c68a3f6da06753e9335d63e27f6b9754dd1920d941135b7ea8224f141adb2 SIZE (rust/crates/portable-atomic-1.9.0.crate) = 166365 SHA256 (rust/crates/proc-macro2-1.0.87.crate) = b3e4daa0dcf6feba26f985457cdf104d4b4256fc5a09547140f3631bb076b19a SIZE (rust/crates/proc-macro2-1.0.87.crate) = 49121 SHA256 (rust/crates/pyo3-0.22.4.crate) = 00e89ce2565d6044ca31a3eb79a334c3a79a841120a98f64eea9f579564cb691 SIZE (rust/crates/pyo3-0.22.4.crate) = 546534 SHA256 (rust/crates/pyo3-build-config-0.22.4.crate) = d8afbaf3abd7325e08f35ffb8deb5892046fcb2608b703db6a583a5ba4cea01e SIZE (rust/crates/pyo3-build-config-0.22.4.crate) = 31296 SHA256 (rust/crates/pyo3-ffi-0.22.4.crate) = ec15a5ba277339d04763f4c23d85987a5b08cbb494860be141e6a10a8eb88022 SIZE (rust/crates/pyo3-ffi-0.22.4.crate) = 69360 SHA256 (rust/crates/pyo3-macros-0.22.4.crate) = 15e0f01b5364bcfbb686a52fc4181d412b708a68ed20c330db9fc8d2c2bf5a43 SIZE (rust/crates/pyo3-macros-0.22.4.crate) = 8188 SHA256 (rust/crates/pyo3-macros-backend-0.22.4.crate) = a09b550200e1e5ed9176976d0060cbc2ea82dc8515da07885e7b8153a85caacb SIZE (rust/crates/pyo3-macros-backend-0.22.4.crate) = 66307 SHA256 (rust/crates/quote-1.0.37.crate) = b5b9d34b8991d19d98081b46eacdd8eb58c6f2b201139f7c5f643cc155a633af SIZE (rust/crates/quote-1.0.37.crate) = 28558 SHA256 (rust/crates/regex-1.11.0.crate) = 38200e5ee88914975b69f657f0801b6f6dccafd44fd9326302a4aaeecfacb1d8 SIZE (rust/crates/regex-1.11.0.crate) = 254059 SHA256 (rust/crates/regex-automata-0.4.8.crate) = 368758f23274712b504848e9d5a6f010445cc8b87a7cdb4d7cbee666c1288da3 SIZE (rust/crates/regex-automata-0.4.8.crate) = 617784 SHA256 (rust/crates/regex-syntax-0.8.5.crate) = 2b15c43186be67a4fd63bee50d0303afffcef381492ebe2c5d87f324e1b8815c SIZE (rust/crates/regex-syntax-0.8.5.crate) = 357541 SHA256 (rust/crates/rustc-hash-1.1.0.crate) = 08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2 SIZE (rust/crates/rustc-hash-1.1.0.crate) = 9331 SHA256 (rust/crates/serde-1.0.210.crate) = c8e3592472072e6e22e0a54d5904d9febf8508f65fb8552499a1abc7d1078c3a SIZE (rust/crates/serde-1.0.210.crate) = 78338 SHA256 (rust/crates/serde_derive-1.0.210.crate) = 243902eda00fad750862fc144cea25caca5e20d615af0a81bee94ca738f1df1f SIZE (rust/crates/serde_derive-1.0.210.crate) = 56020 SHA256 (rust/crates/syn-2.0.79.crate) = 89132cd0bf050864e1d38dc3bbc07a0eb8e7530af26344d3d2bbbef83499f590 SIZE (rust/crates/syn-2.0.79.crate) = 268981 SHA256 (rust/crates/target-lexicon-0.12.16.crate) = 61c41af27dd6d1e27b1b16b489db798443478cef1f06a660c96db617ba5de3b1 SIZE (rust/crates/target-lexicon-0.12.16.crate) = 26488 SHA256 (rust/crates/unicode-ident-1.0.13.crate) = e91b56cd4cadaeb79bbf1a5645f6b4f8dc5bde8834ad5894a8db35fda9efa1fe SIZE (rust/crates/unicode-ident-1.0.13.crate) = 43279 SHA256 (rust/crates/unindent-0.2.3.crate) = c7de7d73e1754487cb58364ee906a499937a0dfabd86bcb980fa99ec8c8fa2ce SIZE (rust/crates/unindent-0.2.3.crate) = 7306

Collapse this list.

Packages (timestamps in pop-ups are UTC):
py39-tiktoken
ABIaarch64amd64armv6armv7i386powerpcpowerpc64powerpc64le
FreeBSD:13:latest--------
FreeBSD:13:quarterly------0.5.20.5.2
FreeBSD:14:latest--------
FreeBSD:14:quarterly------0.5.20.5.2
FreeBSD:15:latest--n/a-n/a-0.5.2_1-
 

py311-tiktoken
ABIaarch64amd64armv6armv7i386powerpcpowerpc64powerpc64le
FreeBSD:13:latest0.8.0_20.8.0_2-0.8.0_10.8.0_2---
FreeBSD:13:quarterly0.7.0_30.7.0_3--0.7.0_3---
FreeBSD:14:latest0.8.0_20.8.0_2-0.8.0_10.8.0_2---
FreeBSD:14:quarterly0.7.0_30.7.0_3--0.7.0_3---
FreeBSD:15:latest0.8.0_20.8.0_2n/a0.8.0_1n/a---
 

Dependencies
NOTE: FreshPorts displays only information on required and default dependencies. Optional dependencies are not covered.
Build dependencies:
  1. py311-setuptools>=62.4 : devel/py-setuptools@py311
  2. py311-setuptools-rust>=1.5.2 : devel/py-setuptools-rust@py311
  3. py311-wheel>=0 : devel/py-wheel@py311
  4. rust>=1.83.0 : lang/rust
  5. python3.11 : lang/python311
  6. py311-build>=0 : devel/py-build@py311
  7. py311-installer>=0 : devel/py-installer@py311
Test dependencies:
  1. python3.11 : lang/python311
Runtime dependencies:
  1. py311-regex>=2022.1.18 : textproc/py-regex@py311
  2. py311-requests>=2.26.0 : www/py-requests@py311
  3. python3.11 : lang/python311
This port is required by:
for Run
  1. devel/py-ddtrace
  2. misc/py-aider-chat
  3. misc/py-litellm
  4. misc/py-openai-whisper
Configuration Options:
===> The following configuration options are available for py311-tiktoken-0.8.0_2: BLOBFILE=off: Blob file support ===> Use 'make config' to modify these settings
Options name:
textproc_py-tiktoken
USES:
cargo python:3.9+
FreshPorts was unable to extract/find any pkg message
Master Sites:
Expand this list (2 items)
Collapse this list.
  1. https://files.pythonhosted.org/packages/source/t/tiktoken/
  2. https://pypi.org/packages/source/t/tiktoken/
Collapse this list.

Number of commits found: 19

Commit History - (may be incomplete: for full details, see links to repositories near top of page)
CommitCreditsLog message
0.8.0_2
01 Dec 2024 09:24:18
commit hash: 1d1fd15a3e499eed9e604dac94bab73c2fea4b2fcommit hash: 1d1fd15a3e499eed9e604dac94bab73c2fea4b2fcommit hash: 1d1fd15a3e499eed9e604dac94bab73c2fea4b2fcommit hash: 1d1fd15a3e499eed9e604dac94bab73c2fea4b2f files touched by this commit		 Mikael Urankar (mikael) search for other commits by this committer 
lang/rust: Bump revisions after 1.83.0

PR:		283000
0.8.0_1
08 Nov 2024 08:24:20
commit hash: ca048c2884dc9822310a87e2dfebb8b1647095e2commit hash: ca048c2884dc9822310a87e2dfebb8b1647095e2commit hash: ca048c2884dc9822310a87e2dfebb8b1647095e2commit hash: ca048c2884dc9822310a87e2dfebb8b1647095e2 files touched by this commit		 Mikael Urankar (mikael) search for other commits by this committer 
lang/rust: Bump revisions after 1.82.0

PR:		282516
0.8.0
14 Oct 2024 05:25:48
commit hash: b9ec26d2939593c8f7a67c1a5af171d2ef88a61bcommit hash: b9ec26d2939593c8f7a67c1a5af171d2ef88a61bcommit hash: b9ec26d2939593c8f7a67c1a5af171d2ef88a61bcommit hash: b9ec26d2939593c8f7a67c1a5af171d2ef88a61b files touched by this commit		 Po-Chuan Hsieh (sunpoet) search for other commits by this committer 
textproc/py-tiktoken: Update to 0.8.0

Changes:	https://github.com/openai/tiktoken/releases
		https://github.com/openai/tiktoken/blob/main/CHANGELOG.md
0.7.0_3
10 Sep 2024 11:00:34
commit hash: 89ab81d0e2eca6b298b0efd3f4577d468abf18d5commit hash: 89ab81d0e2eca6b298b0efd3f4577d468abf18d5commit hash: 89ab81d0e2eca6b298b0efd3f4577d468abf18d5commit hash: 89ab81d0e2eca6b298b0efd3f4577d468abf18d5 files touched by this commit		 Mikael Urankar (mikael) search for other commits by this committer 
lang/rust: Bump revisions after 1.81.0

PR:		281300
0.7.0_2
10 Sep 2024 10:58:07
commit hash: c0d07f1127793ffc97a6c1db78423d0d16bdc5efcommit hash: c0d07f1127793ffc97a6c1db78423d0d16bdc5efcommit hash: c0d07f1127793ffc97a6c1db78423d0d16bdc5efcommit hash: c0d07f1127793ffc97a6c1db78423d0d16bdc5ef files touched by this commit		 Mikael Urankar (mikael) search for other commits by this committer
Author: Siva Mahadevan		 
*/*: remove STRIP_CMD calls in rust based ports

This is not needed after bc4fedc1fec0d359365c04d43be9e32bf101a50e

PR:		246993
Differential Revision:	https://reviews.freebsd.org/D46503
0.7.0_2
26 Aug 2024 08:08:01
commit hash: 7a55f9542531c541f36ae5cf3a58d6ce92ec7bffcommit hash: 7a55f9542531c541f36ae5cf3a58d6ce92ec7bffcommit hash: 7a55f9542531c541f36ae5cf3a58d6ce92ec7bffcommit hash: 7a55f9542531c541f36ae5cf3a58d6ce92ec7bff files touched by this commit		 Mikael Urankar (mikael) search for other commits by this committer 
lang/rust: Bump revisions after 1.80.1

PR:		280490
0.7.0_1
18 Jun 2024 10:59:14
commit hash: 21476ca8a9b5feb3485a3fa29299c0b447c90350commit hash: 21476ca8a9b5feb3485a3fa29299c0b447c90350commit hash: 21476ca8a9b5feb3485a3fa29299c0b447c90350commit hash: 21476ca8a9b5feb3485a3fa29299c0b447c90350 files touched by this commit		 Mikael Urankar (mikael) search for other commits by this committer 
lang/rust: Bump revisions after 1.79.0

PR:		279707
0.7.0
19 May 2024 17:15:56
commit hash: 741835f949140931e2525cc378007fbc212c27a3commit hash: 741835f949140931e2525cc378007fbc212c27a3commit hash: 741835f949140931e2525cc378007fbc212c27a3commit hash: 741835f949140931e2525cc378007fbc212c27a3 files touched by this commit		 Po-Chuan Hsieh (sunpoet) search for other commits by this committer 
textproc/py-tiktoken: Update to 0.7.0

Changes:	https://github.com/openai/tiktoken/releases
		https://github.com/openai/tiktoken/blob/main/CHANGELOG.md
0.6.0_2
13 May 2024 11:03:24
commit hash: 2322b661000b69fcc75fe68fec25e5b4e48c2a8ecommit hash: 2322b661000b69fcc75fe68fec25e5b4e48c2a8ecommit hash: 2322b661000b69fcc75fe68fec25e5b4e48c2a8ecommit hash: 2322b661000b69fcc75fe68fec25e5b4e48c2a8e files touched by this commit		 Mikael Urankar (mikael) search for other commits by this committer 
lang/rust: Bump revisions after 1.78.0

PR:		278834
0.6.0_1
23 Mar 2024 09:41:46
commit hash: d632fbe55e45559d03a0bef7211913964aeba45acommit hash: d632fbe55e45559d03a0bef7211913964aeba45acommit hash: d632fbe55e45559d03a0bef7211913964aeba45acommit hash: d632fbe55e45559d03a0bef7211913964aeba45a files touched by this commit		 Mikael Urankar (mikael) search for other commits by this committer 
lang/rust: Bump revisions after 1.77.0

PR:		277786
0.6.0
21 Feb 2024 15:08:04
commit hash: 4e17347e94db300992ec8544fd7cc7705c5d8c18commit hash: 4e17347e94db300992ec8544fd7cc7705c5d8c18commit hash: 4e17347e94db300992ec8544fd7cc7705c5d8c18commit hash: 4e17347e94db300992ec8544fd7cc7705c5d8c18 files touched by this commit		 Po-Chuan Hsieh (sunpoet) search for other commits by this committer 
textproc/py-tiktoken: Update to 0.6.0

Changes:	https://github.com/openai/tiktoken/releases
		https://github.com/openai/tiktoken/blob/main/CHANGELOG.md
0.5.2_2
19 Feb 2024 11:59:23
commit hash: 7f5a7b0e524c8d32727681a4324905e350dc433ccommit hash: 7f5a7b0e524c8d32727681a4324905e350dc433ccommit hash: 7f5a7b0e524c8d32727681a4324905e350dc433ccommit hash: 7f5a7b0e524c8d32727681a4324905e350dc433c files touched by this commit		 Mikael Urankar (mikael) search for other commits by this committer 
lang/rust: Bump revisions after 1.76.0

PR:		276920
0.5.2_1
09 Jan 2024 12:15:54
commit hash: 109c3d4629b84972e660b689d169ac0761c1a519commit hash: 109c3d4629b84972e660b689d169ac0761c1a519commit hash: 109c3d4629b84972e660b689d169ac0761c1a519commit hash: 109c3d4629b84972e660b689d169ac0761c1a519 files touched by this commit		 Mikael Urankar (mikael) search for other commits by this committer 
lang/rust: Bump revisions after 1.75.0

PR:		276033
0.5.2
14 Dec 2023 05:48:32
commit hash: d3c7291cd247fef7bb9faed43d35b38c37dd1c6fcommit hash: d3c7291cd247fef7bb9faed43d35b38c37dd1c6fcommit hash: d3c7291cd247fef7bb9faed43d35b38c37dd1c6fcommit hash: d3c7291cd247fef7bb9faed43d35b38c37dd1c6f files touched by this commit		 Po-Chuan Hsieh (sunpoet) search for other commits by this committer 
textproc/py-tiktoken: Update to 0.5.2

Changes:	https://github.com/openai/tiktoken/releases
		https://github.com/openai/tiktoken/blob/main/CHANGELOG.md
0.5.1_2
08 Dec 2023 15:04:09
commit hash: b2afd89fb26437c5d9810b131370525352e63f71commit hash: b2afd89fb26437c5d9810b131370525352e63f71commit hash: b2afd89fb26437c5d9810b131370525352e63f71commit hash: b2afd89fb26437c5d9810b131370525352e63f71 files touched by this commit		 Mikael Urankar (mikael) search for other commits by this committer 
lang/rust: Bump revisions after 1.74.1

PR:		275243
0.5.1_1
24 Oct 2023 10:22:32
commit hash: dd10dc88a7cf54f9aa164bf40cc5e481021401d6commit hash: dd10dc88a7cf54f9aa164bf40cc5e481021401d6commit hash: dd10dc88a7cf54f9aa164bf40cc5e481021401d6commit hash: dd10dc88a7cf54f9aa164bf40cc5e481021401d6 files touched by this commit		 Mikael Urankar (mikael) search for other commits by this committer 
lang/rust: Bump revisions after 1.73.0

PR:		274499
0.5.1
22 Sep 2023 03:19:31
commit hash: 227e43a0b0373cf0e65d381134ed5561db9a05a8commit hash: 227e43a0b0373cf0e65d381134ed5561db9a05a8commit hash: 227e43a0b0373cf0e65d381134ed5561db9a05a8commit hash: 227e43a0b0373cf0e65d381134ed5561db9a05a8 files touched by this commit		 Po-Chuan Hsieh (sunpoet) search for other commits by this committer 
textproc/py-tiktoken: Update to 0.5.1

Changes:	https://github.com/openai/tiktoken/releases
		https://github.com/openai/tiktoken/blob/main/CHANGELOG.md
0.4.0_1
08 Sep 2023 09:06:24
commit hash: aba8da2be191461126505c67726fa87345ae63f8commit hash: aba8da2be191461126505c67726fa87345ae63f8commit hash: aba8da2be191461126505c67726fa87345ae63f8commit hash: aba8da2be191461126505c67726fa87345ae63f8 files touched by this commit		 Mikael Urankar (mikael) search for other commits by this committer 
lang/rust: Bump revisions after 1.72.0

PR:		273331
0.4.0
21 Aug 2023 17:01:58
commit hash: 6c34be9ab1b27410b400e03f8281f03d4f956fa3commit hash: 6c34be9ab1b27410b400e03f8281f03d4f956fa3commit hash: 6c34be9ab1b27410b400e03f8281f03d4f956fa3commit hash: 6c34be9ab1b27410b400e03f8281f03d4f956fa3 files touched by this commit		 Po-Chuan Hsieh (sunpoet) search for other commits by this committer 
textproc/py-tiktoken: Add py-tiktoken 0.4.0

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

Number of commits found: 19
Login
User Login
Create account

Servers and bandwidth provided by
New York Internet, iXsystems, and RootBSD

This site
What is FreshPorts?
About the authors
Issues
FAQ
How big is it?
Security Policy
Privacy
Blog
Contact

Search
Enter Keywords:
 
more...

Latest Vulnerabilities
vaultwardenDec 20
chromiumDec 19
ungoogled-chromiumDec 19
forgejo*Dec 18
forgejo7*Dec 18
liboqsDec 18
forgejoDec 17
forgejoDec 17
giteaDec 17
giteaDec 17
giteaDec 17
zeekDec 16
py-matrix-synapseDec 15
gitlabDec 12
chromiumDec 11

23 vulnerabilities affecting 297 ports have been reported in the past 14 days

* - modified, not new

All vulnerabilities

Last processed:
2024-12-20 14:15:46 UTC


Ports
Home
Categories
Deleted ports
Sanity Test Failures
Newsfeeds

Statistics
Graphs
NEW Graphs (Javascript)

Calculated hourly:
Port count 33181
Broken 114
Deprecated 186
Ignore 238
Forbidden 1
Restricted 1
No CDROM 1
Vulnerable 35
Expired 12
Set to expire 152
Interactive 0
new 24 hours 2
new 48 hours12
new 7 days46
new fortnight69
new month163
Servers and bandwidth provided by
New York Internet, iXsystems, and RootBSD 		Valid HTML, CSS, and RSS. Copyright © 2000-2024 Dan Langille. All rights reserved.