notbugAs an Amazon Associate I earn from qualifying purchases.
Want a good read? Try FreeBSD Mastery: Jails (IT Mastery Book 15)Want a good monitor light? See my photosAll times are UTC		 Ukraine
This referral link gives you 10% off a Fastmail.com account and gives me a discount on my Fastmail account.

Get notified when packages are built

A new feature has been added. FreshPorts already tracks package built by the FreeBSD project. This information is displayed on each port page. You can now get an email when FreshPorts notices a new package is available for something on one of your watch lists. However, you must opt into that. Click on Report Subscriptions on the right, and New Package Notification box, and click on Update.

Finally, under Watch Lists, click on ABI Package Subscriptions to select your ABI (e.g. FreeBSD:14:amd64) & package set (latest/quarterly) combination for a given watch list. This is what FreshPorts will look for.

Port details
py-tiktoken Fast BPE tokeniser for use with OpenAI's models
0.6.0_1 textproc on this many watch lists=0 search for ports that depend on this port Find issues related to this port Report an issue related to this port View this port on Repology. pkg-fallout 0.6.0_1Version of this port present on the latest quarterly branch.
Maintainer: sunpoet@FreeBSD.org search for ports maintained by this maintainer
Port Added: 2023-08-21 17:15:36
Last Update: 2024-03-23 09:41:46
Commit Hash: d632fbe
Also Listed In: python
License: MIT
WWW:
https://github.com/openai/tiktoken
Description:
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Homepage    cgit ¦ Codeberg ¦ GitHub ¦ GitLab ¦ SVNWeb - no subversion history for this port

Manual pages:
FreshPorts has no man page information for this port.
pkg-plist: as obtained via: make generate-plist
There is no configure plist information for this port.
Dependency lines:
  • ${PYTHON_PKGNAMEPREFIX}tiktoken>0:textproc/py-tiktoken@${PY_FLAVOR}
To install the port:
cd /usr/ports/textproc/py-tiktoken/ && make install clean
To add the package, run one of these commands:
  • pkg install textproc/py-tiktoken
  • pkg install py39-tiktoken
NOTE: If this package has multiple flavors (see below), then use one of them instead of the name specified above.
NOTE: This is a Python port. Instead of py39-tiktoken listed in the above command, you can pick from the names under the Packages section.
PKGNAME: py39-tiktoken
Package flavors (<flavor>: <package>)
  • py39: py39-tiktoken
distinfo:
TIMESTAMP = 1708449482 SHA256 (tiktoken-0.6.0.tar.gz) = ace62a4ede83c75b0374a2ddfa4b76903cf483e9cb06247f566be3bf14e6beed SIZE (tiktoken-0.6.0.tar.gz) = 33404

Expand this list (90 items)

Collapse this list.

SHA256 (rust/crates/aho-corasick-1.1.2.crate) = b2969dcb958b36655471fc61f7e416fa76033bdd4bfed0678d8fee1e2d07a1f0 SIZE (rust/crates/aho-corasick-1.1.2.crate) = 183136 SHA256 (rust/crates/autocfg-1.1.0.crate) = d468802bab17cbc0cc575e9b053f41e72aa36bfa6b7f55e3529ffa43161b97fa SIZE (rust/crates/autocfg-1.1.0.crate) = 13272 SHA256 (rust/crates/bit-set-0.5.3.crate) = 0700ddab506f33b20a03b13996eccd309a48e5ff77d0d95926aa0210fb4e95f1 SIZE (rust/crates/bit-set-0.5.3.crate) = 14470 SHA256 (rust/crates/bit-vec-0.6.3.crate) = 349f9b6a179ed607305526ca489b34ad0a41aed5f7980fa90eb03160b69598fb SIZE (rust/crates/bit-vec-0.6.3.crate) = 19927 SHA256 (rust/crates/bitflags-1.3.2.crate) = bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a SIZE (rust/crates/bitflags-1.3.2.crate) = 23021 SHA256 (rust/crates/bstr-1.9.0.crate) = c48f0051a4b4c5e0b6d365cd04af53aeaa209e3cc15ec2cdb69e73cc87fbd0dc SIZE (rust/crates/bstr-1.9.0.crate) = 380170 SHA256 (rust/crates/cfg-if-1.0.0.crate) = baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd SIZE (rust/crates/cfg-if-1.0.0.crate) = 7934 SHA256 (rust/crates/fancy-regex-0.11.0.crate) = b95f7c0680e4142284cf8b22c14a476e87d61b004a3a0861872b32ef7ead40a2 SIZE (rust/crates/fancy-regex-0.11.0.crate) = 82918 SHA256 (rust/crates/heck-0.4.1.crate) = 95505c38b4572b2d910cecb0281560f54b440a19336cbbcb27bf6ce6adc6f5a8 SIZE (rust/crates/heck-0.4.1.crate) = 11567 SHA256 (rust/crates/indoc-2.0.4.crate) = 1e186cfbae8084e513daff4240b4797e342f988cecda4fb6c939150f96315fd8 SIZE (rust/crates/indoc-2.0.4.crate) = 14311 SHA256 (rust/crates/libc-0.2.153.crate) = 9c198f91728a82281a64e1f4f9eeb25d82cb32a5de251c6bd1b5154d63a8e7bd SIZE (rust/crates/libc-0.2.153.crate) = 740614 SHA256 (rust/crates/lock_api-0.4.11.crate) = 3c168f8615b12bc01f9c17e2eb0cc07dcae1940121185446edc3744920e8ef45 SIZE (rust/crates/lock_api-0.4.11.crate) = 27487 SHA256 (rust/crates/memchr-2.7.1.crate) = 523dc4f511e55ab87b694dc30d0f820d60906ef06413f93d4d7a1385599cc149 SIZE (rust/crates/memchr-2.7.1.crate) = 96307 SHA256 (rust/crates/memoffset-0.9.0.crate) = 5a634b1c61a95585bd15607c6ab0c4e5b226e695ff2800ba0cdccddf208c406c SIZE (rust/crates/memoffset-0.9.0.crate) = 9033 SHA256 (rust/crates/once_cell-1.19.0.crate) = 3fdb12b2476b595f9358c5161aa467c2438859caa136dec86c26fdd2efe17b92 SIZE (rust/crates/once_cell-1.19.0.crate) = 33046 SHA256 (rust/crates/parking_lot-0.12.1.crate) = 3742b2c103b9f06bc9fff0a37ff4912935851bee6d36f3c02bcc755bcfec228f SIZE (rust/crates/parking_lot-0.12.1.crate) = 40967 SHA256 (rust/crates/parking_lot_core-0.9.9.crate) = 4c42a9226546d68acdd9c0a280d17ce19bfe27a46bf68784e4066115788d008e SIZE (rust/crates/parking_lot_core-0.9.9.crate) = 32445 SHA256 (rust/crates/proc-macro2-1.0.78.crate) = e2422ad645d89c99f8f3e6b88a9fdeca7fabeac836b1002371c4367c8f984aae SIZE (rust/crates/proc-macro2-1.0.78.crate) = 47158 SHA256 (rust/crates/pyo3-0.20.2.crate) = 9a89dc7a5850d0e983be1ec2a463a171d20990487c3cfcd68b5363f1ee3d6fe0 SIZE (rust/crates/pyo3-0.20.2.crate) = 434326 SHA256 (rust/crates/pyo3-build-config-0.20.2.crate) = 07426f0d8fe5a601f26293f300afd1a7b1ed5e78b2a705870c5f30893c5163be SIZE (rust/crates/pyo3-build-config-0.20.2.crate) = 30029 SHA256 (rust/crates/pyo3-ffi-0.20.2.crate) = dbb7dec17e17766b46bca4f1a4215a85006b4c2ecde122076c562dd058da6cf1 SIZE (rust/crates/pyo3-ffi-0.20.2.crate) = 64601 SHA256 (rust/crates/pyo3-macros-0.20.2.crate) = 05f738b4e40d50b5711957f142878cfa0f28e054aa0ebdfc3fd137a843f74ed3 SIZE (rust/crates/pyo3-macros-0.20.2.crate) = 7925 SHA256 (rust/crates/pyo3-macros-backend-0.20.2.crate) = 0fc910d4851847827daf9d6cdd4a823fbdaab5b8818325c5e97a86da79e8881f SIZE (rust/crates/pyo3-macros-backend-0.20.2.crate) = 49710 SHA256 (rust/crates/quote-1.0.35.crate) = 291ec9ab5efd934aaf503a6466c5d5251535d108ee747472c3977cc5acc868ef SIZE (rust/crates/quote-1.0.35.crate) = 28136 SHA256 (rust/crates/redox_syscall-0.4.1.crate) = 4722d768eff46b75989dd134e5c353f0d6296e5aaa3132e776cbdb56be7731aa SIZE (rust/crates/redox_syscall-0.4.1.crate) = 24858 SHA256 (rust/crates/regex-1.10.3.crate) = b62dbe01f0b06f9d8dc7d49e05a0785f153b00b2c227856282f671e0318c9b15 SIZE (rust/crates/regex-1.10.3.crate) = 253101 SHA256 (rust/crates/regex-automata-0.4.5.crate) = 5bb987efffd3c6d0d8f5f89510bb458559eab11e4f869acb20bf845e016259cd SIZE (rust/crates/regex-automata-0.4.5.crate) = 617406 SHA256 (rust/crates/regex-syntax-0.8.2.crate) = c08c74e62047bb2de4ff487b251e4a92e24f48745648451635cec7d591162d9f SIZE (rust/crates/regex-syntax-0.8.2.crate) = 347228 SHA256 (rust/crates/rustc-hash-1.1.0.crate) = 08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2 SIZE (rust/crates/rustc-hash-1.1.0.crate) = 9331 SHA256 (rust/crates/scopeguard-1.2.0.crate) = 94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49 SIZE (rust/crates/scopeguard-1.2.0.crate) = 11619 SHA256 (rust/crates/serde-1.0.196.crate) = 870026e60fa08c69f064aa766c10f10b1d62db9ccd4d0abb206472bee0ce3b32 SIZE (rust/crates/serde-1.0.196.crate) = 77097 SHA256 (rust/crates/serde_derive-1.0.196.crate) = 33c85360c95e7d137454dc81d9a4ed2b8efd8fbe19cee57357b32b9771fccb67 SIZE (rust/crates/serde_derive-1.0.196.crate) = 55775 SHA256 (rust/crates/smallvec-1.13.1.crate) = e6ecd384b10a64542d77071bd64bd7b231f4ed5940fba55e98c3de13824cf3d7 SIZE (rust/crates/smallvec-1.13.1.crate) = 34952 SHA256 (rust/crates/syn-2.0.49.crate) = 915aea9e586f80826ee59f8453c1101f9d1c4b3964cd2460185ee8e299ada496 SIZE (rust/crates/syn-2.0.49.crate) = 250652 SHA256 (rust/crates/target-lexicon-0.12.13.crate) = 69758bda2e78f098e4ccb393021a0963bb3442eac05f135c30f61b7370bbafae SIZE (rust/crates/target-lexicon-0.12.13.crate) = 25424 SHA256 (rust/crates/unicode-ident-1.0.12.crate) = 3354b9ac3fae1ff6755cb6db53683adb661634f67557942dea4facebec0fee4b SIZE (rust/crates/unicode-ident-1.0.12.crate) = 42168 SHA256 (rust/crates/unindent-0.2.3.crate) = c7de7d73e1754487cb58364ee906a499937a0dfabd86bcb980fa99ec8c8fa2ce SIZE (rust/crates/unindent-0.2.3.crate) = 7306 SHA256 (rust/crates/windows-targets-0.48.5.crate) = 9a2fa6e2155d7247be68c096456083145c183cbbbc2764150dda45a87197940c SIZE (rust/crates/windows-targets-0.48.5.crate) = 6904 SHA256 (rust/crates/windows_aarch64_gnullvm-0.48.5.crate) = 2b38e32f0abccf9987a4e3079dfb67dcd799fb61361e53e2882c3cbaf0d905d8 SIZE (rust/crates/windows_aarch64_gnullvm-0.48.5.crate) = 418492 SHA256 (rust/crates/windows_aarch64_msvc-0.48.5.crate) = dc35310971f3b2dbbf3f0690a219f40e2d9afcf64f9ab7cc1be722937c26b4bc SIZE (rust/crates/windows_aarch64_msvc-0.48.5.crate) = 798483 SHA256 (rust/crates/windows_i686_gnu-0.48.5.crate) = a75915e7def60c94dcef72200b9a8e58e5091744960da64ec734a6c6e9b3743e SIZE (rust/crates/windows_i686_gnu-0.48.5.crate) = 844891 SHA256 (rust/crates/windows_i686_msvc-0.48.5.crate) = 8f55c233f70c4b27f66c523580f78f1004e8b5a8b659e05a4eb49d4166cca406 SIZE (rust/crates/windows_i686_msvc-0.48.5.crate) = 864300 SHA256 (rust/crates/windows_x86_64_gnu-0.48.5.crate) = 53d40abd2583d23e4718fddf1ebec84dbff8381c07cae67ff7768bbf19c6718e SIZE (rust/crates/windows_x86_64_gnu-0.48.5.crate) = 801619 SHA256 (rust/crates/windows_x86_64_gnullvm-0.48.5.crate) = 0b7b52767868a23d5bab768e390dc5f5c55825b6d30b86c844ff2dc7414044cc SIZE (rust/crates/windows_x86_64_gnullvm-0.48.5.crate) = 418486 SHA256 (rust/crates/windows_x86_64_msvc-0.48.5.crate) = ed94fce61571a4006852b7389a063ab983c02eb1bb37b47f8272ce92d06d9538 SIZE (rust/crates/windows_x86_64_msvc-0.48.5.crate) = 798412

Collapse this list.

Packages (timestamps in pop-ups are UTC):
py39-tiktoken
ABIaarch64amd64armv6armv7i386powerpcpowerpc64powerpc64le
FreeBSD:13:latest0.6.0_10.6.0_1--0.6.0_1---
FreeBSD:13:quarterly0.6.0_10.6.0_1--0.6.0_1-0.5.20.5.2
FreeBSD:14:latest0.6.0_10.6.0_1--0.6.0_1---
FreeBSD:14:quarterly0.5.20.6.0_1-0.5.20.6.0_1-0.5.20.5.2
FreeBSD:15:latest0.6.00.6.0_1n/a0.5.2_1n/a-0.5.2_1-
FreeBSD:15:quarterly--n/a-n/a---
Dependencies
NOTE: FreshPorts displays only information on required and default dependencies. Optional dependencies are not covered.
Build dependencies:
  1. py39-setuptools>=62.4 : devel/py-setuptools@py39
  2. py39-setuptools-rust>=1.5.2 : devel/py-setuptools-rust@py39
  3. py39-wheel>=0 : devel/py-wheel@py39
  4. rust>=1.77.0 : lang/rust
  5. python3.9 : lang/python39
  6. py39-build>=0 : devel/py-build@py39
  7. py39-installer>=0 : devel/py-installer@py39
Test dependencies:
  1. python3.9 : lang/python39
Runtime dependencies:
  1. py39-regex>=2022.1.18 : textproc/py-regex@py39
  2. py39-requests>=2.26.0 : www/py-requests@py39
  3. python3.9 : lang/python39
This port is required by:
for Run
  1. misc/py-litellm
  2. misc/py-openai-whisper
Configuration Options:
===> The following configuration options are available for py39-tiktoken-0.6.0_1: BLOBFILE=off: Blob file support ===> Use 'make config' to modify these settings
Options name:
textproc_py-tiktoken
USES:
cargo python
FreshPorts was unable to extract/find any pkg message
Master Sites:
Expand this list (2 items)
Collapse this list.
  1. https://files.pythonhosted.org/packages/source/t/tiktoken/
  2. https://pypi.org/packages/source/t/tiktoken/
Collapse this list.

Number of commits found: 10

Commit History - (may be incomplete: for full details, see links to repositories near top of page)
CommitCreditsLog message
0.6.0_1
23 Mar 2024 09:41:46
commit hash: d632fbe55e45559d03a0bef7211913964aeba45acommit hash: d632fbe55e45559d03a0bef7211913964aeba45acommit hash: d632fbe55e45559d03a0bef7211913964aeba45acommit hash: d632fbe55e45559d03a0bef7211913964aeba45a files touched by this commit		 Mikael Urankar (mikael) search for other commits by this committer 
lang/rust: Bump revisions after 1.77.0

PR:		277786
0.6.0
21 Feb 2024 15:08:04
commit hash: 4e17347e94db300992ec8544fd7cc7705c5d8c18commit hash: 4e17347e94db300992ec8544fd7cc7705c5d8c18commit hash: 4e17347e94db300992ec8544fd7cc7705c5d8c18commit hash: 4e17347e94db300992ec8544fd7cc7705c5d8c18 files touched by this commit		 Po-Chuan Hsieh (sunpoet) search for other commits by this committer 
textproc/py-tiktoken: Update to 0.6.0

Changes:	https://github.com/openai/tiktoken/releases
		https://github.com/openai/tiktoken/blob/main/CHANGELOG.md
0.5.2_2
19 Feb 2024 11:59:23
commit hash: 7f5a7b0e524c8d32727681a4324905e350dc433ccommit hash: 7f5a7b0e524c8d32727681a4324905e350dc433ccommit hash: 7f5a7b0e524c8d32727681a4324905e350dc433ccommit hash: 7f5a7b0e524c8d32727681a4324905e350dc433c files touched by this commit		 Mikael Urankar (mikael) search for other commits by this committer 
lang/rust: Bump revisions after 1.76.0

PR:		276920
0.5.2_1
09 Jan 2024 12:15:54
commit hash: 109c3d4629b84972e660b689d169ac0761c1a519commit hash: 109c3d4629b84972e660b689d169ac0761c1a519commit hash: 109c3d4629b84972e660b689d169ac0761c1a519commit hash: 109c3d4629b84972e660b689d169ac0761c1a519 files touched by this commit		 Mikael Urankar (mikael) search for other commits by this committer 
lang/rust: Bump revisions after 1.75.0

PR:		276033
0.5.2
14 Dec 2023 05:48:32
commit hash: d3c7291cd247fef7bb9faed43d35b38c37dd1c6fcommit hash: d3c7291cd247fef7bb9faed43d35b38c37dd1c6fcommit hash: d3c7291cd247fef7bb9faed43d35b38c37dd1c6fcommit hash: d3c7291cd247fef7bb9faed43d35b38c37dd1c6f files touched by this commit		 Po-Chuan Hsieh (sunpoet) search for other commits by this committer 
textproc/py-tiktoken: Update to 0.5.2

Changes:	https://github.com/openai/tiktoken/releases
		https://github.com/openai/tiktoken/blob/main/CHANGELOG.md
0.5.1_2
08 Dec 2023 15:04:09
commit hash: b2afd89fb26437c5d9810b131370525352e63f71commit hash: b2afd89fb26437c5d9810b131370525352e63f71commit hash: b2afd89fb26437c5d9810b131370525352e63f71commit hash: b2afd89fb26437c5d9810b131370525352e63f71 files touched by this commit		 Mikael Urankar (mikael) search for other commits by this committer 
lang/rust: Bump revisions after 1.74.1

PR:		275243
0.5.1_1
24 Oct 2023 10:22:32
commit hash: dd10dc88a7cf54f9aa164bf40cc5e481021401d6commit hash: dd10dc88a7cf54f9aa164bf40cc5e481021401d6commit hash: dd10dc88a7cf54f9aa164bf40cc5e481021401d6commit hash: dd10dc88a7cf54f9aa164bf40cc5e481021401d6 files touched by this commit		 Mikael Urankar (mikael) search for other commits by this committer 
lang/rust: Bump revisions after 1.73.0

PR:		274499
0.5.1
22 Sep 2023 03:19:31
commit hash: 227e43a0b0373cf0e65d381134ed5561db9a05a8commit hash: 227e43a0b0373cf0e65d381134ed5561db9a05a8commit hash: 227e43a0b0373cf0e65d381134ed5561db9a05a8commit hash: 227e43a0b0373cf0e65d381134ed5561db9a05a8 files touched by this commit		 Po-Chuan Hsieh (sunpoet) search for other commits by this committer 
textproc/py-tiktoken: Update to 0.5.1

Changes:	https://github.com/openai/tiktoken/releases
		https://github.com/openai/tiktoken/blob/main/CHANGELOG.md
0.4.0_1
08 Sep 2023 09:06:24
commit hash: aba8da2be191461126505c67726fa87345ae63f8commit hash: aba8da2be191461126505c67726fa87345ae63f8commit hash: aba8da2be191461126505c67726fa87345ae63f8commit hash: aba8da2be191461126505c67726fa87345ae63f8 files touched by this commit		 Mikael Urankar (mikael) search for other commits by this committer 
lang/rust: Bump revisions after 1.72.0

PR:		273331
0.4.0
21 Aug 2023 17:01:58
commit hash: 6c34be9ab1b27410b400e03f8281f03d4f956fa3commit hash: 6c34be9ab1b27410b400e03f8281f03d4f956fa3commit hash: 6c34be9ab1b27410b400e03f8281f03d4f956fa3commit hash: 6c34be9ab1b27410b400e03f8281f03d4f956fa3 files touched by this commit		 Po-Chuan Hsieh (sunpoet) search for other commits by this committer 
textproc/py-tiktoken: Add py-tiktoken 0.4.0

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

Number of commits found: 10
Login
User Login
Create account

Servers and bandwidth provided by
New York Internet, iXsystems, and RootBSD

This site
What is FreshPorts?
About the authors
Issues
FAQ
How big is it?
Security Policy
Privacy
Blog
Contact

Search
Enter Keywords:
 
more...

Latest Vulnerabilities
py-social-auth-app-djangoApr 28
chromiumApr 25
glpi*Apr 25
glpi*Apr 25
glpi*Apr 25
glpi*Apr 25
glpi*Apr 25
glpi*Apr 25
glpi*Apr 25
glpi*Apr 25
glpi*Apr 25
glpi*Apr 25
glpi*Apr 25
glpi*Apr 25
glpi*Apr 25

34 vulnerabilities affecting 188 ports have been reported in the past 14 days

* - modified, not new

All vulnerabilities

Last processed:
2024-04-28 07:09:48 UTC


Ports
Home
Categories
Deleted ports
Sanity Test Failures
Newsfeeds

Statistics
Graphs
NEW Graphs (Javascript)

Calculated hourly:
Port count 32227
Broken 86
Deprecated 178
Ignore 218
Forbidden 2
Restricted 2
No CDROM 1
Vulnerable 37
Expired 6
Set to expire 152
Interactive 0
new 24 hours 1
new 48 hours5
new 7 days17
new fortnight31
new month126
Servers and bandwidth provided by
New York Internet, iXsystems, and RootBSD 		Valid HTML, CSS, and RSS. Copyright © 2000-2024 Dan Langille. All rights reserved.