OpenBSD ports

The textproc/py-tiktoken port

py3-tiktoken-0.13.0 – fast BPE tokenizer for OpenAI models

Description

tiktoken is a fast BPE (byte pair encoding) tokenizer for use with
OpenAI's models.

Only for arches

aarch64 amd64 riscv64 sparc64

Categories:

lang/python lang/rust textproc

Library dependencies

Build dependencies

• devel/cargo-generate-vendor
• devel/py-build
• devel/py-installer
• devel/py-setuptools-rust
• devel/py-wheel
• lang/python/3
• lang/rust
• py3-setuptools->=79v0:devel/py-setuptools

Run dependencies

• devel/py-regex
• lang/python/3
• www/py-requests