T

Install Tokenizer with winget

Microsoft.Tokenizer · Microsoft Corporation

Install command

winget install -e --id Microsoft.Tokenizer

About Tokenizer

Contains Typescript and C# implementation of byte pair encoding (BPE) tokenizer for OpenAI LLMs, it is based on open sourced rust implementation in the OpenAI tiktoken.

Version
v1.3.3
Publisher
Microsoft Corporation
Website
github.com
Source
winget-pkgs manifest
Download
Direct installer (.zip)
License
MIT
Min OS
Windows 10+
Tags
Microsoft.ML.TokenizersMicrosoft.DeepDev.TokenizerLibtypescripttokenizationrequirescmd
Catalog refreshed: 5/28/2026

Versions

1.3.3latest

Related

7zr
7zr
7zip.7zr
A
Activity
Drrakendu78.Activity
A
AdGuard DNS CLI
AdGuard.AdGuardDNSCLI
A
Amethyst
Geoxor.Amethyst

More from Microsoft Corporation

A
AI Shell
Microsoft.AIShell
A
APM – Agent Package Manager
Microsoft.APM
Account Lockout Status
Account Lockout Status
Microsoft.AccountLockoutStatus
Active Directory Rights Management Services Client 2.1
Active Directory Rights Management Services Client 2.1
Microsoft.RMSClient
A
App Installer
Microsoft.AppInstaller
Aspire CLI
Aspire CLI
Microsoft.Aspire
Attack Surface Reduction Test Tool
Attack Surface Reduction Test Tool
Microsoft.ASRTestTool
A
Azure Developer CLI
Microsoft.Azd