tokenizerx

Description:

tokenizerx

Pure Dart tokenizer for RWKV project, EleutherAI neox, and pythia #
Setup #
flutter pub add tokenizerx
copied to clipboard
Usage #
import 'package:tokenizerx/tokenizerx.dart';

final tokenizer = Tokenizer();

// Encode into token int : [12092, 3645, 2]
final tokens = tokenizer.encode("Hello World!");

// Decode back to "Hello World!"
final decoded = tokenizer.decode(tokens);
copied to clipboard
references #
This code was inpired in the python implementation of @saharNooby https://gist.github.com/saharNooby/bb54519a7d3735afb6949825608c00f0 and the javascript implementation of @PicoCreator https://github.com/RWKV/RWKV-tokenizer-node/blob/main/src/tokenizer.js