flutter_tiktoken

Description:

flutter tiktoken

flutter_tiktoken #
flutter_tiktoken is a flutter offline package for a fast BPE tokeniser for OpenAI models.
flutter_tiktoken is a copy package in https://github.com/mvitlov/tiktoken and fixed the issue build spent many time (more than one hour) when build in release mode.



Open Ai Official Calcuate
flutter_tiktoken









Usage #
init flutter_tiktoken
void main() async {
WidgetsFlutterBinding.ensureInitialized();
await TiktokenDataProcessCenter().initata();
runApp(const MyApp());
}
copied to clipboard
calcuateTokens
static int _calcuateTokens(
{required String text, String modelName = 'gpt-4'}) {
final encoding = encodingForModel(modelName);
final numTokens = encoding.encode(text).length;
return numTokens;
}
copied to clipboard