japanese_word_tokenizer

Description:

japanese word tokenizer

Japanese Word Tokenizer #
A lightweight Japanese word tokenizer library for Dart and Flutter based on wakachigaki.
Features #

Tokenizes Japanese text into individual words
Lightweight and fast
Easy to use and integrate into Dart and Flutter projects
Based on the wakachigaki tokenizer algorithm by Yushak Inoue

Installation #
Add the following dependency to your pubspec.yaml file:
dependencies:
japanese_word_tokenizer: ^0.0.3

copied to clipboard
Sample usage
Import the package in your Dart code:
import 'package:japanese_word_tokenizer/japanese_word_tokenizer.dart';
copied to clipboard
In your main.dart
void main() {
String text = 'ここでテキストを分かち書きします';
List<dynamic> tokens = tokenize(text);
print(tokens);
}
copied to clipboard
Sample output
[ここで, テキスト, を, 分か, ち, 書き, します]
copied to clipboard