Description:

tokenizer

tokenizer #


Converts a stream of strings into stream of tokens splitted by delimiters
Installation #
dart pub add tokenizer
copied to clipboard
or
flutter pub add tokenizer
copied to clipboard
Example #
import 'dart:async';

import 'package:tokenizer/tokenizer.dart';

void main(List<String> args) async {
final string = 'Hello, world';
final tokenizer = Tokenizer({',', ' '});
final c = StreamController<String>();

c.add(string);
c.close();

final tokens = await c.stream.transform(tokenizer.transformer).toList();
print(tokens); // ["Hello", ",", " ", "world"]
}
copied to clipboard
License #
MIT

License

For personal and professional use. You cannot resell or redistribute these repositories in their original state.

Share

• Share on Facebook
• Share on Twitter