pipeline-diacritizer 1.0.2

Description:

pipelinediacritizer 1.0.2

Multi-components system for automatic Arabic diacritics restoration
About
This tool is a command-line application written in Python 3 that automatically add diacritics to
raw undiacritized Arabic text. To accomplish this task, it uses several techniques: Deep Learning, rule-based and
statistical corrections. The deep learning part was implemented using Tensorflow. It was released as a support for the
research paper:
"Multi-components system for automatic Arabic diacritization" at ECIR2020.
Installation
This tool is available as a Python 3 package pipeline-diacritizer installable through pip. For installation
instructions check the
Download and installation wiki page.
Functions
This tool has 4 main functions: preprocessing of the data, training on the processed data, testing, and
restoring the diacritics of an undiacritized text. In addition, it can calculates some statistics on a given dataset and
the ratio of Out-of-Vocabulary words in a testing set according to a train set.
This is a quick introduction to the most important ones, without mentioning all the possible options for each one. For
additional options, consider calling any subcommand with the option --help or -h (ex:
pipeline_diacritizer train --help) or check the wiki
for more details.
Preprocessing
Before feeding the new data to this application for training or testing, it needs to be converted to the standard format
of this application: one sentence per line, where a sentence is delimited by a dot, a comma, or an end of line
character.
$ pipeline_diacritizer process <source_file> <destination_file>

If the data is not yet partitioned into training, validation and testing sets, the program can help in this task using
the following command:
$ pipeline_diacritizer partition <dataset_file>

Training
To run the training and validation on selected training/validation sets, use the next command:
$ pipeline_diacritizer train --train-data <train_file> --val-data <val_file>

Testing
To evaluate the performances of the application on a testing set, use this command:
$ pipeline_diacritizer test <test_file>

Diacritization
The following command restores the diacritics of the Arabic words from the supplied text file and outputs a diacritized
copy:
$ pipeline_diacritizer diacritize <text_file>

Statistics
To get some statistics about the dataset, such as the count of tokens, arabic words, numbers... use the following
command:
$ pipeline_diacritizer stat <dataset_file>

OoV Counting
To calculate the ratio of the Out-of-Vocabulary words between the train set and the validation/test set, use the next
command:
$ pipeline_diacritizer oov <train_file> <test_file>

License
Pipeline-diacritizer code is licensed under
MIT License.