GitLocker: The Coding Marketplace

Description:

openagents 0.1.0

Mask-Predict
Download model

Description
Dataset
Model

MASK-PREDICT
[WMT14 English-German]
download (.tar.bz2)

MASK-PREDICT
[WMT14 German-English]
download (.tar.bz2)

MASK-PREDICT
[WMT16 English-Romanian]
download (.tar.bz2)

MASK-PREDICT
[WMT16 Romanian-English]
download (.tar.bz2)

MASK-PREDICT
[WMT17 English-Chinese]
download (.tar.bz2)

MASK-PREDICT
[WMT17 Chinese-English]
download (.tar.bz2)

Preprocess
text=PATH_YOUR_DATA
output_dir=PATH_YOUR_OUTPUT
src=source_language
tgt=target_language
model_path=PATH_TO_MASKPREDICT_MODEL_DIR
python preprocess.py --source-lang src−−target−lang{tgt} --trainpref text/train−−validpreftext/valid --testpref text/test−−destdir{output_dir}/data-bin --workers 60 --srcdict Missing superscript or subscript argumentMissing superscript or subscript argument{src}tgt/dict.{src}.txt --tgtdict ${model_path}/maskPredictMissing superscript or subscript argumentMissing superscript or subscript argument{tgt}/dict.${tgt}.txt
Train
model_dir=PLACE_TO_SAVE_YOUR_MODEL
python train.py outputdir/data−bin−−archberttransformerseq2seq−−share−all−embeddings−−criterionlabelsmoothedlengthcrossentropy−−label−smoothing0.1−−lr5e−4−−warmup−init−lr1e−7−−min−lr1e−9−−lr−schedulerinversesqrt−−warmup−updates10000−−optimizeradam−−adam−betas′(0.9,0.999)′−−adam−eps1e−6−−tasktranslationself−−max−tokens8192−−weight−decay0.01−−dropout0.3−−encoder−layers6−−encoder−embed−dim512−−decoder−layers6−−decoder−embed−dim512−−fp16−−max−source−positions10000−−max−target−positions10000−−max−update300000−−seed0−−save−dir{model_dir}
Evaluation
python generate_cmlm.py outputdir/data−bin−−path{model_dir}/checkpoint_best_average.pt --task translation_self --remove-bpe --max-sentences 20 --decoding-iterations 10 --decoding-strategy mask_predict
License
MASK-PREDICT is CC-BY-NC 4.0.
The license applies to the pre-trained models as well.
Citation
Please cite as:
@inproceedings{ghazvininejad2019MaskPredict,
title = {Mask-Predict: Parallel Decoding of Conditional Masked Language Models},
author = {Marjan Ghazvininejad, Omer Levy, Yinhan Liu, Luke Zettlemoyer},
booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing},
year = {2019},
}