Description:

ttsmostestmturk 0.0.2

tts-mos-test-mturk







Command-line interface (CLI) to evaluate text-to-speech (TTS) mean opinion score (MOS) studies done on Amazon Mechanical Turk (MTurk).
Features

init: initialize project from .json-file
masks

create: create empty mask
mask-workers-by-id: mask workers by their WorkerId
mask-workers-by-age-group: mask workers by their age group
mask-workers-by-gender: mask workers by their gender
mask-workers-by-assignments-count: mask workers by their count of assignments
mask-workers-by-masked-ratings-count: mask workers by their count of masked ratings
mask-workers-by-correlation: mask workers by their algorithm/sentence correlation
mask-workers-by-correlation-percent: mask workers by their algorithm/sentence correlation (percentage-wise)
mask-assignments-by-id: mask assignments by their AssignmentId
mask-assignments-by-device: mask assignments by their listening device
mask-assignments-by-status: mask assignments by their status
mask-assignments-by-time: mask assignments by their submit time
mask-rating-outliers: mask outlying ratings
merge-masks: merge masks together
reverse-mask: reverse mask


stats

print-mos: print MOS and CI95
print-masking-stats: print masking statistics
print-worker-stats: print worker statistics for each algorithm
print-assignment-stats: print assignment statistics for each worker
print-sentence-stats: print sentence statistics for each algorithm
print-data: print all data points


mturk

prepare-approval: generate approval file
prepare-rejection: generate rejection file
prepare-bonus-payment: generate bonus payment file



Installation
pip install tts-mos-test-mturk --user

Usage
usage: mos-cli [-h] [-v] {init,masks,stats,mturk} ...

CLI to evaluate text-to-speech MOS studies done on MTurk.

positional arguments:
{init,masks,stats,mturk}
description
init initialize project from .json-file
masks masks commands
stats stats commands
mturk mturk commands

options:
-h, --help show this help message and exit
-v, --version show program's version number and exit

Project JSON example
{
"algorithms": [
"alg1",
"alg2",
"alg3",
"alg4"
],
"files": [
"file1",
"file2",
"file3"
],
"workers": {
"worker1": {
"gender": "male",
"age_group": "18-29",
"assignments": {
"assignment1": {
"device": "headphone",
"state": "Approved",
"hit": "hit1",
"time": "13.07.23 05:08:04",
"ratings": [
{
"algorithm": "alg1",
"file": "file1",
"votes": {
"naturalness": 3,
"intelligibility": 5
}
},
{
"algorithm": "alg2",
"file": "file3",
"votes": {
"naturalness": 2,
"intelligibility": 4
}
}
]
}
}
}
}
}

For a longer example see etc/example.json. It contains 4 algorithms and 120 files which were rated by 36 dummy workers in batches of 10 files per assignment. An example parsing of that file is under etc/example.sh.
Roadmap

add masks mask-assignments-not-of-last-month/week/day
make device, state, hit and time optional
make tax value optional

Dependencies

numpy
pandas
tqdm
ordered-set>=4.1.0
mean-opinion-score==0.0.2

Contributing
If you notice an error, please don't hesitate to open an issue.
Development setup
# update
sudo apt update
# install Python 3.8, 3.9, 3.10 & 3.11 for ensuring that tests can be run
sudo apt install python3-pip \
python3.8 python3.8-dev python3.8-distutils python3.8-venv \
python3.9 python3.9-dev python3.9-distutils python3.9-venv \
python3.10 python3.10-dev python3.10-distutils python3.10-venv \
python3.11 python3.11-dev python3.11-distutils python3.11-venv
# install pipenv for creation of virtual environments
python3.8 -m pip install pipenv --user

# check out repo
git clone https://github.com/stefantaubert/tts-mos-test-mturk.git
cd tts-mos-test-mturk
# create virtual environment
python3.8 -m pipenv install --dev

Running the tests
# first install the tool like in "Development setup"
# then, navigate into the directory of the repo (if not already done)
cd tts-mos-test-mturk
# activate environment
python3.8 -m pipenv shell
# run tests
tox

Final lines of test result output:
py38: commands succeeded
py39: commands succeeded
py310: commands succeeded
py311: commands succeeded
congratulations :)

License
MIT License
Acknowledgments

Ribeiro, F., Florêncio, D., Zhang, C., & Seltzer, M. (2011). CrowdMOS: An approach for crowdsourcing mean opinion score studies. 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2416–2419. https://doi.org/10.1109/ICASSP.2011.5946971

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID 416228727 – CRC 1410
Citation
If you want to cite this repo, you can use this BibTeX-entry generated by GitHub (see About => Cite this repository).
Taubert, S. (2024). tts-mos-test-mturk (Version 0.0.2) [Computer software]. https://doi.org/10.5281/zenodo.10461539

Changelog

v0.0.2 (2024-01-05)

Bugfix:

mask mask-workers-by-correlation-percent: sorting was not always correct


Added:

mask mask-workers-by-correlation-percent: added option to include masked workers in percentage calculation
mturk prepare-bonus-payment: added logging of fees for Mechanical Turk
mturk prepare-approval: added logging of fees for Mechanical Turk
added mask mask-assignments-by-status
added mask mask-assignments-by-time
added parsing of HITId
added option to mask assignments before preparing rejection CSV
added ! before mask name reverses mask on input
added masks mask-assignments-by-id
added masks mask-workers-by-id


Changed:

moved template creation and preparation to another repository
removed worktime parsing




v0.0.1 (2023-02-23)

Initial release

License

For personal and professional use. You cannot resell or redistribute these repositories in their original state.

Share

• Share on Facebook
• Share on Twitter