GitLocker: The Coding Marketplace

Description:

pronunciationdictionary 0.0.6

pronunciation-dictionary

Library to save and load pronunciation dictionaries (language-independent).
Features

Load dictionary from file or URL

Parsing of

line comments
pronunciation comments
numbers indicating alternative pronunciations for words
weights

Multiprocessing for faster deserialization

Save dictionary to file

including numbers for alternative pronunciations
include weights
set word/weight/pronunciation separator

Select pronunciation via

first/last
longest/shortest
highest/lowest weight
random
weight

Get phoneme set

Example dictionaries and deserialization arguments

Montreal Forced Aligner dictionaries

encoding: "UTF-8"

CMU

encoding: "ISO-8859-1"
consider_numbers: True
consider_pronunciation_comments: True

LibriSpeech

encoding: "UTF-8"

Prosodylab
Old: CMU 0.7b

encoding: "ISO-8859-1"
consider_comments: True
consider_numbers: True

Excerpt from CMU (as example)
a.d. EY2 D IY1
a.m. EY2 EH1 M
a.s EY1 Z
aaa T R IH2 P AH0 L EY1
aaberg AA1 B ER0 G
aachen AA1 K AH0 N
aachener AA1 K AH0 N ER0
aaker AA1 K ER0
aalborg AO1 L B AO0 R G # place, danish
aalborg(2) AA1 L B AO0 R G

Installation
pip install pronunciation-dictionary --user

Usage
from pronunciation_dictionary import load_dict, save_dict, MultiprocessingOptions, DeserializationOptions, SerializationOptions

Example
from pathlib import Path

from pronunciation_dictionary import (DeserializationOptions,
MultiprocessingOptions, SerializationOptions,
get_phoneme_set, load_dict_from_url, save_dict)

dictionary = load_dict_from_url(
"https://raw.githubusercontent.com/cmusphinx/cmudict/master/cmudict.dict",
"ISO-8859-1",
DeserializationOptions(False, True, True, False),
MultiprocessingOptions(4, None, 10000)
)

phoneme_set = get_phoneme_set(dictionary)

print(phoneme_set)
# {'Z', 'EY1', 'AH0', 'F', 'AE0', 'UW0', 'CH', 'G', 'V', 'AY1', 'AO2', 'ZH', 'AA1', 'IY1', 'AW0', 'T', 'TH', 'AY2', 'DH', 'S', 'W', 'ER1', 'AA2', 'AE2', 'AE1', 'AW1', 'UW1', 'AH1', 'Y', 'EY2', 'AO0', 'OW2', 'OY2', 'IY2', 'JH', 'N', 'NG', 'P', 'IH2', 'M', 'OW0', 'L', 'UH1', 'IY0', 'EY0', 'HH', 'IH0', 'SH', 'AH2', 'AW2', 'EH2', 'OW1', 'D', 'R', 'IH1', 'AO1', 'B', 'UH2', 'UH0', 'ER0', 'UW2', 'ER2', 'EH0', 'AY0', 'AA0', 'EH1', 'OY1', 'OY0', 'K'}

pronunciations_distmantle = dictionary.get("dismantle")

for pronunciation, weight in pronunciations_distmantle.items():
print(pronunciation, weight)
# ('D', 'IH0', 'S', 'M', 'AE1', 'N', 'T', 'AH0', 'L') 1.0
# ('D', 'IH0', 'S', 'M', 'AE1', 'N', 'AH0', 'L') 1.0

save_dict(dictionary, Path("/tmp/cmu.dict"), "UTF-8",
SerializationOptions("DOUBLE-SPACE", False, False))

head /tmp/cmu.dict
# 'bout B AW1 T
# 'cause K AH0 Z
# 'course K AO1 R S
# 'cuse K Y UW1 Z
# 'em AH0 M
# 'frisco F R IH1 S K OW0
# 'gain G EH1 N
# 'kay K EY1
# 'm AH0 M
# 'n AH0 N

Roadmap

replace SerializationOptions, DeserializationOptions and MultiprocessingOptions with parameters
add default parameter values
add more tests

Development setup
# update
sudo apt update
# install Python 3.8, 3.9, 3.10 & 3.11 for ensuring that tests can be run
sudo apt install python3-pip \
python3.8 python3.8-dev python3.8-distutils python3.8-venv \
python3.9 python3.9-dev python3.9-distutils python3.9-venv \
python3.10 python3.10-dev python3.10-distutils python3.10-venv \
python3.11 python3.11-dev python3.11-distutils python3.11-venv \
python3.12 python3.12-dev python3.12-distutils python3.12-venv
# install pipenv for creation of virtual environments
python3.8 -m pip install pipenv --user

# check out repo
git clone https://github.com/stefantaubert/pronunciation-dictionary.git
cd pronunciation-dictionary
# create virtual environment
python3.8 -m pipenv install --dev

Running the tests
# first install the tool like in "Development setup"
# then, navigate into the directory of the repo (if not already done)
cd pronunciation-dictionary
# activate environment
python3.8 -m pipenv shell
# run tests
tox

Final lines of test result output:
py38: commands succeeded
py39: commands succeeded
py310: commands succeeded
py311: commands succeeded
py312: commands succeeded
congratulations :)

License
MIT License
Acknowledgments
Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID 416228727 – CRC 1410
Citation
If you want to cite this repo, you can use this BibTeX-entry generated by GitHub (see About => Cite this repository).
Taubert, S. (2024). pronunciation-dictionary (Version 0.0.6) [Computer software]. https://doi.org/10.5281/zenodo.7386813