promptcraft 0.4.8

Description:

promptcraft 0.4.8

PromptCraft
A Prompt Perturbation Toolkit for Prompt Robustness Analysis



Table of Contents

Installation
Character Editing

Character Replacement
Character Deletion
Character Insertion
Character Swap
Keyboard Typos
Optical Character Recognition (OCR)


Word Manipulation

Synonym Replacement
Word Insertion
Word Swap
Word Deletion
Insert Punctuation
Word Split


Sentence Paraphrasing

Back Translation based on 🤗 Hugging Face MarianMTModel
Back Translation based on Google Translator
Paraphrasing
Formal Style
Casual Style
Passive Style
Active Style


Parallel Processing
Structure of the Code
Citation
Acknowledgement

Installation
pip install promptcraft

Character Editing
Character-level Prompt Perturbation
CharacterPerturb class for manipulating character in a sentence
from promptcraft import character

sentence = "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May."
level = 0.25 # Percentage of characters that will be edited
character_tool = character.CharacterPerturb(sentence=sentence, level=level)

Character Replacement
Randomly replace level percentage characters from the sentence
char_replace = character_tool.character_replacement()

Character Deletion
Randomly delete level percentage characters from the sentence
char_delete = character_tool.character_deletion()

Character Insertion
Randomly insert level percentage characters to the sentence
char_insert = character_tool.character_insertion()

Character Swap
Randomly swap level percentage characters in the sentence
NOTE: including self-swapping
char_swap = character_tool.character_swap()

Keyboard Typos
Randomly substitute level percentage characters in the sentence
with a randomly chosen character which is near the original character in the Keyboard (USA Full-size Layout)
NOTE:
(1) We applied keyboard_distance=1, i.e., the nearest character, number, or samples.
(2) If it is a character, we randomly chose lowercase or uppercase.
char_keyboard = character_tool.keyboard_typos()

Optical Character Recognition
Randomly substitute level percentage characters in the sentence with a common OCR map error
char_ocr = character_tool.optical_character_recognition()

Word Manipulation
Word-level Prompt Perturbation
WordPerturb class for manipulating words in a sentence
NOTE: the number of words in a sentence is only the valid words without considering spaces, special symbols, and punctuations
from promptcraft import word

sentence = "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May."
level = 0.25 # Percentage of words that will be manipulated
word_tool = word.WordPerturb(sentence=sentence, level=level)

Synonym Replacement
Randomly choose n words from the sentence that are not stop words.
Replace each of these words with one of its synonyms chosen at random.
Problem 1: Without any synonyms
Problem 2: Fewer positions than needed positions
word_synonym = word_tool.synonym_replacement()

Word Insertion
Find a random synonym of a random word in the sentence that is not a stop word.
Insert that synonym into a random position in the sentence.
Do this n times.
word_insert = word_tool.word_insertion()

Word Swap
Randomly choose two words in the sentence and swap their positions.
Do this n times.
word_swap = word_tool.word_swap()

Word Deletion
Each word in the sentence can be randomly removed with probability p.
word_delete = word_tool.word_deletion()

Insert Punctuation
Randomly insert punctuation in the sentence with probability p.
word_punctuation = word_tool.insert_punctuation()

Word Split
Randomly split a word to two tokens randomly
word_split = word_tool.word_split()

Sentence Paraphrasing
Sentence-level Prompt Perturbation
SentencePerturb class for directly manipulating a sentence
from promptcraft import sentence

sen = "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May."
sentence_tool = sentence.SentencePerturb(sentence=sen)

Back Translation by Hugging Face
Back translate the sentence (English → German → English) via 🤗 Hugging Face MarianMTModel
back_trans_hf = sentence_tool.back_translation_hugging_face()

Back Translation by Google Translator
Back translate the sentence (English → German → English) via Google Translate API
back_trans_google = sentence_tool.back_translation_google()

Paraphrasing
Paraphrasing the sentence via Parrot Paraphraser
considering
(1) Adequency: Is the meaning preserved adequately?
(2) Fluency: Is the paraphrase fluent English?
(3) Diversity: (Lexical / Phrasal / Syntactical): How much has the paraphrase changed the original sentence?
sen_paraphrase = sentence_tool.paraphrase()

Formal Style
Transform the sentence style to Formal
sen_formal = sentence_tool.formal()

Casual Style
Transform the sentence style to Casual
sen_casual = sentence_tool.casual()

Passive Style
Transform the sentence style to Passive
sen_passive = sentence_tool.passive()

Active Style
Transform the sentence style to Active
sen_active = sentence_tool.active()

Parallel Processing
Since all the methods are executed on the CPU,
they can be performed in parallel using the multiprocessing package.
Structure of the Code
At the root of the project, you will see:
.
├── LICENSE
├── README.md
├── promptcraft
│   ├── __init__.py
│   ├── character.py
│   ├── parrot.py
│   ├── sentence.py
│   ├── styleformer.py
│   └── word.py
├── setup.cfg
└── setup.py

Citation
If you find our toolkit useful, please consider citing our repo and toolkit in your publications. We provide a BibTeX entry below.
@misc{JiaPromptCraft23,
author = {Jia, Shuyue},
title = {{PromptCraft}: A Prompt Perturbation Toolkit},
year = {2023},
publisher = {GitHub},
journal = {GitHub Repository},
howpublished = {\url{https://github.com/SuperBruceJia/promptcraft}},
}

@misc{JiaAwesomeLLM23,
author = {Jia, Shuyue},
title = {Awesome {LLM} Self-Consistency},
year = {2023},
publisher = {GitHub},
journal = {GitHub Repository},
howpublished = {\url{https://github.com/SuperBruceJia/Awesome-LLM-Self-Consistency}},
}

@misc{JiaAwesomeSTS23,
author = {Jia, Shuyue},
title = {Awesome Semantic Textual Similarity},
year = {2023},
publisher = {GitHub},
journal = {GitHub Repository},
howpublished = {\url{https://github.com/SuperBruceJia/Awesome-Semantic-Textual-Similarity}},
}

Acknowledgement
This work was finished during my 2023 fall semester research rotation
at the Department of Electrical and Computer Engineering, Boston University.