ViLLM

What We Do

ViLLM is an end-to-end framework for finetuning, evaluation, and deployment of Vietnamese large language models.
The framework is designed to be easy to use, flexible, and efficient.
This study encompasses 10 tasks with 14 models, 20 datasets, and 31 distinct evaluation metrics.

Model
GPT-4 GPT-3.5 Gemini LLaMa-70B LLaMa-13B LLaMa-7B URA-LLaMa-70B URA-LLaMa-13B URA-LLaMa-7B MixSUra GemSUra-7B GemSUra-2B Vietcuna-7B Vistral

Task	Dataset	Metric
Question Answering	XQuaD	Exact Match, F1 Score
Question Answering	MLQA	Exact Match, F1 Score
Text Summarization	VietNews	ROUGE-1, ROUGE-2, ROUGE-L, SummaC, BERTScore, Coverage, Density, Compression
Text Summarization	WikiLingua
Sentiment Analysis	VLSP 2016	Accuracy, F1 Score, AUC ROC, Expected Calibration Error at top-10, Accuracy at 10% coverage
Sentiment Analysis	UiT-VSFC
Text Classification	UiT-VSMEC	Accuracy, F1 Score, AUC ROC, Expected Calibration Error at top-10, Accuracy at 10% coverage
Text Classification	PhoATIS
Knowledge	ZaloE2E	Exact Match, F1 Score
Knowledge	ViMMRC	Accuracy, F1 Score, AUC ROC, Expected Calibration Error at top-10, Accuracy at 10% coverage
Toxicity Detection	UiT-ViCTSD	Accuracy, F1 Score, AUC ROC, Expected Calibration Error at top-10, Accuracy at 10% coverage
Toxicity Detection	UiT-ViHSD
Information Retrieval	mMARCO	Mean Reciprocal Rank in top-10, Boosted Mean Reciprocal Rank in top-10, Normalized Discounted Cumulative Gain in top-10, Boosted Normalized Discounted Cumulative Gain in top-10
Information Retrieval	mRobust04
Language Modeling	MLQA-MLM	Exact Match, Character Error Rate, Word Error Rate, Character Edit Distance, Word Edit Distance, Perplexity
Language Modeling	VSEC
Reasoning	Synthetic Reasoning - Natural Language	Exact Match, F1 Score, Equivalent
	Synthetic Reasoning - Abstract Symbol
	MATH
Machine Translation	PhoMT	Bilingual Evaluation Understudy, hLEPOR
Machine Translation	OPUS100	Bilingual Evaluation Understudy, hLEPOR
Bias & Toxicity in generation	XQuaD	Demographic Representations of Races, Demographic Representations of Genders, Stereotypical Associations of Races, Stereotypical Associations of genders, Toxicity score
	MLQA
	VietNews
	WikiLingua
	PhoMT
	OPUS100