Crossing Linguistic Horizons

Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models

Zero-Shot Text Classification Leaderboard

Models UiT-VSMEC PhoATIS
AC↑ F1↑ AR↑ ECE↓ A@10↑ AC↑ F1↑ AR↑ ECE↓ A@10↑
URA-LLaMa 70B 0.40 ± 0.02 0.32 ± 0.02 0.68 ± 0.01 0.14 ± 0.02 0.60 ± 0.06 0.56 ± 0.02 0.48 ± 0.03 0.85 ± 0.00 0.25 ± 0.02 0.56 ± 0.06
URA-LLaMa 13B 0.29 ± 0.02 0.25 ± 0.02 0.52 ± 0.01 0.09 ± 0.01 0.23 ± 0.05 0.10 ± 0.01 0.10 ± 0.01 0.72 ± 0.00 0.52 ± 0.01 0.14 ± 0.04
URA-LLaMa 7B 0.13 ± 0.01 0.11 ± 0.01 0.50 ± 0.01 0.15 ± 0.01 0.21 ± 0.05 0.04 ± 0.01 0.04 ± 0.02 0.77 ± 0.00 0.30 ± 0.01 0.04 ± 0.02
LLaMa-2 13B 0.11 ± 0.01 0.10 ± 0.01 0.49 ± 0.01 0.31 ± 0.01 0.09 ± 0.04 0.03 ± 0.01 0.02 ± 0.00 0.45 ± 0.01 0.28 ± 0.01 0.03 ± 0.02
LLaMa-2 7B 0.07 ± 0.01 0.08 ± 0.01 0.52 ± 0.01 0.35 ± 0.01 0.07 ± 0.03 0.00 ± 0.06 0.00 ± 0.06 0.61 ± 0.01 0.32 ± 0.00 0.00 ± 0.00
Vietcuna 7B 0.05 ± 0.01 0.02 ± 0.01 0.52 ± 0.01 0.95 ± 0.01 0.03 ± 0.02 0.05 ± 0.01 0.01 ± 0.00 0.66 ± 0.00 0.20 ± 0.01 0.01 ± 0.21
GPT-3.5 0.43 ± 0.02 0.37 ± 0.02 - 0.29 ± 0.02 0.43 ± 0.06 0.44 ± 0.02 0.38 ± 0.03 - 0.38 ± 0.02 0.44 ± 0.05
GPT-4 0.49 ± 0.02 0.46 ± 0.02 - 0.35 ± 0.02 0.50 ± 0.06 0.89 ± 0.01 0.69 ± 0.02 - 0.83 ± 0.01 0.89 ± 0.03