Introducing Telugu LLM Labs! An independent joint initiative by @ravithejads and me to advance the Telugu NLP space and give a better AI experience for 100 million Telugu speakers in the world! Link: huggingface.co/Telugu-LLM-La… Goal: > Contribute to open datasets focusing on Telugu (native script + romanized). > Share our experiments (models, embeddings, etc) building LLMs and SLMs focused on Telugu. We are starting with the release of two dataset contributions today! 1. Romanized Telugu Pretraining dataset Much of the online conversations (eg: WhatsApp or YouTube comments) happen in romanized Telugu! Instead of typing "ఎలా ఉన్నారు?" (How are you?), people type "ela unnaru?" using a romanized script for most online interactions. Datasets in romanized telugu are scarce whether one wants to do additional pretraining on a model or finetune it. Most datasets are available only in Telugu native script. To this effect, we are releasing "uonlp_culturaX_telugu_romanized_100k", where we have romanized the first 108k rows from the culturaX_telugu dataset and it is available on @huggingface This dataset is ideal if you want to do additional pre-training for CLM (casual language model/next-word prediction) for a tiny LLM like TinyLlama 1.1B. 2. SFT (Supervised Finetuning Dataset) In Telugu (native + romanized) If you want to fine-tune an existing language model the most popular way is to collect/curate instruction datasets and train on these prompt/response pairs. However, there is a severe scarcity of instruction datasets in Indic languages. One way to do it is via translation where existing English Instruction datasets are translated to Telugu etc. But there are challenges as there are grammar prompts, code prompts, etc that are not relevant or incorrectly translated (eg: code for loops) to Telugu or other native languages. To this effect, we are releasing two datasets "yahma_alpaca_cleaned_telugu_filtered_and_romanized" and "teknium_GPTeacher_general_instruct_telugu_filtered_and_romanized" where we have taken the base datasets (understood from prefix) and translated as well as transliterated the rows to Telugu. We have further filtered them with NLP classification systems to remove the rows that are "English Language Specific" or "Coding related" so that the resultant dataset is cleaner and more contextual. All of these are released on Huggingface Hub in the "Telugu-LLM-Labs" space. We are experimenting with the training of open-source models like Llama 2, Mistral, and TinyLlama on these Telugu translation and transliteration datasets. Please stay tuned for more updates!