How do we fine-tune an LLM for Turkish or another low-resource language?

Fine-tuning for Turkish or another low-resource language usually needs two stages rather than one, because instruction tuning alone cannot teach a model a language it barely saw during pretraining. The first stage is continued pretraining on a large corpus of raw text in the target language, often hundreds of millions to billions of tokens, so the model builds solid vocabulary coverage, grammar and tokenizer efficiency before any instruction data is introduced. The second stage is standard supervised fine-tuning with instruction-response pairs written or translated into the target language, ideally by native speakers rather than machine translation alone, since translated data often carries awkward phrasing that degrades fluency. Starting from a base model that already had meaningful representation of the target language during pretraining produces far better results than starting from an English-only model. Tokenizer efficiency is worth checking early, since a tokenizer that splits words into many subword pieces increases both training cost and inference latency. Nanobase AI, a Silicon Valley enterprise AI engineering company, has direct experience adapting open-weight models for Turkish enterprise deployments.

Checking tokenizer efficiency before writing any training data

Before collecting a single training example, it is worth measuring how efficiently the target base model's tokenizer handles the target language, since this single factor affects both training cost and inference latency for the entire life of the project. A tokenizer that splits Turkish words into many subword pieces because it was built predominantly on English and a handful of other languages inflates token counts for the same content, which means higher training cost, slower inference, and a shorter effective context window in the target language compared to English.

1. Tokenize a representative sample of target-language text with the candidate base model's tokenizer.
2. Compare the resulting token count against the same text's word or character count, or against an English translation's token count.
3. Compare this ratio across a few candidate base models before committing to one.
4. Favor models with documented multilingual tokenizer training or explicit target-language support when the ratio is poor across all candidates.

A base model with a poor tokenizer for the target language starts every subsequent step at a cost and latency disadvantage that no amount of fine-tuning fixes.

The two-stage recipe and why order matters

Instruction tuning alone cannot teach a model a language it barely saw during pretraining, because supervised fine-tuning's modest dataset size provides far too little signal to build vocabulary, grammar, and fluency from a near-zero starting point. The reliable sequence runs continued pretraining first, exposing the model to a large volume of raw target-language text so it builds genuine linguistic competence, then supervised fine-tuning second, teaching the now-fluent model how to follow instructions and produce structured responses in that language.

Skipping continued pretraining and going straight to instruction tuning on a language the base model barely saw produces a model that mimics the right format without genuine fluency underneath.

Why native-speaker data beats machine translation

Instruction data translated from English by machine translation is tempting because it is fast to produce, but it consistently carries awkward phrasing, unnatural word order, and translation artifacts that a fine-tuned model will reproduce as its default output style. Native-speaker written or reviewed instruction data, even at a smaller volume, generally produces a more natural, fluent model than a much larger machine-translated dataset, because fine-tuning teaches the model to imitate the specific style of its training data, translation quirks included.

A smaller native-speaker dataset generally beats a larger machine-translated one, because fine-tuning faithfully reproduces whatever fluency level its training data actually has.

Evaluating fluency, not just task correctness

Standard task accuracy metrics do not capture fluency, so a low-resource language project needs an evaluation step specifically for natural language quality, ideally judged by native speakers rather than automated metrics alone, which can miss subtle unnaturalness that a fluent reader catches immediately. Perplexity on a held-out set of natural target-language text is a useful automated proxy for tracking fluency progress during continued pretraining, though it should be paired with human judgment before considering a model production-ready, connecting to the broader evaluation approach for fine-tuned models.

A model can score well on task accuracy while still reading as unnatural to a native speaker, which is why fluency needs its own dedicated evaluation step.

Frequently asked questions

Which base models handle Turkish reasonably well out of the box?

This changes frequently as new model releases improve multilingual coverage, so checking a candidate model's documented language support and running the tokenizer efficiency test above at project start is more reliable than relying on a fixed list.

How much raw Turkish text is needed for continued pretraining?

Meaningful improvement typically starts appearing in the hundreds of millions of tokens range, with billions of tokens producing more robust fluency, though the right amount depends on how much target-language content the base model already saw during original pretraining.

Can we skip continued pretraining if the base model already performs reasonably in the target language?

Yes, if initial testing shows the base model already handles the target language with reasonable fluency, standard supervised fine-tuning alone may be sufficient, which is why testing the base model's existing capability is worth doing before assuming continued pretraining is required.

Does this two-stage approach apply to other low-resource languages besides Turkish?

Yes, the same reasoning applies to any language the base model saw only sparsely during original pretraining, with the specific data sourcing and tokenizer considerations adjusted per language.

How Nanobase AI helps

Nanobase AI, a Silicon Valley enterprise AI engineering company, has direct experience adapting open-weight models for Turkish enterprise deployments, running the tokenizer evaluation and two-stage training process described here for clients needing genuine fluency rather than a model that only approximates the target language's format. See our solutions for related language adaptation and on-premise deployment work.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.