Fine-Tuning interview questions & answers

Ek new hire ko speed pe laane ke do tareeke socho. Method one: har single shift pe unhe ek fresh checklist do -- wo aaj perfectly follow karenge, lekin agar kal wo checklist nahi di toh unhe kuch pata nahi hoga ki karna kya hai. Method two: unhe mahino tak floor pe drills karwao jab tak sahi response muscle memory na ban jaaye -- ab tu chupke se checklist hata bhi de toh wo sahi hi karenge, kyunki unka react karne ka tareeka permanently badal chuka hai.

Ab tak is course me tune jo bhi seekha hai wo method one hai, scale pe. Ek prompt ek checklist hai jo tu model ko har single call pe deta hai -- behtar instructions, lekin har baar dobara explain karni padti hai. RAG (Ch4) model ko ek reference binder (retrieved documents) deta hai jise wo us ek call ke dauraan flip kar sakta hai, phir bhool jaata hai. Dono me se koi bhi model ko khud nahi badalta; dono baar-baar, har call pe dobara supply hote hain.

Fine-tuning method two hai. Tu model ko chaahe gaye behaviour ke sau-sau examples dikhata hai, aur use wapas prompt ki tarah feed karne ke bajaye, un pe model ki training continue karta hai -- uske andar ke weights permanently shift ho jaate hain, isliye wo behaviour ab baked in ho jaata hai. Ab tujhe har call pe use dobara explain nahi karna padta.

OpenAI ki fine-tuning API me, ye shift literal hoti hai: tu examples ki ek file upload karta hai, usi pe ek training job chalti hai, aur wo ek bilkul nayi model ID return karti hai. Us model ID ko call karo aur trained behaviour bas... wahaan hota hai, pehle se kaafi chhote prompt ke saath.

Yahaan wo trap hai jisme almost har beginner phasta hai: "chalo isse hamare product manual pe fine-tune kar dete hain taaki isse hamare products pata ho jaayein." Ye fine-tuning ka kaam nahi hai. Facts pe train karna ek reliable lookup nahi banata -- ye ek probability distribution shift karta hai, isliye model aksar confident sound karega jabki product facts galat bata raha hoga, aur jo usne 'seekha' hai use tu cite ya update nahi kar sakta. Fine-tuning model ko kaise respond karna hai ye sikhati hai -- uska tone, uska format, ek narrow task pe uski habits -- usse kya pata hai wo nahi. Facts jo current, correct aur citable hone chahiye, wo RAG ka kaam hai, fine-tuning ka nahi.

Yahi se is poore chapter ki ladder milti hai: pehle ek behtar prompt try karo, phir agar current facts chahiye toh RAG use karo, aur fine-tuning sabse aakhir me reach karo -- sirf tab jab in dono se wo consistency na mile jo tujhe chahiye.

🌍 Real-world example: ek support team chahti hai ki har reply unke brand jaisa sound kare -- warm, teen sentences, hamesha ek follow-up offer karta hua. Har prompt me ek lamba style guide paste karne ke bajaye, wo model ko us voice ke sau-sau examples pe fine-tune karte hain; neeche wale code ka short prompt wahi hai jo fine-tuned model ko lambe wale ki jagah chahiye hota hai.

💡 Fine-tuning = model ki training ko apne khud ke examples pe continue karna taaki uske weights ek chaahe gaye behaviour ki taraf shift ho jaayein.

💡 Weights = model ke internal numbers; fine-tuning inhe badalne ka process hai -- prompting aur RAG inhe kabhi nahi badalte.

💡 The ladder = prompting -> RAG -> fine-tuning, effort aur cost ke order me; agla rung tabhi chado jab neeche wala rung tujhe wo consistency na de sake jo chahiye.

Standard definition (interview me bolo): Fine-tuning is the process of continuing a pretrained model's training on your own labelled examples so its weights shift toward a specific behaviour; unlike prompting or RAG, which supply information at request time and are re-sent on every call, fine-tuning bakes that behaviour into the model itself -- it changes how the model responds, not what it factually knows, which is why it complements RAG rather than replacing it.

import tiktoken

# The long instruction-heavy system prompt you'd need WITHOUT fine-tuning --
# every rule spelled out, every single call.
long_system_prompt = """You are a customer support assistant for an e-commerce company.
Always respond in a warm, friendly, and professional tone.
Never use slang or overly casual language.
Always format your response as a short paragraph, no bullet points.
Always end your response by asking if there is anything else you can help with.
Never mention competitor brands or products.
Always confirm the customer's order number before discussing order details.
If the customer sounds upset, acknowledge their frustration before solving the problem.
Keep responses under 150 words unless the customer asks for more detail.
Never make promises about refund timelines outside official policy."""

# The short prompt a FINE-TUNED model needs instead -- the voice/tone/format rules
# above are now baked into the weights, so they don't have to be re-explained.
short_prompt_for_finetuned_model = "Customer: My order hasn't arrived yet, what do I do?"

# tiktoken is OpenAI's own tokenizer -- it counts tokens the way OpenAI models see
# them. It is not a general-purpose tokenizer: other model families use their own
# tokenizers and would count the same text differently.
enc = tiktoken.get_encoding("o200k_base")

long_tokens = enc.encode(long_system_prompt)
short_tokens = enc.encode(short_prompt_for_finetuned_model)

print("long instruction-heavy system prompt tokens:", len(long_tokens))
print("short prompt (fine-tuned model) tokens:", len(short_tokens))
print("per-call token saving:", len(long_tokens) - len(short_tokens))

Ek naye support hire ko onboard karna socho. Tu (a) unhe ek cheat-sheet de sakta hai jo wo har call se pehle re-read kare, (b) unhe past tickets ka ek folder de sakta hai jo per-call flip through kare, ya (c) unhe hafton ki training de sakta hai jab tak company ki voice aur rules unke instinct me na aa jaayein, koi cheat-sheet ki zaroorat hi na rahe. Ye, order me, prompting, RAG, aur fine-tuning hain -- aur is ladder pe tu ek-ek rung chadhta hai, sirf tab jab neeche wala rung tujhe zaroori consistency na de paaye.

API me, ye directly map hota hai: prompting = instructions jo tu har call resend karta hai; RAG = documents jo tu retrieve karke har call me paste karta hai; fine-tuning = weight updates jo ek baar bake ho jaate hain, isliye model bina kisi ke bhi wahi behavior karta hai. Har upar wala rung neeche wale se zyada setup aur maintenance cost karta hai, isliye zyadatar teams ko actually fine-tune karne ki zaroorat kabhi padti hi nahi -- pehle prompting try karo, phir RAG, aur fine-tuning tabhi lo jab dono me se koi bhi zaroori consistency na de paaye.

Kya chahiye?                        Kya use karo
---------------                     ------------
Apne docs pe grounded answers   ->  RAG (NOT fine-tuning!)
Consistent style/tone            ->  Fine-tuning
Ek specific output format         ->  Pehle prompt engineering
Ek narrow, repeated task          ->  Fine-tuning (agar prompting/RAG nahi kaam kare)
Sasta/chhota model                ->  Ek chhote model ko fine-tune karo bade jaisa banane ke liye
Chhote prompts, kam latency        ->  Fine-tuning (behaviour bake ho gaya, kam repeat karna)

                 | Prompting     | RAG                    | Fine-tuning
-----------------|---------------|------------------------|-----------------------------
Setup time       | Minutes       | Ghante                 | Ghanto-se-din tak
Data chahiye      | Kuch nahi     | Tere documents          | Curated example pairs (aim
                 |               |                        | karo kuch sau consistent ones ka)
Data freshness   | Static        | Document add karte hi   | Frozen jab tak retrain
                 |               | update ho jaata hai     | na karo
Best for         | General       | Tere docs se facts       | Voice/tone, format, narrow
                 | instructions  |                        | task behaviour
Pehle try karo   | FIRST         | SECOND                  | LAST

Fine-tuning genuinely tab worth hai jab: ek consistent brand voice hazaaron replies me chahiye, ek rigid output format har baar chahiye, ek narrow repeated task hai, ya ek chhote, sasta model ko itna smart banana hai ki wo bade ki jagah le sake. Kab NAHI: jab tujhe apne khud ke documents se fresh ya factual answers chahiye (ye RAG ka kaam hai, fine-tuning ka nahi), ya tune abhi tak prompt tune karne ki koshish hi nahi ki.

Ek production pattern jo naam se jaanna zaroori hai: distillation. Iske general form me, tu ek chhote model ko train karta hai ki wo ek bade, mehenge model ki poori output distribution (soft targets, sirf final answer nahi) match kare -- yehi tareeka hai jisse distil* model family banayi gayi thi. Version jo tujhe job pe milega wo simpler hai: bade model ke answers ko kaafi inputs pe sample karo, phir chhote model ko un sampled answers pe supervised-fine-tune karo -- ye bas ordinary fine-tuning (SFT) hai, human-written labels ki jagah ek bade model ke outputs ko "gold" labels ki tarah use karke.

🌍 Real-world example: ek team sasti customer-support replies chahti hai. Har ticket pe bade model ko pay karne ki jagah, wo usse kuch sau achhi replies generate karwate hain, phir un replies pe ek chhote model ko fine-tune (ya distil) karte hain -- same voice aur quality, per-reply bahut kam compute.

💡 Prompting = instructions jo har call resend hote hain; zero setup, lekin har baar repeat hote hain aur documents ya facts ko reliably hold nahi kar sakte.

💡 RAG = relevant documents retrieve karke har call me prompt me paste karna, taaki answers grounded aur fresh reh sakein.

💡 Fine-tuning = apne examples pe training continue karna taaki behaviour weights me bake ho jaaye, har call re-explain karne ki jagah.

💡 Distillation = ek chhote model ko ek bade model jaisa banane ki training -- ya toh uski poori output distribution (general/technical form), ya, practice me zyada common, uske sampled answers ko SFT training data ki tarah use karke.

Standard definition (interview me bolo): Fine-tuning is the last rung of a ladder that starts with prompting and RAG -- reach for it only when you need consistent style, a rigid output format, a narrow repeated task, or a smaller model made good enough to replace a bigger one, since fresh or factual answers belong to RAG and every rung above prompting costs more setup and ongoing maintenance than the one below it.

import tiktoken

# A small illustrative brand-voice fine-tuning dataset (chat-style examples).
dataset = [
    {"messages": [
        {"role": "system", "content": "You are a friendly brand-voice support agent."},
        {"role": "user", "content": "My order hasn't arrived yet."},
        {"role": "assistant", "content": "So sorry about that! Let's track it down together - could you share your order number?"},
    ]},
    {"messages": [
        {"role": "system", "content": "You are a friendly brand-voice support agent."},
        {"role": "user", "content": "Can I get a refund?"},
        {"role": "assistant", "content": "Of course! I'll get that refund started right away - just confirm the order number for me."},
    ]},
    {"messages": [
        {"role": "system", "content": "You are a friendly brand-voice support agent."},
        {"role": "user", "content": "The product arrived damaged."},
        {"role": "assistant", "content": "Oh no, that's not the experience we want for you! Let's get a replacement sent out today."},
    ]},
    {"messages": [
        {"role": "system", "content": "You are a friendly brand-voice support agent."},
        {"role": "user", "content": "How long does shipping take?"},
        {"role": "assistant", "content": "Great question! Shipping usually takes 3-5 business days, and I'll share tracking as soon as it ships."},
    ]},
]

# tiktoken is OpenAI-only -- it does not count tokens for Claude or open-source
# models. This estimate is scoped to the OpenAI fine-tuning path only.
enc = tiktoken.get_encoding("o200k_base")

def count_example_tokens(example):
    return sum(len(enc.encode(m["content"])) for m in example["messages"])

per_example = [count_example_tokens(ex) for ex in dataset]
dataset_tokens = sum(per_example)
print("per-example tokens:", per_example)
print("dataset total:", dataset_tokens, "tokens")

for n_epochs in (1, 3):
    print(f"n_epochs={n_epochs} -> training tokens (dataset_tokens x epochs):", dataset_tokens * n_epochs)

Kya bana rahe hain: ek brand-voice support bot ek fictional cloud-storage product ke liye, "Nimbus" -- is chapter ka FREE, first-impression project. Ye poora fine-tuning workflow end to end hai, sirf ek step chhod ke jo ise padhne wala koi bhi actually perform nahi kar sakta: ek paid hosted training job submit karna. Baaki sab -- voice design karna, examples curate karna, JSONL banana, use validate karna, train/val split karna, training cost ko tokens mein estimate karna, aur hosted API ki expect ki hui exact job payload construct karna -- genuinely neeche C:/lcv pe chalta hai, zero API key ke saath.

Ye project assume karta hai ki tumne is chapter ke topics 3-6 (dataset prep, validation, hosted API, hyperparameters) abhi tak nahi dekhe -- sab kuch yahan zero se re-derive kiya gaya hai, self-contained.

---

Step 0 -- Ek-line framing (pehle padho)

Fine-tuning ye badalta hai ki model kaise respond karta hai, kya jaanta hai wo nahi. Ise 200 replies ek consistent voice mein dikhao aur ye us voice mein likhna shuru kar deta hai -- ye tumhari refund policy ya ship dates verifiable facts ki tarah NAHI seekhta (wo RAG ka kaam hai). To ye project jaan-boojh kar voice, tone aur format ke baare mein hai, kabhi bhi bot ko nayi information sikhaane ke baare mein nahi. Neeche har example ka jawab wo human agent bhi de sakta hai jo facts pehle se jaanta hai; hum WOH encode kar rahe hain KAISE wo use phrase karega.

💡 JSONL = ek text file mein har line pe ek JSON object (JSON array NAHI). Har line ek training example hai. 💡 Validator = ek script jo JSONL file ko structural errors ke liye check karta hai, use train karne pe paisa kharch karne SE PEHLE. 💡 Job payload = wo dict of arguments jo ek hosted fine-tuning API call expect karta hai -- ise construct karna aur bhejna do alag cheezein hain.

---

Step 1 -- Consistent voice design karo

Ek bhi example likhne se pehle, voice ko ek chhoti si spec ke roop mein likh do jis se har example check kiya jaa sake:

brand: Nimbus (cloud storage support)
tone: warm, plain-English, confident
banned words: "unfortunately", "as per policy", "kindly", "team will look into it"
rules: hamesha EK concrete next step batao; "we" bolo kabhi "the team" nahi;
       koi corporate jargon ya hedging nahi; replies ~60 words se kam rakho

Ye code se zyada important kyun hai: ek fine-tuned model sirf utna hi consistent ho sakta hai jitne tumhare examples hain. Agar aadhe examples "unfortunately" bolte hain aur aadhe nahi, model ek blurry average voice seekhta hai, ek sharp voice nahi. Spec ko EK fixed system prompt mein badlo aur wahi exact string har ek training example mein reuse karo -- examples ke beech alag-alag system prompts hona ek sabse common wajah hai jo fine-tune ko "kaam nahi karta" bana deti hai.

---

Step 2 -- Example set curate karo

Chhe chhote support conversations, har ek spec follow karta hua: ek refund, ek sync bug, ek downgrade sawaal, ek account-deletion request, ek crash report, aur ek feature request. Har example ek messages list hai -- system (fixed voice prompt), user (customer ka message), assistant (on-voice reply). Ek real project ko kuch sau consistent examples chahiye honge, hazaron sloppy examples ki jagah -- quality aur consistency volume se kahin zyada matter karte hain -- par chhe is poore pipeline ka har step real mein prove karne ke liye kaafi hai.

Ek rule jo abhi internalize karna zaroori hai, kyunki ye ek common beginner trap hai: kabhi bhi apni validation examples ko training examples likhte waqt dekhne mat do -- agar tum dono ko usi mutthi bhar templates se generate karte ho, to tumhara held-out split (Step 5) overfitting actually catch nahi karega, kyunki "held out" ka matlab tabhi hai jab examples genuinely alag situations ho.

---

Step 3 -- JSONL banao (zero se re-derive kiya gaya)

Hosted chat fine-tuning format (specifically OpenAI API se scoped -- doosre providers/frameworks alag shapes use karte hain) hai har line pe ek JSON object, kabhi bhi sab kuch wrap karta ek JSON array nahi. Har line ka object ek key rakhta hai, messages, jo Step 2 wali wahi three-role list hold karta hai. Neeche ka code chhe examples mein se har ek ko apni khud ki line mein likhta hai ek trailing newline ke saath -- koi square brackets nahi, examples ke beech koi commas nahi, bas har line pe ek poora object. Ye step real mein chala; pehli raw line ko dekhna confirm karta hai ki ye genuinely ek flat JSON object hai, kisi list ka hissa nahi.

---

Step 4 -- Validator zero se re-derive karo, aur ek GOOD aur ek BAD file pe chalao

Training run pe kuch bhi kharch karne se pehle, ek validator har line ko un mistakes ke liye check karta hai jo actually ek fine-tune todh deti hain: kya har line valid JSON bhi hai, kya har message ka role allowed hai (system / user / assistant), kya har example ka AAKHRI message assistant turn hai (yehi ek cheez actually seekhi jaa rahi hai -- ek user-last row kuch nahi sikhaati), aur kya koi duplicate examples hain (identical messages lists, jo ek training slot waste karte hain aur model ko us ek reply ki taraf skew kar dete hain).

Hamare chhe saaf examples pe chalane se, validator zero errors report karta hai. Ye prove karne ke liye ki ye actually problems catch karta hai aur sirf decoration nahi hai, ek doosri file banayi gayi teen jaan-boojh kar alag, real mistakes ke saath: ek row assistant ki jagah user message pe khatam hoti hai, ek row mein genuinely malformed JSON hai (ek missing comma), aur ek pichhle example ka exact duplicate hai. Validator ne teenon pakde, har ek ke saath ek distinct, specific error message -- yehi wo check hai jo tum hosted API ko touch karne se pehle chalate ho.

---

Step 5 -- Train/val split

Ek validated file ko phir bhi ek held-out split chahiye: examples jin pe model kabhi train nahi hota, baad mein ye check karne ke liye use hote hain ki kya wo actually voice seekh raha hai ya bas memorize kar raha hai. Neeche ka split ek fixed seed se shuffle karta hai (taaki ye reproducible ho) aur roughly 20% validation ke liye leta hai, kam se kam ek example tak round up karke. Chhe examples pe ye paanch training ke liye hai aur ek held out -- chhota, par bilkul yehi split logic ek real dataset mein kuch sau examples tak scale karti hai.

---

Step 6 -- Training tokens estimate karo (kabhi bhi koi currency nahi)

Training cost dataset_tokens x epochs se scale hoti hai, aur yahan koi currency figure print karna ek aisa number sikhaana hoga jo provider ke price list update karte hi rot ho jaata hai -- to ye estimator sirf tokens print karta hai. tiktoken OpenAI-only hai -- ye OpenAI API ke liye sahi se tokens count karta hai aur Claude ya doosre providers ko undercount karta hai, to is number ko shuru se hi OpenAI-scoped maano. Har message mein add kiya gaya +4 per-message formatting overhead ka ek rough approximation hai, provider ka exact accounting nahi -- real byte-for-byte billing jo bhi provider ke current docs kahein wahi hai. Chhe-example dataset ke total tokens ko 3 epochs se multiply karne pe neeche wala real training-token count milta hai -- yehi number actually ek hosted training bill drive karta hai, chaahe provider aaj use kaise bhi price kare.

---

Step 7 -- Job payload construct karo -- aur wahin ruk jao

Yehi wo step hai jis tak har pichla chapter build-up le jaata hai, aur yehi wo step hai jo ye project jaan-boojh kar aage nahi le jaata. Current OpenAI Python SDK expect karta hai ki ek training job model, training_file, validation_file, aur modern method={"type": "supervised", "supervised": {"hyperparameters": {...}}} block ke saath banaya jaaye (flat hyperparameters= argument abhi bhi exist karta hai par legacy shape hai). Neeche ka code bilkul wahi shape ek plain Python dict ke roop mein banaata hai -- training_file aur validation_file yahan placeholders hain kyunki ye normally ek alag files.create(file=..., purpose="fine-tune") upload call se return hue file IDs hote, aur model naam bhi ek placeholder hai, kyunki model ID ek volatile fact hai jo tumhe provider ki current fine-tunable list se dekhna chahiye jab tum ise actually banao, kisi tutorial se copy nahi karna chahiye.

Ye dict print hota hai, kabhi bheja nahi jaata. Ise construct karna aur uski shape confirm karna real, verified kaam hai: ye current SDK ke argument names se bilkul match karta hai. Ise submit karna wo ek step hai jis pe ye project khatam nahi ho sakta.

---

Step 8 -- Appendix: illustrative submission (EXECUTE NAHI hua)

Upar sab kuch bina API key ke real mein chala. Ye appendix sirf isliye hai taaki "aage kya hota hai" ki shape honest aur complete rahe -- ye clearly labelled hai, aur isme se kuch bhi real output nahi hai:

# ---- ILLUSTRATIVE (hosted fine-tuning is not executed here; see notes) ----
file_train = client.files.create(file=open("tone_bot_train.jsonl", "rb"), purpose="fine-tune")
file_val = client.files.create(file=open("tone_bot_val.jsonl", "rb"), purpose="fine-tune")
job = client.fine_tuning.jobs.create(
    model="<current-fine-tunable-model-id>",
    training_file=file_train.id,
    validation_file=file_val.id,
    method={"type": "supervised", "supervised": {"hyperparameters": {"n_epochs": 3}}},
)
# job.status ek plain string hai ("validating_files" queued rehte waqt, "succeeded" khatam hone pe);
# job.fine_tuned_model job object pe ek SIBLING field hai, None jab tak succeed na ho --
# job.status.fine_tuned_model error dega, kyunki ek str ke paas aisa koi attribute nahi hota.
job = client.fine_tuning.jobs.retrieve(job.id)
print(job.status, job.fine_tuned_model)

Hosted fine-tuning job creation, ye likhte waqt, un organisations ke liye band hai jinhone pehle kabhi fine-tune nahi kiya, aur provider ne kaha hai ki wo generally self-serve fine-tuning wind down kar raha hai -- to upar sab kuch vocabulary samjho, perform karne wala step nahi: files upload, ek job object jiska status poll kiya jaa sake, aur ek naya model ID jo finished job pe dikhta hai. Yehi vocabulary hai jo interviews mein poochi jaati hai, aur ye survive karti hai chaahe ise khud chalaana abhi possible na ho. Hosted fine-tuning availability, pricing aur eligibility provider ke product decisions hain jo badalte rehte hain -- ise use karne ki planning se pehle provider ke current fine-tuning docs check karo.

---

✅ Jo tumne abhi banaya -- aur resume framing

Tumne ek consistent brand voice ko ek explicit spec ke roop mein design kiya, ek chhota dataset curate kiya jo use actually follow karta hai, ek genuinely valid JSONL file likhi, ek validator banaya aur chalaya jo teen real, distinct classes ke dataset errors pakadta hai, ek reproducible train/val split produce kiya, sahi tokenizer se dataset ki real token cost measure ki, aur -- current SDK ke real argument names se sahi match karta -- exact payload construct kiya jo ek hosted fine-tuning job ko chahiye hoga, kuch bhi kharch kiye bina ya aisi access pe depend kiye bina jo isse padhne wale kisi ke paas reliably nahi hai.

Ise seedha kaho, kyunki ye sach hai aur yehi point hai: ek validated dataset plus ek reproducible pipeline HI portfolio piece hai. Submitted job deliverable nahi hai -- ye fact ki tumhara JSONL real hai, tumhara validator genuinely bad rows pakadta hai, aur tumhara split reproducible hai, yehi wo cheez hai jo ek interviewer actually inspect aur trust kar sakta hai, chaahe kisi specific hosted provider ka fine-tuning darwaza is mahine khula ho ya nahi.

Resume/interview ke liye: "Built a fine-tuning data pipeline for a brand-voice support bot: designed a voice spec, curated and validated a JSONL training set (catching malformed JSON, wrong last-message role, and duplicate examples), produced a reproducible train/val split, estimated training cost in tokens, and constructed a hosted fine-tuning job payload against the current SDK's argument shape." Ye sentence dikhata hai ki tum poori pipeline samajhte ho, sirf ek API call kiya nahi.

Standard definition (interview me bolo): a fine-tuning project's real deliverable is the validated dataset and the reproducible pipeline that produced it -- a consistent voice spec, a JSONL file that passes structural validation, a genuine train/val split, and a correctly-shaped job payload -- because that artefact set is what proves the work regardless of whether a specific submitted training job ever ran.

import json
import random

# ---------- voice spec: the ONE thing every example must obey ----------
VOICE_SPEC = {
    "brand": "Nimbus (cloud storage support)",
    "tone": ["warm", "plain-English", "confident"],
    "banned_words": ["unfortunately", "as per policy", "kindly", "team will look into it"],
    "rules": [
        "always name ONE concrete next step",
        "use 'we', never 'the team'",
        "no corporate jargon or hedging",
        "keep replies under ~60 words",
    ],
}

SYSTEM_PROMPT = (
    "You are Nimbus support. Voice: warm, plain-English, confident. "
    "Never say 'unfortunately' or 'as per policy'. Always name one concrete next step. "
    "Say 'we', never 'the team'. Keep replies under 60 words."
)

# ---------- curated examples: SAME system prompt every time ----------
RAW_EXAMPLES = [
    ("My files stopped syncing since this morning.",
     "That's on us, not your setup -- we're aware and already rolling out a fix. "
     "Restart the Nimbus app once it lands (within the hour) and syncing will resume automatically."),
    ("I was charged twice for my subscription.",
     "Sorry for the mix-up -- we've flagged the duplicate charge and it'll refund to your original "
     "payment method within 3-5 business days. I'll email you the reference number in a minute."),
    ("How do I get my storage back after downgrading my plan?",
     "Downgrading just lowers your limit going forward -- your existing files stay put. "
     "If you're over the new limit, delete or archive files until you're under it, then sync resumes."),
    ("Can you delete my account and all my data?",
     "Done as soon as you confirm -- head to Settings -> Account -> Delete, and everything clears "
     "within 24 hours. Want me to send that link now?"),
    ("Your app crashes every time I open a video file.",
     "That's a real bug, not a fluke -- we've reproduced it and a fix ships Thursday. "
     "Until then, open videos from the web app instead; it isn't affected."),
    ("I want a feature to auto-tag photos by location.",
     "Love that idea -- I'm logging it for our product roadmap right now. "
     "Follow the changelog page and you'll see it if we build it."),
]

examples = [
    {"messages": [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": u},
        {"role": "assistant", "content": a},
    ]}
    for u, a in RAW_EXAMPLES
]

# ---------- write JSONL: one JSON object per LINE, never a JSON array ----------
GOOD_PATH = "tone_bot_good.jsonl"
with open(GOOD_PATH, "w", encoding="utf-8") as f:
    for ex in examples:
        f.write(json.dumps(ex) + "\n")

# ---------- re-derived validator (self-contained, no earlier topic assumed) ----------
ALLOWED_ROLES = {"system", "user", "assistant"}

def validate_jsonl(path):
    errors, seen, rows = [], set(), 0
    with open(path, encoding="utf-8") as fh:
        for i, line in enumerate(fh, start=1):
            line = line.strip()
            if not line:
                continue
            try:
                obj = json.loads(line)
            except json.JSONDecodeError as e:
                errors.append(f"line {i}: bad JSON - {e}")
                continue
            msgs = obj.get("messages")
            if not isinstance(msgs, list) or not msgs:
                errors.append(f"line {i}: missing or empty 'messages' list")
                continue
            for m in msgs:
                if m.get("role") not in ALLOWED_ROLES:
                    errors.append(f"line {i}: unknown role {m.get('role')!r}")
            if msgs[-1].get("role") != "assistant":
                errors.append(f"line {i}: last message must be assistant")
            key = json.dumps(msgs, sort_keys=True)
            if key in seen:
                errors.append(f"line {i}: duplicate example (identical messages)")
            seen.add(key)
            rows += 1
    return rows, errors

rows, errors = validate_jsonl(GOOD_PATH)
print(f"validator on GOOD file -> rows={rows} errors={errors}")

# ---------- a BAD file with 3 real, distinct violations ----------
BAD_PATH = "tone_bot_bad.jsonl"
bad_lines = [
    '{"messages": [{"role": "system", "content": "' + SYSTEM_PROMPT + '"}, '
    '{"role": "user", "content": "Where is my refund?"}, '
    '{"role": "user", "content": "Any update?"}]}',      # last message is user, not assistant
    '{"messages": [{"role": "system", "content": "oops" "broken}',  # malformed JSON
    json.dumps(examples[0]),   # first copy
    json.dumps(examples[0]),   # exact duplicate -- same messages list, byte for byte
]
with open(BAD_PATH, "w", encoding="utf-8") as f:
    for line in bad_lines:
        f.write(line + "\n")

rows_b, errors_b = validate_jsonl(BAD_PATH)
print(f"validator on BAD file -> rows={rows_b}")
for e in errors_b:
    print(" ", e)

# ---------- train/val split ----------
def split_train_val(items, val_ratio=0.2, seed=42):
    items = list(items)
    rnd = random.Random(seed)
    rnd.shuffle(items)
    n_val = max(1, round(len(items) * val_ratio))
    return items[n_val:], items[:n_val]

train, val = split_train_val(examples, val_ratio=0.2, seed=42)
print(f"split(seed=42, 80/20) on {len(examples)} rows -> train={len(train)} val={len(val)}")

with open("tone_bot_train.jsonl", "w", encoding="utf-8") as f:
    for ex in train:
        f.write(json.dumps(ex) + "\n")
with open("tone_bot_val.jsonl", "w", encoding="utf-8") as f:
    for ex in val:
        f.write(json.dumps(ex) + "\n")

# ---------- token estimate (tiktoken is OpenAI-only -- it does not count for Claude
# or other providers, and the per-message overhead below is an approximation, not
# the provider's exact billing formula) ----------
import tiktoken
enc = tiktoken.get_encoding("o200k_base")

def count_example_tokens(ex):
    total = 0
    for m in ex["messages"]:
        total += len(enc.encode(m["content"]))
        total += 4  # approximate per-message overhead
    return total

per_example = [count_example_tokens(ex) for ex in examples]
dataset_tokens = sum(per_example)
print(f"tiktoken o200k_base per-example -> {per_example}")
print(f"dataset total -> {dataset_tokens} tokens")
print(f"x3 epochs = {dataset_tokens * 3} training tokens")

# ---------- construct the job payload as a dict, print it, DO NOT SEND ----------
job_payload = {
    "model": "<current-fine-tunable-model-id>",  # model IDs change -- check the provider's current fine-tunable list
    "training_file": "file-<returned-by-files.create-on-train-jsonl>",
    "validation_file": "file-<returned-by-files.create-on-val-jsonl>",
    "method": {
        "type": "supervised",
        "supervised": {
            "hyperparameters": {
                "n_epochs": 3
            }
        },
    },
    "suffix": "nimbus-tone-bot",
}
print("CONSTRUCTED JOB PAYLOAD (dict, NOT sent to the API):")
print(json.dumps(job_payload, indent=2))
print("type(job_payload) ->", type(job_payload))